[SPARK-47653][SS] Add support for negative numeric types and range scan key encoder

[anishshri-db]

What changes were proposed in this pull request?

Add support for negative numeric types and range scan key encoder

Why are the changes needed?

Without this change, sort ordering for -ve numbers is not maintained on iteration. Negative numbers would appear last previously. Note that only non-floating integer types such as short, integer, long are supported for signed values. For float/double, we cannot simply prepend a sign byte given the way floating point values are stored in the IEEE 754 floating point representation. Additionally we also need to flip all the bits and convert them back to the original value on read, in order to support these types.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Added unit tests

[info] - rocksdb range scan - with prefix scan - with colFamiliesEnabled=true (with changelog checkpointing) (164 milliseconds)
[info] - rocksdb range scan - with prefix scan - with colFamiliesEnabled=true (without changelog checkpointing) (95 milliseconds)
[info] - rocksdb range scan - with prefix scan - with colFamiliesEnabled=false (with changelog checkpointing) (155 milliseconds)
[info] - rocksdb range scan - with prefix scan - with colFamiliesEnabled=false (without changelog checkpointing) (82 milliseconds)
12:55:54.184 WARN org.apache.spark.sql.execution.streaming.state.RocksDBStateStoreSuite:

===== POSSIBLE THREAD LEAK IN SUITE o.a.s.sql.execution.streaming.state.RocksDBStateStoreSuite, threads: rpc-boss-3-1 (daemon=true), ForkJoinPool.commonPool-worker-2 (daemon=true), shuffle-boss-6-1 (daemon=true), ForkJoinPool.commonPool-worker-1 (daemon=true) =====
[info] Run completed in 8 seconds, 888 milliseconds.
[info] Total number of tests run: 44
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 44, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[success] Total time: 21 s, completed Mar 29, 2024, 12:55:54 PM

Was this patch authored or co-authored using generative AI tooling?

No

[anishshri-db]

@HeartSaVioR @neilramaswamy @sahnib - PTAL, thx !

[neilramaswamy]

Mostly nits, but once concern about the reasoning for why we don't support negative floats/doubles.

[sahnib]

LGTM

[neilramaswamy]

LGTM.

[HeartSaVioR]

Only a couple of nits.

[HeartSaVioR]

+1 pending CI

[HeartSaVioR]

Thanks! Merging to master.