[SPARK-47310][SS] Add micro-benchmark for merge operations for multiple values in value portion of state store

[anishshri-db]

What changes were proposed in this pull request?

Add microbenchmark for merge operations for multiple values in value portion of state store

Why are the changes needed?

Micro-benchmark to understand performance with/without rows tracking around merge operations

As shown in the results, merge without tracking is consistently 3x faster

merging 10000 rows with 10 values per key (10000 rows to overwrite - rate 100):  Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
--------------------------------------------------------------------------------------------------------------------------------------------------------------
RocksDB (trackTotalNumberOfRows: true)                                                    519            533           7          0.0       51916.6       1.0X
RocksDB (trackTotalNumberOfRows: false)                                                   171            177           3          0.1       17083.9       3.0X

GH Actions here:

• https://github.com/anishshri-db/spark/actions/runs/8559698160
• https://github.com/anishshri-db/spark/actions/runs/8559694994

Difference is even more running locally (> 7x faster without tracking)

Does this PR introduce any user-facing change?

No

How was this patch tested?

Test only change

Was this patch authored or co-authored using generative AI tooling?

No

[anishshri-db]

@sahnib @HeartSaVioR - PTAL, thx !

[sahnib]

LGTM, thanks for adding this.

[HeartSaVioR]

+1

[HeartSaVioR]

Thanks! Merging to master.