Skip to content

Commit

Permalink
[SPARK-47278][BUILD] Upgrade rocksdbjni to 8.11.3
Browse files Browse the repository at this point in the history 
### What changes were proposed in this pull request?
This pr aims to upgrade rocksdbjni from 8.8.1 to 8.11.3.

### Why are the changes needed?
The new version bring some bug fixes:

- Fix a corner case with auto_readahead_size where Prev Operation returns NOT SUPPORTED error when scans direction is changed from forward to backward.
- Avoid destroying the periodic task scheduler's default timer in order to prevent static destruction order issues.
- Fix double counting of BYTES_WRITTEN ticker when doing writes with transactions.
- Fix a WRITE_STALL counter that was reporting wrong value in few cases.
- A lookup by MultiGet in a TieredCache that goes to the local flash cache and finishes with very low latency, i.e before the subsequent call to WaitAll, is ignored, resulting in a false negative and a memory leak.
- Fix bug in auto_readahead_size that combined with IndexType::kBinarySearchWithFirstKey + fails or iterator lands at a wrong key
- Fixed some cases in which DB file corruption was detected but ignored on creating a backup with BackupEngine.
- Fix bugs where rocksdb.blobdb.blob.file.synced includes blob files failed to get synced and rocksdb.blobdb.blob.file.bytes.written includes blob bytes failed to get written.
- Fixed a possible memory leak or crash on a failure (such as I/O error) in automatic atomic flush of multiple column families.
- Fixed some cases of in-memory data corruption using mmap reads with BackupEngine, sst_dump, or ldb.
- Fixed issues with experimental preclude_last_level_data_seconds option that could interfere with expected data tiering.
- Fixed the handling of the edge case when all existing blob files become unreferenced. Such files are now correctly deleted.

The full release notes as follows:

- https://github.com/facebook/rocksdb/releases/tag/v8.9.1
- https://github.com/facebook/rocksdb/releases/tag/v8.10.0
- https://github.com/facebook/rocksdb/releases/tag/v8.11.3

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
Pass GitHub Actions

### Was this patch authored or co-authored using generative AI tooling?
No

Closes #45365 from LuciferYang/rocksdbjni-8.11.3.

Lead-authored-by: yangjie01 <yangjie01@baidu.com>
Co-authored-by: YangJie <yangjie01@baidu.com>
Signed-off-by: Kent Yao <yao@apache.org>
  • Loading branch information
@LuciferYang @yaooqinn
LuciferYang authored and yaooqinn committed Mar 7, 2024
1 parent 620297a commit 3112762
Show file tree
Hide file tree
Showing 4 changed files with 97 additions and 97 deletions.
2 changes: 1 addition & 1 deletion dev/deps/spark-deps-hadoop-3-hive-2.3
View file
Original file line number Diff line number Diff line change
Expand Up @@ -242,7 +242,7 @@ parquet-jackson/1.13.1//parquet-jackson-1.13.1.jar
pickle/1.3//pickle-1.3.jar
py4j/0.10.9.7//py4j-0.10.9.7.jar
remotetea-oncrpc/1.1.2//remotetea-oncrpc-1.1.2.jar
rocksdbjni/8.8.1//rocksdbjni-8.8.1.jar
rocksdbjni/8.11.3//rocksdbjni-8.11.3.jar
scala-collection-compat_2.13/2.7.0//scala-collection-compat_2.13-2.7.0.jar
scala-compiler/2.13.12//scala-compiler-2.13.12.jar
scala-library/2.13.12//scala-library-2.13.12.jar
Expand Down
2 changes: 1 addition & 1 deletion pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -684,7 +684,7 @@
<dependency>
<groupId>org.rocksdb</groupId>
<artifactId>rocksdbjni</artifactId>
<version>8.8.1</version>
<version>8.11.3</version>
</dependency>
<dependency>
<groupId>${leveldbjni.group}</groupId>
Expand Down
94 changes: 47 additions & 47 deletions sql/core/benchmarks/StateStoreBasicOperationsBenchmark-jdk21-results.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,110 +2,110 @@
put rows
================================================================================================

OpenJDK 64-Bit Server VM 21.0.1+12-LTS on Linux 5.15.0-1053-azure
OpenJDK 64-Bit Server VM 21.0.2+13-LTS on Linux 5.15.0-1057-azure
AMD EPYC 7763 64-Core Processor
putting 10000 rows (10000 rows to overwrite - rate 100): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------------------
In-memory 6 7 1 1.8 556.4 1.0X
RocksDB (trackTotalNumberOfRows: true) 43 44 2 0.2 4292.9 0.1X
RocksDB (trackTotalNumberOfRows: false) 15 16 1 0.7 1525.0 0.4X
In-memory 5 6 1 1.8 549.2 1.0X
RocksDB (trackTotalNumberOfRows: true) 40 42 2 0.2 4031.0 0.1X
RocksDB (trackTotalNumberOfRows: false) 15 15 1 0.7 1486.0 0.4X

OpenJDK 64-Bit Server VM 21.0.1+12-LTS on Linux 5.15.0-1053-azure
OpenJDK 64-Bit Server VM 21.0.2+13-LTS on Linux 5.15.0-1057-azure
AMD EPYC 7763 64-Core Processor
putting 10000 rows (5000 rows to overwrite - rate 50): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------------------------------------------------
In-memory 6 7 1 1.8 550.7 1.0X
RocksDB (trackTotalNumberOfRows: true) 36 37 1 0.3 3607.8 0.2X
RocksDB (trackTotalNumberOfRows: false) 15 16 1 0.7 1535.3 0.4X
In-memory 5 6 1 1.9 539.1 1.0X
RocksDB (trackTotalNumberOfRows: true) 39 40 1 0.3 3947.0 0.1X
RocksDB (trackTotalNumberOfRows: false) 15 16 1 0.7 1493.7 0.4X

OpenJDK 64-Bit Server VM 21.0.1+12-LTS on Linux 5.15.0-1053-azure
OpenJDK 64-Bit Server VM 21.0.2+13-LTS on Linux 5.15.0-1057-azure
AMD EPYC 7763 64-Core Processor
putting 10000 rows (1000 rows to overwrite - rate 10): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------------------------------------------------
In-memory 5 6 1 1.9 535.0 1.0X
RocksDB (trackTotalNumberOfRows: true) 29 31 1 0.3 2940.4 0.2X
RocksDB (trackTotalNumberOfRows: false) 15 16 1 0.7 1518.9 0.4X
In-memory 5 6 0 1.9 522.0 1.0X
RocksDB (trackTotalNumberOfRows: true) 39 40 1 0.3 3868.2 0.1X
RocksDB (trackTotalNumberOfRows: false) 15 16 1 0.7 1489.6 0.4X

OpenJDK 64-Bit Server VM 21.0.1+12-LTS on Linux 5.15.0-1053-azure
OpenJDK 64-Bit Server VM 21.0.2+13-LTS on Linux 5.15.0-1057-azure
AMD EPYC 7763 64-Core Processor
putting 10000 rows (0 rows to overwrite - rate 0): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------------
In-memory 5 6 1 1.9 518.3 1.0X
RocksDB (trackTotalNumberOfRows: true) 28 29 1 0.4 2755.7 0.2X
RocksDB (trackTotalNumberOfRows: false) 15 16 0 0.7 1512.0 0.3X
In-memory 5 6 0 1.9 513.4 1.0X
RocksDB (trackTotalNumberOfRows: true) 38 39 1 0.3 3833.9 0.1X
RocksDB (trackTotalNumberOfRows: false) 15 15 0 0.7 1488.5 0.3X


================================================================================================
delete rows
================================================================================================

OpenJDK 64-Bit Server VM 21.0.1+12-LTS on Linux 5.15.0-1053-azure
OpenJDK 64-Bit Server VM 21.0.2+13-LTS on Linux 5.15.0-1057-azure
AMD EPYC 7763 64-Core Processor
trying to delete 10000 rows from 10000 rows(10000 rows are non-existing - rate 100): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------------------------------------------------------------------------------
In-memory 0 1 0 27.2 36.8 1.0X
RocksDB (trackTotalNumberOfRows: true) 28 29 0 0.4 2784.0 0.0X
RocksDB (trackTotalNumberOfRows: false) 15 16 1 0.7 1538.2 0.0X
In-memory 0 0 0 27.9 35.8 1.0X
RocksDB (trackTotalNumberOfRows: true) 39 39 1 0.3 3856.1 0.0X
RocksDB (trackTotalNumberOfRows: false) 15 16 0 0.7 1519.1 0.0X

OpenJDK 64-Bit Server VM 21.0.1+12-LTS on Linux 5.15.0-1053-azure
OpenJDK 64-Bit Server VM 21.0.2+13-LTS on Linux 5.15.0-1057-azure
AMD EPYC 7763 64-Core Processor
trying to delete 10000 rows from 10000 rows(5000 rows are non-existing - rate 50): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
-----------------------------------------------------------------------------------------------------------------------------------------------------------------
In-memory 4 5 1 2.4 413.9 1.0X
RocksDB (trackTotalNumberOfRows: true) 36 37 1 0.3 3583.1 0.1X
RocksDB (trackTotalNumberOfRows: false) 15 16 0 0.6 1546.3 0.3X
In-memory 4 4 0 2.5 400.9 1.0X
RocksDB (trackTotalNumberOfRows: true) 39 40 1 0.3 3918.6 0.1X
RocksDB (trackTotalNumberOfRows: false) 15 15 0 0.7 1461.5 0.3X

OpenJDK 64-Bit Server VM 21.0.1+12-LTS on Linux 5.15.0-1053-azure
OpenJDK 64-Bit Server VM 21.0.2+13-LTS on Linux 5.15.0-1057-azure
AMD EPYC 7763 64-Core Processor
trying to delete 10000 rows from 10000 rows(1000 rows are non-existing - rate 10): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
-----------------------------------------------------------------------------------------------------------------------------------------------------------------
In-memory 5 5 1 2.2 459.3 1.0X
RocksDB (trackTotalNumberOfRows: true) 42 43 1 0.2 4189.6 0.1X
RocksDB (trackTotalNumberOfRows: false) 15 16 0 0.6 1539.5 0.3X
In-memory 4 5 0 2.3 443.1 1.0X
RocksDB (trackTotalNumberOfRows: true) 40 41 1 0.3 3994.9 0.1X
RocksDB (trackTotalNumberOfRows: false) 15 15 1 0.7 1453.0 0.3X

OpenJDK 64-Bit Server VM 21.0.1+12-LTS on Linux 5.15.0-1053-azure
OpenJDK 64-Bit Server VM 21.0.2+13-LTS on Linux 5.15.0-1057-azure
AMD EPYC 7763 64-Core Processor
trying to delete 10000 rows from 10000 rows(0 rows are non-existing - rate 0): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------------------------------------------------------------------------
In-memory 5 5 1 2.1 471.5 1.0X
RocksDB (trackTotalNumberOfRows: true) 43 44 1 0.2 4323.6 0.1X
RocksDB (trackTotalNumberOfRows: false) 15 16 0 0.6 1546.7 0.3X
In-memory 5 5 0 2.2 456.1 1.0X
RocksDB (trackTotalNumberOfRows: true) 40 41 1 0.3 3994.5 0.1X
RocksDB (trackTotalNumberOfRows: false) 15 15 0 0.7 1461.0 0.3X


================================================================================================
evict rows
================================================================================================

OpenJDK 64-Bit Server VM 21.0.1+12-LTS on Linux 5.15.0-1053-azure
OpenJDK 64-Bit Server VM 21.0.2+13-LTS on Linux 5.15.0-1057-azure
AMD EPYC 7763 64-Core Processor
evicting 10000 rows (maxTimestampToEvictInMillis: 9999) from 10000 rows: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------------------------------------------------------------------
In-memory 4 5 0 2.2 449.1 1.0X
RocksDB (trackTotalNumberOfRows: true) 41 42 1 0.2 4146.8 0.1X
RocksDB (trackTotalNumberOfRows: false) 15 15 0 0.7 1534.8 0.3X
In-memory 4 5 0 2.2 448.8 1.0X
RocksDB (trackTotalNumberOfRows: true) 39 40 1 0.3 3932.4 0.1X
RocksDB (trackTotalNumberOfRows: false) 16 16 1 0.6 1564.3 0.3X

OpenJDK 64-Bit Server VM 21.0.1+12-LTS on Linux 5.15.0-1053-azure
OpenJDK 64-Bit Server VM 21.0.2+13-LTS on Linux 5.15.0-1057-azure
AMD EPYC 7763 64-Core Processor
evicting 5000 rows (maxTimestampToEvictInMillis: 4999) from 10000 rows: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------------------------------------
In-memory 4 4 0 2.5 405.7 1.0X
RocksDB (trackTotalNumberOfRows: true) 22 23 1 0.4 2233.4 0.2X
RocksDB (trackTotalNumberOfRows: false) 9 9 0 1.1 919.4 0.4X
In-memory 4 4 0 2.4 411.1 1.0X
RocksDB (trackTotalNumberOfRows: true) 21 22 0 0.5 2109.2 0.2X
RocksDB (trackTotalNumberOfRows: false) 9 9 0 1.1 933.0 0.4X

OpenJDK 64-Bit Server VM 21.0.1+12-LTS on Linux 5.15.0-1053-azure
OpenJDK 64-Bit Server VM 21.0.2+13-LTS on Linux 5.15.0-1057-azure
AMD EPYC 7763 64-Core Processor
evicting 1000 rows (maxTimestampToEvictInMillis: 999) from 10000 rows: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
-----------------------------------------------------------------------------------------------------------------------------------------------------
In-memory 3 4 0 2.9 349.5 1.0X
RocksDB (trackTotalNumberOfRows: true) 7 7 0 1.4 698.9 0.5X
RocksDB (trackTotalNumberOfRows: false) 4 4 0 2.3 438.7 0.8X
In-memory 3 4 0 2.9 348.8 1.0X
RocksDB (trackTotalNumberOfRows: true) 7 7 0 1.5 673.5 0.5X
RocksDB (trackTotalNumberOfRows: false) 4 4 0 2.3 441.3 0.8X

OpenJDK 64-Bit Server VM 21.0.1+12-LTS on Linux 5.15.0-1053-azure
OpenJDK 64-Bit Server VM 21.0.2+13-LTS on Linux 5.15.0-1057-azure
AMD EPYC 7763 64-Core Processor
evicting 0 rows (maxTimestampToEvictInMillis: -1) from 10000 rows: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------------------------------------------------------------
In-memory 0 0 0 25.7 38.8 1.0X
RocksDB (trackTotalNumberOfRows: true) 3 3 0 3.3 307.0 0.1X
In-memory 0 0 0 24.2 41.3 1.0X
RocksDB (trackTotalNumberOfRows: true) 3 3 0 3.3 306.1 0.1X
RocksDB (trackTotalNumberOfRows: false) 3 3 0 3.3 305.9 0.1X


0 comments on commit 3112762

Please sign in to comment.