Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-42718][BUILD] Upgrade rocksdbjni to 7.10.2 #40337

Closed
wants to merge 2 commits into from

Conversation

LuciferYang
Copy link
Contributor

@LuciferYang LuciferYang commented Mar 8, 2023

What changes were proposed in this pull request?

This pr aims to upgrade rocksdbjni from 7.9.2 to 7.10.2.

Why are the changes needed?

This version bring the performance related DB reads:

The full release notes as follows:

Does this PR introduce any user-facing change?

No

How was this patch tested?

  • Pass GitHub Actions
  • Manual test RocksDBBenchmark:

7.9.2

[INFO] Running org.apache.spark.util.kvstore.RocksDBBenchmark
                                                count   mean    min     max     95th
dbClose                                         4       0.406   0.288   0.706   0.706
dbCreation                                      4       68.701  4.019   262.992 262.992
naturalIndexCreateIterator                      1024    0.005   0.002   1.473   0.007
naturalIndexDescendingCreateIterator            1024    0.006   0.005   0.061   0.007
naturalIndexDescendingIteration                 1024    0.007   0.004   0.278   0.011
naturalIndexIteration                           1024    0.007   0.004   0.071   0.012
randomDeleteIndexed                             1024    0.042   0.031   0.415   0.060
randomDeletesNoIndex                            1024    0.015   0.013   0.044   0.018
randomUpdatesIndexed                            1024    0.085   0.032   30.240  0.094
randomUpdatesNoIndex                            1024    0.054   0.050   0.740   0.059
randomWritesIndexed                             1024    0.127   0.034   54.828  0.125
randomWritesNoIndex                             1024    0.063   0.053   1.895   0.087
refIndexCreateIterator                          1024    0.004   0.003   0.022   0.005
refIndexDescendingCreateIterator                1024    0.003   0.003   0.026   0.004
refIndexDescendingIteration                     1024    0.006   0.005   0.044   0.008
refIndexIteration                               1024    0.010   0.005   0.051   0.015
sequentialDeleteIndexed                         1024    0.022   0.017   0.119   0.032
sequentialDeleteNoIndex                         1024    0.015   0.012   0.046   0.018
sequentialUpdatesIndexed                        1024    0.069   0.055   2.321   0.098
sequentialUpdatesNoIndex                        1024    0.067   0.046   0.927   0.088
sequentialWritesIndexed                         1024    0.085   0.055   2.308   0.113
sequentialWritesNoIndex                         1024    0.055   0.046   2.872   0.060

7.10.2

[INFO] Running org.apache.spark.util.kvstore.RocksDBBenchmark
                                                count   mean    min     max     95th
dbClose                                         4       0.348   0.287   0.521   0.521
dbCreation                                      4       69.416  3.810   267.620 267.620
naturalIndexCreateIterator                      1024    0.005   0.002   1.389   0.007
naturalIndexDescendingCreateIterator            1024    0.005   0.005   0.057   0.007
naturalIndexDescendingIteration                 1024    0.006   0.004   0.260   0.009
naturalIndexIteration                           1024    0.006   0.004   0.053   0.011
randomDeleteIndexed                             1024    0.028   0.019   0.306   0.046
randomDeletesNoIndex                            1024    0.015   0.012   0.045   0.018
randomUpdatesIndexed                            1024    0.086   0.032   29.664  0.109
randomUpdatesNoIndex                            1024    0.033   0.029   0.817   0.036
randomWritesIndexed                             1024    0.120   0.033   52.703  0.137
randomWritesNoIndex                             1024    0.039   0.033   1.734   0.044
refIndexCreateIterator                          1024    0.004   0.004   0.017   0.006
refIndexDescendingCreateIterator                1024    0.003   0.002   0.027   0.004
refIndexDescendingIteration                     1024    0.006   0.005   0.043   0.008
refIndexIteration                               1024    0.007   0.005   0.056   0.010
sequentialDeleteIndexed                         1024    0.022   0.017   1.388   0.024
sequentialDeleteNoIndex                         1024    0.015   0.012   0.037   0.018
sequentialUpdatesIndexed                        1024    0.041   0.035   0.852   0.050
sequentialUpdatesNoIndex                        1024    0.038   0.030   0.682   0.050
sequentialWritesIndexed                         1024    0.048   0.038   2.031   0.062
sequentialWritesNoIndex                         1024    0.035   0.030   2.617   0.039

@LuciferYang LuciferYang marked this pull request as draft March 8, 2023 12:44
@github-actions github-actions bot added the BUILD label Mar 8, 2023
@LuciferYang LuciferYang marked this pull request as ready for review March 9, 2023 07:03
Copy link
Member

@dongjoon-hyun dongjoon-hyun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, LGTM. Thank you, @LuciferYang and @srowen .
Merged to master for Apache Spark 3.5.0.

snmvaughan pushed a commit to snmvaughan/spark that referenced this pull request Jun 20, 2023
### What changes were proposed in this pull request?
This pr aims to upgrade rocksdbjni from 7.9.2 to 7.10.2.

### Why are the changes needed?
This version bring the performance related DB reads:

- facebook/rocksdb#10975

The full release notes as follows:

- https://github.com/facebook/rocksdb/releases/tag/v7.10.2

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?

- Pass GitHub Actions
- Manual test `RocksDBBenchmark`:

**7.9.2**

```
[INFO] Running org.apache.spark.util.kvstore.RocksDBBenchmark
                                                count   mean    min     max     95th
dbClose                                         4       0.406   0.288   0.706   0.706
dbCreation                                      4       68.701  4.019   262.992 262.992
naturalIndexCreateIterator                      1024    0.005   0.002   1.473   0.007
naturalIndexDescendingCreateIterator            1024    0.006   0.005   0.061   0.007
naturalIndexDescendingIteration                 1024    0.007   0.004   0.278   0.011
naturalIndexIteration                           1024    0.007   0.004   0.071   0.012
randomDeleteIndexed                             1024    0.042   0.031   0.415   0.060
randomDeletesNoIndex                            1024    0.015   0.013   0.044   0.018
randomUpdatesIndexed                            1024    0.085   0.032   30.240  0.094
randomUpdatesNoIndex                            1024    0.054   0.050   0.740   0.059
randomWritesIndexed                             1024    0.127   0.034   54.828  0.125
randomWritesNoIndex                             1024    0.063   0.053   1.895   0.087
refIndexCreateIterator                          1024    0.004   0.003   0.022   0.005
refIndexDescendingCreateIterator                1024    0.003   0.003   0.026   0.004
refIndexDescendingIteration                     1024    0.006   0.005   0.044   0.008
refIndexIteration                               1024    0.010   0.005   0.051   0.015
sequentialDeleteIndexed                         1024    0.022   0.017   0.119   0.032
sequentialDeleteNoIndex                         1024    0.015   0.012   0.046   0.018
sequentialUpdatesIndexed                        1024    0.069   0.055   2.321   0.098
sequentialUpdatesNoIndex                        1024    0.067   0.046   0.927   0.088
sequentialWritesIndexed                         1024    0.085   0.055   2.308   0.113
sequentialWritesNoIndex                         1024    0.055   0.046   2.872   0.060
```

**7.10.2**

```
[INFO] Running org.apache.spark.util.kvstore.RocksDBBenchmark
                                                count   mean    min     max     95th
dbClose                                         4       0.348   0.287   0.521   0.521
dbCreation                                      4       69.416  3.810   267.620 267.620
naturalIndexCreateIterator                      1024    0.005   0.002   1.389   0.007
naturalIndexDescendingCreateIterator            1024    0.005   0.005   0.057   0.007
naturalIndexDescendingIteration                 1024    0.006   0.004   0.260   0.009
naturalIndexIteration                           1024    0.006   0.004   0.053   0.011
randomDeleteIndexed                             1024    0.028   0.019   0.306   0.046
randomDeletesNoIndex                            1024    0.015   0.012   0.045   0.018
randomUpdatesIndexed                            1024    0.086   0.032   29.664  0.109
randomUpdatesNoIndex                            1024    0.033   0.029   0.817   0.036
randomWritesIndexed                             1024    0.120   0.033   52.703  0.137
randomWritesNoIndex                             1024    0.039   0.033   1.734   0.044
refIndexCreateIterator                          1024    0.004   0.004   0.017   0.006
refIndexDescendingCreateIterator                1024    0.003   0.002   0.027   0.004
refIndexDescendingIteration                     1024    0.006   0.005   0.043   0.008
refIndexIteration                               1024    0.007   0.005   0.056   0.010
sequentialDeleteIndexed                         1024    0.022   0.017   1.388   0.024
sequentialDeleteNoIndex                         1024    0.015   0.012   0.037   0.018
sequentialUpdatesIndexed                        1024    0.041   0.035   0.852   0.050
sequentialUpdatesNoIndex                        1024    0.038   0.030   0.682   0.050
sequentialWritesIndexed                         1024    0.048   0.038   2.031   0.062
sequentialWritesNoIndex                         1024    0.035   0.030   2.617   0.039
```

Closes apache#40337 from LuciferYang/SPARK-42718.

Lead-authored-by: yangjie01 <yangjie01@baidu.com>
Co-authored-by: YangJie <yangjie01@baidu.com>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
3 participants