-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(db): Configurable maximum number of open RocksDB files #1401
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rocksdb::Options::set_max_open_files(). This allows Merkle tree users to limit the number of open files required by RocksDB, which otherwise grows beyond normal OS limits (8192 files) and requires large amounts of memory (currently, beyond 32GB).
vbar
changed the title
Configurable maximum number of open files
feat: Configurable maximum number of open files
Mar 11, 2024
slowli
reviewed
Mar 13, 2024
slowli
previously approved these changes
Mar 13, 2024
slowli
approved these changes
Mar 14, 2024
slowli
changed the title
feat: Configurable maximum number of open files
feat(db): Configurable maximum number of open RocksDB files
Mar 14, 2024
github-merge-queue bot
pushed a commit
that referenced
this pull request
Mar 22, 2024
🤖 I have created a release *beep* *boop* --- ## [22.0.0](core-v21.1.0...core-v22.0.0) (2024-03-21) ### ⚠ BREAKING CHANGES * Use protocol version v22 as latest ([#1432](#1432)) ### Features * add docs for Dal ([#1273](#1273)) ([66ceb0b](66ceb0b)) * adds debug_traceBlockByNumber.callFlatTracer ([#1413](#1413)) ([d2a5e36](d2a5e36)) * **api:** introduce mempool cache ([#1460](#1460)) ([c5d6c4b](c5d6c4b)) * **commitment-generator:** `events_queue` shadow mode ([#1138](#1138)) ([9bb47fa](9bb47fa)) * **contract-verifier:** Allow sc code reverification ([#1455](#1455)) ([5a37b42](5a37b42)) * **database:** add an optional master replica max connections settings ([#1472](#1472)) ([e5c8127](e5c8127)) * **db:** Configurable maximum number of open RocksDB files ([#1401](#1401)) ([b00c052](b00c052)) * **en:** Check recipient contract and function selector in consistency checker ([#1367](#1367)) ([ea5c684](ea5c684)) * Follow-up for DAL split ([#1464](#1464)) ([c072288](c072288)) * **node_framework:** Ergonomic improvements ([#1453](#1453)) ([09b6887](09b6887)) * **node_framework:** Only store each wiring layer once ([#1468](#1468)) ([4a393dc](4a393dc)) * **node_framework:** Support for preconditions and oneshot tasks ([#1398](#1398)) ([65ea881](65ea881)) * **node-framework:** Add eth sender layer ([#1390](#1390)) ([0affdf8](0affdf8)) * **node-framework:** Add housekeeper layer ([#1409](#1409)) ([702e739](702e739)) * Remove batch dry_run ([#1076](#1076)) ([b82d093](b82d093)) * Separate Prover and Server DAL ([#1334](#1334)) ([103a56b](103a56b)) * **storage:** Make the storage caches task cancellation aware ([#1430](#1430)) ([ab532bb](ab532bb)) * support running consensus from snapshot (BFT-418) ([#1429](#1429)) ([f9f4d38](f9f4d38)) * **tx-sender:** Limit concurrent tx submissions ([#1473](#1473)) ([4bdf3ca](4bdf3ca)) * Use protocol version v22 as latest ([#1432](#1432)) ([1757412](1757412)) * **vm:** Prestate tracer implementation ([#1306](#1306)) ([c36be65](c36be65)) ### Bug Fixes * **core:** drop correct index of proof_generation_details table during database migration ([#1199](#1199)) ([76a6821](76a6821)) * **dal:** correct `tx_index_in_l1_batch` in `get_vm_events_for_l1_batch` ([#1463](#1463)) ([8bf37ac](8bf37ac)) * **node_framework:** Fix main node example ([#1470](#1470)) ([ac4a744](ac4a744)) * **protocol:** Remove verifier address from protocol upgrade ([#1443](#1443)) ([90dee73](90dee73)) * **reth:** use reth instead of geth ([#1410](#1410)) ([bd98dc7](bd98dc7)) * **verified sources fetcher:** Use correct `contact_name` for `SolSingleFile` ([#1433](#1433)) ([0764227](0764227)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
montekki
pushed a commit
that referenced
this pull request
Mar 22, 2024
🤖 I have created a release *beep* *boop* --- ## [22.0.0](core-v21.1.0...core-v22.0.0) (2024-03-21) ### ⚠ BREAKING CHANGES * Use protocol version v22 as latest ([#1432](#1432)) ### Features * add docs for Dal ([#1273](#1273)) ([66ceb0b](66ceb0b)) * adds debug_traceBlockByNumber.callFlatTracer ([#1413](#1413)) ([d2a5e36](d2a5e36)) * **api:** introduce mempool cache ([#1460](#1460)) ([c5d6c4b](c5d6c4b)) * **commitment-generator:** `events_queue` shadow mode ([#1138](#1138)) ([9bb47fa](9bb47fa)) * **contract-verifier:** Allow sc code reverification ([#1455](#1455)) ([5a37b42](5a37b42)) * **database:** add an optional master replica max connections settings ([#1472](#1472)) ([e5c8127](e5c8127)) * **db:** Configurable maximum number of open RocksDB files ([#1401](#1401)) ([b00c052](b00c052)) * **en:** Check recipient contract and function selector in consistency checker ([#1367](#1367)) ([ea5c684](ea5c684)) * Follow-up for DAL split ([#1464](#1464)) ([c072288](c072288)) * **node_framework:** Ergonomic improvements ([#1453](#1453)) ([09b6887](09b6887)) * **node_framework:** Only store each wiring layer once ([#1468](#1468)) ([4a393dc](4a393dc)) * **node_framework:** Support for preconditions and oneshot tasks ([#1398](#1398)) ([65ea881](65ea881)) * **node-framework:** Add eth sender layer ([#1390](#1390)) ([0affdf8](0affdf8)) * **node-framework:** Add housekeeper layer ([#1409](#1409)) ([702e739](702e739)) * Remove batch dry_run ([#1076](#1076)) ([b82d093](b82d093)) * Separate Prover and Server DAL ([#1334](#1334)) ([103a56b](103a56b)) * **storage:** Make the storage caches task cancellation aware ([#1430](#1430)) ([ab532bb](ab532bb)) * support running consensus from snapshot (BFT-418) ([#1429](#1429)) ([f9f4d38](f9f4d38)) * **tx-sender:** Limit concurrent tx submissions ([#1473](#1473)) ([4bdf3ca](4bdf3ca)) * Use protocol version v22 as latest ([#1432](#1432)) ([1757412](1757412)) * **vm:** Prestate tracer implementation ([#1306](#1306)) ([c36be65](c36be65)) ### Bug Fixes * **core:** drop correct index of proof_generation_details table during database migration ([#1199](#1199)) ([76a6821](76a6821)) * **dal:** correct `tx_index_in_l1_batch` in `get_vm_events_for_l1_batch` ([#1463](#1463)) ([8bf37ac](8bf37ac)) * **node_framework:** Fix main node example ([#1470](#1470)) ([ac4a744](ac4a744)) * **protocol:** Remove verifier address from protocol upgrade ([#1443](#1443)) ([90dee73](90dee73)) * **reth:** use reth instead of geth ([#1410](#1410)) ([bd98dc7](bd98dc7)) * **verified sources fetcher:** Use correct `contact_name` for `SolSingleFile` ([#1433](#1433)) ([0764227](0764227)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What ❔
Added a field to RocksDBOptions, which is used as an argument to rocksdb::Options::set_max_open_files().
Why ❔
This allows Merkle tree users to limit the number of open files required by RocksDB, which otherwise grows beyond normal OS limits (8192 files) and requires large amounts of memory (currently, beyond 32GB).
Checklist
zk fmt
andzk lint
.zk spellcheck
.zk linkcheck
.