Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(db): Configurable maximum number of open RocksDB files #1401

Merged
merged 8 commits into from Mar 14, 2024
Merged

feat(db): Configurable maximum number of open RocksDB files #1401

merged 8 commits into from Mar 14, 2024

Conversation

vbar
Copy link
Contributor

@vbar vbar commented Mar 11, 2024

What ❔

Added a field to RocksDBOptions, which is used as an argument to rocksdb::Options::set_max_open_files().

Why ❔

This allows Merkle tree users to limit the number of open files required by RocksDB, which otherwise grows beyond normal OS limits (8192 files) and requires large amounts of memory (currently, beyond 32GB).

Checklist

  • PR title corresponds to the body of PR (we generate changelog entries from PRs).
  • Tests for the changes have been added / updated.
    • current (integration) tests run as before
    • external program requiring the change is now running
  • Documentation comments have been added / updated.
    • all new fields and methods are documented
  • Code has been formatted via zk fmt and zk lint.
  • Spellcheck has been run via zk spellcheck.
  • Linkcheck has been run via zk linkcheck.
    • no links added (nor modified)

@vbar
Added a field to RocksDBOptions, which is used as an argument to
be3881c 
rocksdb::Options::set_max_open_files(). This allows Merkle tree users
to limit the number of open files required by RocksDB, which otherwise
grows beyond normal OS limits (8192 files) and requires large amounts
of memory (currently, beyond 32GB).
@vbar
Added RocksDBWrapper constructor.
6620f82
@vbar
Imported RocksDBOptions.
cc2b679
@vbar vbar changed the title Configurable maximum number of open files feat: Configurable maximum number of open files Mar 11, 2024
@popzxc popzxc requested a review from slowli March 13, 2024 06:33
slowli
slowli reviewed Mar 13, 2024
View reviewed changes
core/lib/merkle_tree/src/storage/rocksdb.rs Outdated Show resolved Hide resolved
core/lib/storage/src/db.rs Outdated Show resolved Hide resolved
core/lib/storage/src/db.rs Outdated Show resolved Hide resolved
slowli
slowli previously approved these changes Mar 13, 2024
View reviewed changes
core/lib/storage/src/db.rs Outdated Show resolved Hide resolved
@slowli slowli changed the title feat: Configurable maximum number of open files feat(db): Configurable maximum number of open RocksDB files Mar 14, 2024
@slowli slowli added this pull request to the merge queue Mar 14, 2024
Merged via the queue into matter-labs:main with commit b00c052 Mar 14, 2024
30 checks passed
@zksync-era-bot zksync-era-bot mentioned this pull request Mar 14, 2024
Merged
github-merge-queue bot pushed a commit that referenced this pull request Mar 22, 2024
@zksync-era-bot
chore(main): release core 22.0.0 (#1424)
14a8b78 
🤖 I have created a release *beep* *boop*
---


##
[22.0.0](core-v21.1.0...core-v22.0.0)
(2024-03-21)


### ⚠ BREAKING CHANGES

* Use protocol version v22 as latest
([#1432](#1432))

### Features

* add docs for Dal
([#1273](#1273))
([66ceb0b](66ceb0b))
* adds debug_traceBlockByNumber.callFlatTracer
([#1413](#1413))
([d2a5e36](d2a5e36))
* **api:** introduce mempool cache
([#1460](#1460))
([c5d6c4b](c5d6c4b))
* **commitment-generator:** `events_queue` shadow mode
([#1138](#1138))
([9bb47fa](9bb47fa))
* **contract-verifier:** Allow sc code reverification
([#1455](#1455))
([5a37b42](5a37b42))
* **database:** add an optional master replica max connections settings
([#1472](#1472))
([e5c8127](e5c8127))
* **db:** Configurable maximum number of open RocksDB files
([#1401](#1401))
([b00c052](b00c052))
* **en:** Check recipient contract and function selector in consistency
checker ([#1367](#1367))
([ea5c684](ea5c684))
* Follow-up for DAL split
([#1464](#1464))
([c072288](c072288))
* **node_framework:** Ergonomic improvements
([#1453](#1453))
([09b6887](09b6887))
* **node_framework:** Only store each wiring layer once
([#1468](#1468))
([4a393dc](4a393dc))
* **node_framework:** Support for preconditions and oneshot tasks
([#1398](#1398))
([65ea881](65ea881))
* **node-framework:** Add eth sender layer
([#1390](#1390))
([0affdf8](0affdf8))
* **node-framework:** Add housekeeper layer
([#1409](#1409))
([702e739](702e739))
* Remove batch dry_run
([#1076](#1076))
([b82d093](b82d093))
* Separate Prover and Server DAL
([#1334](#1334))
([103a56b](103a56b))
* **storage:** Make the storage caches task cancellation aware
([#1430](#1430))
([ab532bb](ab532bb))
* support running consensus from snapshot (BFT-418)
([#1429](#1429))
([f9f4d38](f9f4d38))
* **tx-sender:** Limit concurrent tx submissions
([#1473](#1473))
([4bdf3ca](4bdf3ca))
* Use protocol version v22 as latest
([#1432](#1432))
([1757412](1757412))
* **vm:** Prestate tracer implementation
([#1306](#1306))
([c36be65](c36be65))


### Bug Fixes

* **core:** drop correct index of proof_generation_details table during
database migration
([#1199](#1199))
([76a6821](76a6821))
* **dal:** correct `tx_index_in_l1_batch` in
`get_vm_events_for_l1_batch`
([#1463](#1463))
([8bf37ac](8bf37ac))
* **node_framework:** Fix main node example
([#1470](#1470))
([ac4a744](ac4a744))
* **protocol:** Remove verifier address from protocol upgrade
([#1443](#1443))
([90dee73](90dee73))
* **reth:** use reth instead of geth
([#1410](#1410))
([bd98dc7](bd98dc7))
* **verified sources fetcher:** Use correct `contact_name` for
`SolSingleFile`
([#1433](#1433))
([0764227](0764227))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
montekki pushed a commit that referenced this pull request Mar 22, 2024
@zksync-era-bot @montekki
chore(main): release core 22.0.0 (#1424)
9b35900 
🤖 I have created a release *beep* *boop*
---


##
[22.0.0](core-v21.1.0...core-v22.0.0)
(2024-03-21)


### ⚠ BREAKING CHANGES

* Use protocol version v22 as latest
([#1432](#1432))

### Features

* add docs for Dal
([#1273](#1273))
([66ceb0b](66ceb0b))
* adds debug_traceBlockByNumber.callFlatTracer
([#1413](#1413))
([d2a5e36](d2a5e36))
* **api:** introduce mempool cache
([#1460](#1460))
([c5d6c4b](c5d6c4b))
* **commitment-generator:** `events_queue` shadow mode
([#1138](#1138))
([9bb47fa](9bb47fa))
* **contract-verifier:** Allow sc code reverification
([#1455](#1455))
([5a37b42](5a37b42))
* **database:** add an optional master replica max connections settings
([#1472](#1472))
([e5c8127](e5c8127))
* **db:** Configurable maximum number of open RocksDB files
([#1401](#1401))
([b00c052](b00c052))
* **en:** Check recipient contract and function selector in consistency
checker ([#1367](#1367))
([ea5c684](ea5c684))
* Follow-up for DAL split
([#1464](#1464))
([c072288](c072288))
* **node_framework:** Ergonomic improvements
([#1453](#1453))
([09b6887](09b6887))
* **node_framework:** Only store each wiring layer once
([#1468](#1468))
([4a393dc](4a393dc))
* **node_framework:** Support for preconditions and oneshot tasks
([#1398](#1398))
([65ea881](65ea881))
* **node-framework:** Add eth sender layer
([#1390](#1390))
([0affdf8](0affdf8))
* **node-framework:** Add housekeeper layer
([#1409](#1409))
([702e739](702e739))
* Remove batch dry_run
([#1076](#1076))
([b82d093](b82d093))
* Separate Prover and Server DAL
([#1334](#1334))
([103a56b](103a56b))
* **storage:** Make the storage caches task cancellation aware
([#1430](#1430))
([ab532bb](ab532bb))
* support running consensus from snapshot (BFT-418)
([#1429](#1429))
([f9f4d38](f9f4d38))
* **tx-sender:** Limit concurrent tx submissions
([#1473](#1473))
([4bdf3ca](4bdf3ca))
* Use protocol version v22 as latest
([#1432](#1432))
([1757412](1757412))
* **vm:** Prestate tracer implementation
([#1306](#1306))
([c36be65](c36be65))


### Bug Fixes

* **core:** drop correct index of proof_generation_details table during
database migration
([#1199](#1199))
([76a6821](76a6821))
* **dal:** correct `tx_index_in_l1_batch` in
`get_vm_events_for_l1_batch`
([#1463](#1463))
([8bf37ac](8bf37ac))
* **node_framework:** Fix main node example
([#1470](#1470))
([ac4a744](ac4a744))
* **protocol:** Remove verifier address from protocol upgrade
([#1443](#1443))
([90dee73](90dee73))
* **reth:** use reth instead of geth
([#1410](#1410))
([bd98dc7](bd98dc7))
* **verified sources fetcher:** Use correct `contact_name` for
`SolSingleFile`
([#1433](#1433))
([0764227](0764227))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@slowli slowli slowli approved these changes

Assignees
No one assigned
Labels
external-contribution
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@vbar @slowli