chore: Merge main to kl factory #1716
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
🤖 I have created a release *beep* *boop* --- ## [23.0.0](core-v22.1.0...core-v23.0.0) (2024-04-16) ### ⚠ BREAKING CHANGES * **vm:** 1 5 0 support ([#1508](#1508)) ### Features * **api:** Add `tokens_whitelisted_for_paymaster` ([#1545](#1545)) ([6da89cd](6da89cd)) * **api:** Log info about estimated fee ([#1611](#1611)) ([daed58c](daed58c)) * Archive old prover jobs ([#1516](#1516)) ([201476c](201476c)) * Archiving of prover in gpu_prover_queue ([#1537](#1537)) ([a970629](a970629)) * **block-reverter:** only require private key for sending revert transactions ([#1579](#1579)) ([27de6b7](27de6b7)) * **config:** Initialize log config from files as well ([#1566](#1566)) ([9e7db59](9e7db59)) * **configs:** Implement new format of configs and implement protobuf for it ([#1501](#1501)) ([086ba5b](086ba5b)) * **db:** Wrap sqlx errors in DAL ([#1522](#1522)) ([6e9ed8c](6e9ed8c)) * EN Pruning ([#1418](#1418)) ([cea6578](cea6578)) * **en:** add consistency checker condition in db pruner ([#1653](#1653)) ([5ed92b9](5ed92b9)) * **en:** add manual vacuum step in db pruning ([#1652](#1652)) ([c818be3](c818be3)) * **en:** Rate-limit L2 client requests ([#1500](#1500)) ([3f55f1e](3f55f1e)) * **en:** Rework storing and using protective reads ([#1515](#1515)) ([13c0c45](13c0c45)) * **en:** support for snapshots recovery in version_sync_task.rs ([#1585](#1585)) ([f911276](f911276)) * **eth-watch:** Brush up Ethereum watcher component ([#1596](#1596)) ([b0b8f89](b0b8f89)) * Expose component configs as info metrics ([#1584](#1584)) ([7c8ae40](7c8ae40)) * **external-node:** external node distributed operation mode ([#1457](#1457)) ([777ffca](777ffca)) * Extract commitment generator into a separate crate ([#1636](#1636)) ([f763d1f](f763d1f)) * Extract eth_watch and shared metrics into separate crates ([#1572](#1572)) ([4013771](4013771)) * Finalize fee address migration ([#1617](#1617)) ([713f56b](713f56b)) * fix availability checker ([#1574](#1574)) ([b2f21fb](b2f21fb)) * **genesis:** Add genesis config generator ([#1671](#1671)) ([45164fa](45164fa)) * **genesis:** mark system contracts bytecodes as known ([#1554](#1554)) ([5ffec51](5ffec51)) * Migrate gas limit to u64 ([#1538](#1538)) ([56dc049](56dc049)) * **node-framework:** Add consensus support ([#1546](#1546)) ([27fe475](27fe475)) * **node-framework:** Add consistency checker ([#1527](#1527)) ([3c28c25](3c28c25)) * remove unused variables in prover configs ([#1564](#1564)) ([d32a019](d32a019)) * Remove zksync-rs SDK ([#1559](#1559)) ([cc78e1d](cc78e1d)) * soft removal of `events_queue` table ([#1504](#1504)) ([5899bc6](5899bc6)) * **sqlx:** Use offline mode by default ([#1539](#1539)) ([af01edd](af01edd)) * Use config for max number of circuits ([#1573](#1573)) ([9fcb87e](9fcb87e)) * Validium ([#1461](#1461)) ([132a169](132a169)) * **vm:** 1 5 0 support ([#1508](#1508)) ([a6ccd25](a6ccd25)) ### Bug Fixes * **api:** Change error code for Web3Error::NotImplemented ([#1521](#1521)) ([0a13602](0a13602)) * **cache:** use factory deps cache correctly ([#1547](#1547)) ([a923e11](a923e11)) * **CI:** Less flaky CI ([#1536](#1536)) ([2444b53](2444b53)) * **configs:** Make genesis fields optional ([#1555](#1555)) ([2d0ef46](2d0ef46)) * contract verifier config test ([#1583](#1583)) ([030d447](030d447)) * **contract-verifier-api:** permissive cors for contract verifier api server ([#1525](#1525)) ([423f4a7](423f4a7)) * **db:** Fix "values cache update task failed" panics ([#1561](#1561)) ([f7c5c14](f7c5c14)) * **en:** do not log error when whitelisted_tokens_for_aa is not supported ([#1600](#1600)) ([06c87f5](06c87f5)) * **en:** Fix DB pool for Postgres metrics on EN ([#1675](#1675)) ([c51ca91](c51ca91)) * **en:** improved tree recovery logs ([#1619](#1619)) ([ef12df7](ef12df7)) * **en:** Reduce amount of data in snapshot header ([#1528](#1528)) ([afa1cf1](afa1cf1)) * **eth-client:** Use local FeeHistory type ([#1552](#1552)) ([5a512e8](5a512e8)) * instruction count diff always N/A in VM perf comparison ([#1608](#1608)) ([c0f3104](c0f3104)) * **vm:** Fix storage oracle and estimation ([#1634](#1634)) ([932b14b](932b14b)) * **vm:** Increase log demuxer cycles on far calls ([#1575](#1575)) ([90eb9d8](90eb9d8)) ### Performance Improvements * **db:** rework "finalized" block SQL query ([#1524](#1524)) ([2b27290](2b27290)) * **merkle tree:** Manage indices / filters in RocksDB ([#1550](#1550)) ([6bbfa06](6bbfa06)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). --------- Co-authored-by: romanbrodetskiy <rb@matterlabs.dev>
ExternalIO contained unnecessary sleeps, which slowed down block processing.
## What ❔ - Adds boxed L2 client implementation and uses it in DI instead of a specific HTTP-based client. - Adds a mock L2 client. - Uses a mock L1 and L2 clients to test EN lifecycle in Rust, namely that it can start and timely terminates when sent a SIGINT. ## Why ❔ - L2 client abstraction is better for the node framework. - We have a frequent issue of newly added managed tasks being non-responsive, which leads to unnecessary delays during EN shutdown (30s in the worst case). - Using TypeScript integration tests for this purpose is slower and provides less control. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. - [x] Spellcheck has been run via `zk spellcheck`. - [x] Linkcheck has been run via `zk linkcheck`. test(en): Add basic rust tests for EN
## What ❔ ZKSYNC_HOME has been removed from server. It's required for future integration with file based configs ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. - [ ] Spellcheck has been run via `zk spellcheck`. - [ ] Linkcheck has been run via `zk linkcheck`. --------- Signed-off-by: Danil <deniallugo@gmail.com> Co-authored-by: Alex Ostrovski <aov@matterlabs.dev>
## What ❔ Extract `zksync_core::block_reverter` to separate the crate in the `node` folder ## Why ❔ Part of the to "modularize the codebase" process, required for legolizer and publishing on crates.io. ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. - [ ] Spellcheck has been run via `zk spellcheck`. - [ ] Linkcheck has been run via `zk linkcheck`.
#1711) So far we used a simpler interface where statekeeper was provided the next miniblock only after it persisted the previous one. With the current statekeeper implementation it severely limited the throughput. I've updated consensus API to allow for storing certificates asynchronously to improve the throughput.
## What ❔ Dedups system requirements in External Node docs and makes them (hopefully) more clear ## Why ❔ Users are being confused by unclear system requirements mentioned in two different places in docs. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. - [x] Spellcheck has been run via `zk spellcheck`. - [x] Linkcheck has been run via `zk linkcheck`.
## What ❔ Adding miniblock sealing according to max payload size. ## Why ❔ Compliance with max payload size imposed in consensus modules. ## Notes * Building the payload incrementally isn’t feasible because the miniblock’s transactions are flushed to DB before being reconstructed to be encoded as the payload. So the payload size accumulator added to `MiniblockUpdates` is just a heuristic done by encoding every executed transaction only to find out the estimated encoding size. * `miniblock_max_payload_size` was added to `StateKeeperConfig`, decoupled from `max_payload_size` in `consensus::config::Config`. It is possible to only use the latter, or to make the latter define/override the former. The state keeper max size should be less than 1/2 the consensus max size, because it never rejects the latest transaction, and doesn’t account for some metadata fields. Actual value in that respect was not set yet, because I wasn’t sure about values used for different non-test environments, or if we even want to keep this approach. * `MiniblockMaxPayloadSizeSealer` was added, implementing `should_seal_miniblock()` alone (without `should_seal_l1_batch_unconditionally`, as in trait `IoSealCriteria`). This isn’t an ideal solution, so other ideas are welcome. The applied config are: * Statekeeper threshold: 1mb * Consensus threshold: 2.5mb --- Closing [BFT-417](https://linear.app/matterlabs/issue/BFT-417/limit-l2-block-size) ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. - [ ] Spellcheck has been run via `zk spellcheck`. - [ ] Linkcheck has been run via `zk linkcheck`.
StanislavBreadless
requested review from
RomanBrodetski,
perekopskiy,
Deniallugo and
popzxc
as code owners
StanislavBreadless
changed the title
kelemeno
approved these changes
Apr 18, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What ❔
Why ❔
Checklist
zk fmtandzk lint.zk spellcheck.zk linkcheck.