feat(api): introduce mempool cache #1460

RomanBrodetski · 2024-03-19T15:59:48Z

What ❔

This introduces an in-memory cache on ws API servers that caters for eth_newPendingTransactionFilter filter. Currently for each eth_getFilterChanges request to such filter, we do a DB query. This query dominates the DB load of ws DBs.

The cache is updated preiodically, loading the new mempool transactions to a Deque. To return all the transactions that happened after a particular timestamp, we binsearch this data strucute - avoiding additional DB calls.

Why ❔

to relieve some load from the read replica DB.

core/lib/state/src/cache/lru_cache.rs

…nsactionFilter` filter

…anges

popzxc

Looks good. A few minor nits, feel free to ignore.

core/lib/state/src/cache/sequential_cache.rs

…anges

core/lib/dal/sqlx-data.json

core/lib/state/src/cache/sequential_cache.rs

core/lib/state/src/mempool_cache/mod.rs

core/lib/zksync_core/src/api_server/web3/namespaces/eth.rs

core/tests/ts-integration/tests/api/web3.test.ts

…anges

🤖 I have created a release *beep* *boop* --- ## [22.0.0](core-v21.1.0...core-v22.0.0) (2024-03-21) ### ⚠ BREAKING CHANGES * Use protocol version v22 as latest ([#1432](#1432)) ### Features * add docs for Dal ([#1273](#1273)) ([66ceb0b](66ceb0b)) * adds debug_traceBlockByNumber.callFlatTracer ([#1413](#1413)) ([d2a5e36](d2a5e36)) * **api:** introduce mempool cache ([#1460](#1460)) ([c5d6c4b](c5d6c4b)) * **commitment-generator:** `events_queue` shadow mode ([#1138](#1138)) ([9bb47fa](9bb47fa)) * **contract-verifier:** Allow sc code reverification ([#1455](#1455)) ([5a37b42](5a37b42)) * **database:** add an optional master replica max connections settings ([#1472](#1472)) ([e5c8127](e5c8127)) * **db:** Configurable maximum number of open RocksDB files ([#1401](#1401)) ([b00c052](b00c052)) * **en:** Check recipient contract and function selector in consistency checker ([#1367](#1367)) ([ea5c684](ea5c684)) * Follow-up for DAL split ([#1464](#1464)) ([c072288](c072288)) * **node_framework:** Ergonomic improvements ([#1453](#1453)) ([09b6887](09b6887)) * **node_framework:** Only store each wiring layer once ([#1468](#1468)) ([4a393dc](4a393dc)) * **node_framework:** Support for preconditions and oneshot tasks ([#1398](#1398)) ([65ea881](65ea881)) * **node-framework:** Add eth sender layer ([#1390](#1390)) ([0affdf8](0affdf8)) * **node-framework:** Add housekeeper layer ([#1409](#1409)) ([702e739](702e739)) * Remove batch dry_run ([#1076](#1076)) ([b82d093](b82d093)) * Separate Prover and Server DAL ([#1334](#1334)) ([103a56b](103a56b)) * **storage:** Make the storage caches task cancellation aware ([#1430](#1430)) ([ab532bb](ab532bb)) * support running consensus from snapshot (BFT-418) ([#1429](#1429)) ([f9f4d38](f9f4d38)) * **tx-sender:** Limit concurrent tx submissions ([#1473](#1473)) ([4bdf3ca](4bdf3ca)) * Use protocol version v22 as latest ([#1432](#1432)) ([1757412](1757412)) * **vm:** Prestate tracer implementation ([#1306](#1306)) ([c36be65](c36be65)) ### Bug Fixes * **core:** drop correct index of proof_generation_details table during database migration ([#1199](#1199)) ([76a6821](76a6821)) * **dal:** correct `tx_index_in_l1_batch` in `get_vm_events_for_l1_batch` ([#1463](#1463)) ([8bf37ac](8bf37ac)) * **node_framework:** Fix main node example ([#1470](#1470)) ([ac4a744](ac4a744)) * **protocol:** Remove verifier address from protocol upgrade ([#1443](#1443)) ([90dee73](90dee73)) * **reth:** use reth instead of geth ([#1410](#1410)) ([bd98dc7](bd98dc7)) * **verified sources fetcher:** Use correct `contact_name` for `SolSingleFile` ([#1433](#1433)) ([0764227](0764227)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

This introduces an in-memory cache on ws API servers that caters for `eth_newPendingTransactionFilter` filter. Currently for each `eth_getFilterChanges` request to such filter, we do a DB query. This query dominates the DB load of ws DBs. The cache is updated preiodically, loading the new mempool transactions to a `Deque`. To return all the transactions that happened after a particular timestamp, we binsearch this data strucute - avoiding additional DB calls. to relieve some load from the read replica DB.

🤖 I have created a release *beep* *boop* --- ## [22.0.0](core-v21.1.0...core-v22.0.0) (2024-03-21) ### ⚠ BREAKING CHANGES * Use protocol version v22 as latest ([#1432](#1432)) ### Features * add docs for Dal ([#1273](#1273)) ([66ceb0b](66ceb0b)) * adds debug_traceBlockByNumber.callFlatTracer ([#1413](#1413)) ([d2a5e36](d2a5e36)) * **api:** introduce mempool cache ([#1460](#1460)) ([c5d6c4b](c5d6c4b)) * **commitment-generator:** `events_queue` shadow mode ([#1138](#1138)) ([9bb47fa](9bb47fa)) * **contract-verifier:** Allow sc code reverification ([#1455](#1455)) ([5a37b42](5a37b42)) * **database:** add an optional master replica max connections settings ([#1472](#1472)) ([e5c8127](e5c8127)) * **db:** Configurable maximum number of open RocksDB files ([#1401](#1401)) ([b00c052](b00c052)) * **en:** Check recipient contract and function selector in consistency checker ([#1367](#1367)) ([ea5c684](ea5c684)) * Follow-up for DAL split ([#1464](#1464)) ([c072288](c072288)) * **node_framework:** Ergonomic improvements ([#1453](#1453)) ([09b6887](09b6887)) * **node_framework:** Only store each wiring layer once ([#1468](#1468)) ([4a393dc](4a393dc)) * **node_framework:** Support for preconditions and oneshot tasks ([#1398](#1398)) ([65ea881](65ea881)) * **node-framework:** Add eth sender layer ([#1390](#1390)) ([0affdf8](0affdf8)) * **node-framework:** Add housekeeper layer ([#1409](#1409)) ([702e739](702e739)) * Remove batch dry_run ([#1076](#1076)) ([b82d093](b82d093)) * Separate Prover and Server DAL ([#1334](#1334)) ([103a56b](103a56b)) * **storage:** Make the storage caches task cancellation aware ([#1430](#1430)) ([ab532bb](ab532bb)) * support running consensus from snapshot (BFT-418) ([#1429](#1429)) ([f9f4d38](f9f4d38)) * **tx-sender:** Limit concurrent tx submissions ([#1473](#1473)) ([4bdf3ca](4bdf3ca)) * Use protocol version v22 as latest ([#1432](#1432)) ([1757412](1757412)) * **vm:** Prestate tracer implementation ([#1306](#1306)) ([c36be65](c36be65)) ### Bug Fixes * **core:** drop correct index of proof_generation_details table during database migration ([#1199](#1199)) ([76a6821](76a6821)) * **dal:** correct `tx_index_in_l1_batch` in `get_vm_events_for_l1_batch` ([#1463](#1463)) ([8bf37ac](8bf37ac)) * **node_framework:** Fix main node example ([#1470](#1470)) ([ac4a744](ac4a744)) * **protocol:** Remove verifier address from protocol upgrade ([#1443](#1443)) ([90dee73](90dee73)) * **reth:** use reth instead of geth ([#1410](#1410)) ([bd98dc7](bd98dc7)) * **verified sources fetcher:** Use correct `contact_name` for `SolSingleFile` ([#1433](#1433)) ([0764227](0764227)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

## What ❔ Small changes after #1460: - `MempoolCache` is moved to the core crate as per its domain. - Some minor things like dropping DB connections earlier. ## Why ❔ Marginally simplifies maintenance. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. - [x] Spellcheck has been run via `zk spellcheck`. - [x] Linkcheck has been run via `zk linkcheck`.

🤖 I have created a release *beep* *boop* --- ## [12.2.0](prover-v12.1.0...prover-v12.2.0) (2024-03-28) ### Features * **api:** introduce mempool cache ([#1460](#1460)) ([c5d6c4b](c5d6c4b)) * **commitment-generator:** `events_queue` shadow mode ([#1138](#1138)) ([9bb47fa](9bb47fa)) * Drop prover tables in core database ([#1436](#1436)) ([0d78122](0d78122)) * Follow-up for DAL split ([#1464](#1464)) ([c072288](c072288)) * **prover:** export prover traces through OTLP ([#1427](#1427)) ([16dce75](16dce75)) * **prover:** File-info tool to help prover debugging ([#1216](#1216)) ([9759907](9759907)) * Separate Prover and Server DAL ([#1334](#1334)) ([103a56b](103a56b)) * support running consensus from snapshot (BFT-418) ([#1429](#1429)) ([f9f4d38](f9f4d38)) ### Bug Fixes * **api:** Fix API server shutdown flow ([#1425](#1425)) ([780f6b0](780f6b0)) * **prover:** Remove FriProtocolVersionId ([#1510](#1510)) ([6aa51b0](6aa51b0)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: Roman Brodetski <rb@matterlabs.dev>

RomanBrodetski marked this pull request as draft March 19, 2024 15:59

RomanBrodetski commented Mar 19, 2024

View reviewed changes

core/lib/state/src/cache/lru_cache.rs Show resolved Hide resolved

feat(api): introduce mempool cache that serves the `eth_newPendingTra…

a020bb2

…nsactionFilter` filter

RomanBrodetski force-pushed the rb-pla-672-optimize-db-reads-for-getfilterchanges branch from e5bd818 to a020bb2 Compare March 19, 2024 16:56

RomanBrodetski added 6 commits March 19, 2024 17:56

Delete core/lib/dal/sqlx-data.json

2e9f53a

Delete prover/prover_dal/sqlx-data.json

379a1fd

minort

4d89021

minor

4aeee8b

minor

09f0ce4

Merge branch 'main' into rb-pla-672-optimize-db-reads-for-getfilterch…

eacc857

…anges

RomanBrodetski marked this pull request as ready for review March 19, 2024 17:47

RomanBrodetski requested review from popzxc and perekopskiy March 19, 2024 17:47

RomanBrodetski added 6 commits March 19, 2024 20:08

try to fix tests

a1db53b

try to fix tests

c97182a

fix tests

a3494b0

fix test

c296fbd

will this fix spelling?

fa14ea2

fix spelling

42e5bde

popzxc previously approved these changes Mar 20, 2024

View reviewed changes

core/lib/state/src/cache/sequential_cache.rs Outdated Show resolved Hide resolved

core/lib/state/src/cache/sequential_cache.rs Show resolved Hide resolved

core/lib/state/src/cache/sequential_cache.rs Outdated Show resolved Hide resolved

RomanBrodetski added 2 commits March 20, 2024 12:18

various fixes

d722881

fix tests

5680017

RomanBrodetski dismissed popzxc’s stale review via 5680017 March 20, 2024 11:29

RomanBrodetski added 2 commits March 20, 2024 12:31

fix tests

e083017

Merge branch 'main' into rb-pla-672-optimize-db-reads-for-getfilterch…

71e65d2

…anges

popzxc reviewed Mar 20, 2024

View reviewed changes

core/lib/dal/sqlx-data.json Outdated Show resolved Hide resolved

popzxc reviewed Mar 20, 2024

View reviewed changes

core/lib/state/src/cache/sequential_cache.rs Outdated Show resolved Hide resolved

perekopskiy reviewed Mar 20, 2024

View reviewed changes

core/lib/state/src/mempool_cache/mod.rs Outdated Show resolved Hide resolved

core/lib/zksync_core/src/api_server/web3/namespaces/eth.rs Show resolved Hide resolved

core/tests/ts-integration/tests/api/web3.test.ts Outdated Show resolved Hide resolved

RomanBrodetski added 2 commits March 21, 2024 09:59

Merge branch 'main' into rb-pla-672-optimize-db-reads-for-getfilterch…

7c1efce

…anges

code review comments

9a3279a

RomanBrodetski added 6 commits March 21, 2024 12:08

Delete core/lib/dal/sqlx-data.json

706301c

Delete prover/prover_dal/sqlx-data.json

c1f670b

fix compilation error

e46d3db

fix cargo build

7db9f93

fox

be70ebf

fix

d57d8c5

perekopskiy approved these changes Mar 21, 2024

View reviewed changes

RomanBrodetski added this pull request to the merge queue Mar 21, 2024

Merged via the queue into main with commit c5d6c4b Mar 21, 2024
36 checks passed

RomanBrodetski deleted the rb-pla-672-optimize-db-reads-for-getfilterchanges branch March 21, 2024 17:26

This was referenced Mar 21, 2024

chore(main): release prover 12.2.0 #1439

Merged

chore(main): release core 22.0.0 #1424

Merged

slowli mentioned this pull request Mar 25, 2024

refactor(api): Brush up after #1460 #1491

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(api): introduce mempool cache #1460

feat(api): introduce mempool cache #1460

RomanBrodetski commented Mar 19, 2024 •

edited

Loading

popzxc left a comment

feat(api): introduce mempool cache #1460

feat(api): introduce mempool cache #1460

Conversation

RomanBrodetski commented Mar 19, 2024 • edited Loading

What ❔

Why ❔

popzxc left a comment

Choose a reason for hiding this comment

RomanBrodetski commented Mar 19, 2024 •

edited

Loading