Conversation
WalkthroughReplace the sqlite_web client with a new LocalDb API and LocalDbQuery framework, migrate CLI and common code to LocalDb/LocalDbError, add wasm-exported DB query operations and SQL assets, implement a wasm database sync flow and webapp DB playground, and update wasm-related dependencies. Changes
Sequence Diagram(s)sequenceDiagram
autonumber
participant UI as WebApp UI
participant RC as RaindexClient (wasm)
participant LQ as LocalDbQuery (wasm)
participant LD as LocalDb
participant RPC as HyperRpcClient
participant JS as JS DB Callback
UI->>RC: syncDatabase(dbCb, statusCb, contractAddress)
RC-->>UI: status "Starting database sync..."
RC->>LQ: fetch_all_tables(dbCb)
LQ->>JS: execute SQL (tables query)
JS-->>LQ: JSON result
LQ-->>RC: Table list
alt tables missing
RC->>LQ: create_tables(dbCb)
LQ->>JS: execute SQL (create)
JS-->>LQ: ack
RC->>RC: download_and_decompress_dump(...)
RC->>JS: execute SQL (dump)
end
RC->>LQ: fetch_last_synced_block(dbCb)
LQ->>JS: execute SQL (sync_status select)
JS-->>LQ: JSON result
LQ-->>RC: SyncStatusResponse[]
RC->>LD: LocalDb::new(chain_id, api_token)
LD->>RPC: eth_getLogs / latest block
RPC-->>LD: logs
LD-->>RC: events
RC->>LD: decode_events(events)
LD-->>RC: decoded events
RC->>LD: decoded_events_to_sql(...)
LD-->>RC: SQL string
RC->>JS: execute SQL (inserts/updates)
JS-->>RC: ack
RC-->>UI: status "Database sync complete."
sequenceDiagram
autonumber
participant Caller as wasm caller
participant LQ as LocalDbQuery
participant JS as JS DB Callback
Caller->>LQ: execute_query_json<T>(JS, SQL)
LQ->>JS: call with SQL
JS-->>LQ: Promise
LQ->>LQ: await
alt resolves JSON
LQ-->>Caller: T (deserialized)
else resolves non-JSON / text
LQ-->>Caller: InvalidResponse error
else rejects/throws
LQ-->>Caller: Callback/Promise/DbError
end
Estimated code review effort🎯 4 (Complex) | ⏱️ ~75 minutes Possibly related PRs
Suggested labels
Suggested reviewers
Pre-merge checks and finishing touches❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✨ Finishing touches
🧪 Generate unit tests (beta)
📜 Recent review detailsConfiguration used: CodeRabbit UI Review profile: ASSERTIVE Plan: Pro 📒 Files selected for processing (1)
🧰 Additional context used📓 Path-based instructions (3)crates/*/{src,tests,benches,examples}/**/*.rs📄 CodeRabbit inference engine (.github/copilot-instructions.md)
Files:
crates/**/*.rs📄 CodeRabbit inference engine (AGENTS.md)
Files:
**/*.rs📄 CodeRabbit inference engine (AGENTS.md)
Files:
🧬 Code graph analysis (1)crates/common/src/raindex_client/local_db/query/mod.rs (7)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (16)
🔇 Additional comments (2)
Comment |
There was a problem hiding this comment.
Should we have some test coverage in this module?
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (3)
crates/common/src/raindex_client/local_db/fetch.rs (3)
119-124: Avoid misordering on parse failure in sortingSorting with unwrap_or(0) pushes malformed block numbers to the front. Prefer putting invalids last and reduce repeated parsing.
Apply this diff:
- all_events.sort_by(|a, b| { - let block_a = extract_block_number_from_entry(a).unwrap_or(0); - let block_b = extract_block_number_from_entry(b).unwrap_or(0); - block_a.cmp(&block_b) - }); + all_events.sort_by(|a, b| { + let block_a = extract_block_number_from_entry(a).unwrap_or(u64::MAX); + let block_b = extract_block_number_from_entry(b).unwrap_or(u64::MAX); + block_a.cmp(&block_b) + });Optional: precompute keys once to avoid repeated parsing inside the comparator (store parsed block numbers alongside events, sort by the key, then rebuild).
139-165: Simplify timestamp fetch: stream directly into HashMap with try_collectYou can avoid the intermediate Vec<Result<..>> and a second collect.
- let concurrency = config.max_concurrent_blocks.max(1); - let results: Vec<Result<(u64, String), LocalDbError>> = - futures::stream::iter(block_numbers) - .map(|block_number| { - let client = self.client.clone(); - let max_attempts = config.max_retry_attempts; - async move { - let block_response = retry_with_attempts( - || client.get_block_by_number(block_number), - max_attempts, - ) - .await?; - - let block_data = - block_response.ok_or_else(|| LocalDbError::MissingField { - field: "result".to_string(), - })?; - - Ok((block_number, block_data.timestamp)) - } - }) - .buffer_unordered(concurrency) - .collect() - .await; - - results.into_iter().collect() + let concurrency = config.max_concurrent_blocks.max(1); + futures::stream::iter(block_numbers) + .map(|block_number| { + let client = self.client.clone(); + let max_attempts = config.max_retry_attempts; + async move { + let block_response = + retry_with_attempts(|| client.get_block_by_number(block_number), max_attempts).await?; + let block_data = block_response.ok_or_else(|| LocalDbError::MissingField { + field: "result".to_string(), + })?; + Ok::<(u64, String), LocalDbError>((block_number, block_data.timestamp)) + } + }) + .buffer_unordered(concurrency) + .try_collect() + .await
225-241: Retry policy: consider exponential backoff with jitter and configurabilityConstant backoff risks burst retries and herd effects. Prefer exponential backoff with jitter and make base delay configurable via FetchConfig.
Would you like a follow-up patch switching to backon’s exponential builder with jitter and adding a retry_delay field to FetchConfig?
♻️ Duplicate comments (2)
crates/common/src/raindex_client/local_db/sync.rs (1)
108-111: Hard-coded RPC token breaks every sync
LocalDb::newneeds a valid HyperSync token; wiring in"41e50e69-6da4-4462-b70e-c7b5e7b70f05"guaranteesfetch_eventsfails (401/403) and the whole sync path returns an error. This isn’t just a secret-handling concern—it means the new API can never succeed in any environment. Please accept the token as a parameter (or load it from configuration) and thread the real value through.pub async fn sync_database( &self, db_callback: js_sys::Function, status_callback: js_sys::Function, contract_address: String, + api_token: String, ) -> Result<(), LocalDbError> { let orderbook_address = Address::from_str(&contract_address)?; @@ - let local_db = LocalDb::new( - orderbook_cfg.network.chain_id, - "41e50e69-6da4-4462-b70e-c7b5e7b70f05".to_string(), - )?; + let local_db = LocalDb::new(orderbook_cfg.network.chain_id, api_token)?;crates/common/src/raindex_client/local_db/fetch.rs (1)
219-223: Good fix: zero-attempts now returns a Config error (resolved prior review)Returning LocalDbError::Config for max_attempts == 0 is correct and clears the earlier misclassification.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: ASSERTIVE
Plan: Pro
⛔ Files ignored due to path filters (2)
package-lock.jsonis excluded by!**/package-lock.jsontauri-app/src-tauri/Cargo.lockis excluded by!**/*.lock
📒 Files selected for processing (9)
crates/cli/src/commands/local_db/fetch_events.rs(5 hunks)crates/common/src/raindex_client/local_db/fetch.rs(55 hunks)crates/common/src/raindex_client/local_db/mod.rs(1 hunks)crates/common/src/raindex_client/local_db/query/mod.rs(1 hunks)crates/common/src/raindex_client/local_db/sync.rs(1 hunks)crates/common/src/raindex_client/mod.rs(7 hunks)flake.nix(2 hunks)packages/webapp/package.json(1 hunks)packages/webapp/src/routes/db-playground/+page.svelte(1 hunks)
🧰 Additional context used
📓 Path-based instructions (6)
crates/*/{src,tests,benches,examples}/**/*.rs
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
Rust code must pass clippy with all warnings denied (cargo clippy --workspace --all-targets --all-features -D warnings)
Files:
crates/common/src/raindex_client/local_db/mod.rscrates/cli/src/commands/local_db/fetch_events.rscrates/common/src/raindex_client/local_db/query/mod.rscrates/common/src/raindex_client/mod.rscrates/common/src/raindex_client/local_db/sync.rscrates/common/src/raindex_client/local_db/fetch.rs
crates/**/*.rs
📄 CodeRabbit inference engine (AGENTS.md)
Rust code lives in the workspace under crates/* (e.g., cli, common, bindings, js_api, quote, subgraph, settings, math, integration_tests)
Files:
crates/common/src/raindex_client/local_db/mod.rscrates/cli/src/commands/local_db/fetch_events.rscrates/common/src/raindex_client/local_db/query/mod.rscrates/common/src/raindex_client/mod.rscrates/common/src/raindex_client/local_db/sync.rscrates/common/src/raindex_client/local_db/fetch.rs
**/*.rs
📄 CodeRabbit inference engine (AGENTS.md)
**/*.rs: Format Rust with nix develop -c cargo fmt --all
Lint Rust with nix develop -c rainix-rs-static (preconfigured flags)
Use snake_case for Rust crates/modules and PascalCase for types
Files:
crates/common/src/raindex_client/local_db/mod.rscrates/cli/src/commands/local_db/fetch_events.rscrates/common/src/raindex_client/local_db/query/mod.rscrates/common/src/raindex_client/mod.rscrates/common/src/raindex_client/local_db/sync.rscrates/common/src/raindex_client/local_db/fetch.rs
packages/**
📄 CodeRabbit inference engine (AGENTS.md)
JavaScript/Svelte projects live under packages/* (webapp, ui-components, orderbook)
Files:
packages/webapp/package.jsonpackages/webapp/src/routes/db-playground/+page.svelte
packages/webapp/**/*.{svelte,ts}
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
Webapp must pass Svelte lint/format checks (npm run svelte-lint-format-check -w @rainlanguage/webapp)
Files:
packages/webapp/src/routes/db-playground/+page.svelte
packages/{webapp,ui-components}/**/*.{svelte,ts}
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
If you modify frontend code, you must include a screenshot of the built webapp reflecting the change
Files:
packages/webapp/src/routes/db-playground/+page.svelte
🧠 Learnings (6)
📚 Learning: 2025-10-06T08:05:50.440Z
Learnt from: CR
PR: rainlanguage/rain.orderbook#0
File: AGENTS.md:0-0
Timestamp: 2025-10-06T08:05:50.440Z
Learning: Tauri: nix develop .#tauri-shell --command cargo tauri dev
Applied to files:
flake.nix
📚 Learning: 2025-10-06T08:05:50.440Z
Learnt from: CR
PR: rainlanguage/rain.orderbook#0
File: AGENTS.md:0-0
Timestamp: 2025-10-06T08:05:50.440Z
Learning: Rust: build with nix develop -c cargo build --workspace; test with nix develop -c cargo test
Applied to files:
flake.nix
📚 Learning: 2025-10-06T08:05:50.440Z
Learnt from: CR
PR: rainlanguage/rain.orderbook#0
File: AGENTS.md:0-0
Timestamp: 2025-10-06T08:05:50.440Z
Learning: Applies to {.env.example,packages/webapp/.env.example,tauri-app/.env.example} : Copy .env.example files (root, packages/webapp, tauri-app) and populate PUBLIC_WALLETCONNECT_PROJECT_ID / VITE_WALLETCONNECT_PROJECT_ID as required
Applied to files:
flake.nix
📚 Learning: 2025-10-06T06:07:57.188Z
Learnt from: findolor
PR: rainlanguage/rain.orderbook#2111
File: crates/common/src/raindex_client/mod.rs:13-13
Timestamp: 2025-10-06T06:07:57.188Z
Learning: In Rust, when a module is declared in the same file with `pub mod module_name;`, importing from that module using `use module_name::Item;` is valid and does not require the `self::` prefix. The order of `use` statements and `mod` declarations does not affect module resolution.
Applied to files:
crates/common/src/raindex_client/mod.rs
📚 Learning: 2025-08-01T07:35:13.418Z
Learnt from: brusherru
PR: rainlanguage/rain.orderbook#2036
File: crates/js_api/src/filters/errors.rs:39-46
Timestamp: 2025-08-01T07:35:13.418Z
Learning: In the Rain Orderbook project's PersistentFilterStoreError (crates/js_api/src/filters/errors.rs), the Display implementation already provides user-friendly error messages, so using err.to_string() for both msg and readable_msg in the WasmEncodedError conversion is appropriate and doesn't require separate readable message handling.
Applied to files:
crates/common/src/raindex_client/mod.rs
📚 Learning: 2025-07-09T07:42:48.458Z
Learnt from: findolor
PR: rainlanguage/rain.orderbook#1975
File: crates/js_api/src/config.rs:44-45
Timestamp: 2025-07-09T07:42:48.458Z
Learning: User findolor prefers to defer code improvements and refactoring suggestions to future PRs when they are not directly related to the current PR's scope, even when the suggestions are valid improvements.
Applied to files:
crates/common/src/raindex_client/local_db/sync.rs
🧬 Code graph analysis (6)
crates/common/src/raindex_client/local_db/mod.rs (4)
crates/common/src/raindex_client/local_db/decode.rs (3)
decode_events(95-160)U256(225-225)event(169-178)crates/common/src/raindex_client/local_db/insert.rs (2)
decoded_events_to_sql(84-142)value(31-31)crates/common/src/raindex_client/mod.rs (7)
from(208-212)from(286-288)from(292-294)from(407-409)from(413-418)to_readable_msg(298-403)new(106-124)crates/common/src/hyper_rpc.rs (2)
from(241-243)new(80-94)
crates/cli/src/commands/local_db/fetch_events.rs (1)
crates/common/src/raindex_client/local_db/mod.rs (1)
new(162-165)
crates/common/src/raindex_client/local_db/query/mod.rs (6)
crates/common/src/raindex_client/local_db/query/clear_tables/mod.rs (2)
clear_tables(6-10)clear_tables(17-21)crates/common/src/raindex_client/local_db/query/create_tables/mod.rs (1)
create_tables(20-24)crates/common/src/raindex_client/local_db/query/fetch_last_synced_block/mod.rs (2)
fetch_last_synced_block(18-22)LocalDbQuery(21-21)crates/common/src/raindex_client/local_db/query/update_last_synced_block/mod.rs (1)
update_last_synced_block(6-16)crates/common/src/raindex_client/local_db/query/fetch_tables/mod.rs (1)
LocalDbQuery(14-14)crates/common/src/raindex_client/local_db/mod.rs (1)
from(144-149)
crates/common/src/raindex_client/mod.rs (1)
crates/common/src/raindex_client/local_db/mod.rs (1)
from(144-149)
crates/common/src/raindex_client/local_db/sync.rs (5)
crates/common/src/raindex_client/local_db/query/fetch_last_synced_block/mod.rs (2)
LocalDbQuery(21-21)fetch_last_synced_block(18-22)crates/common/src/raindex_client/local_db/query/fetch_tables/mod.rs (2)
LocalDbQuery(14-14)fetch_all_tables(11-15)crates/common/src/raindex_client/local_db/mod.rs (1)
new(162-165)crates/common/src/raindex_client/mod.rs (2)
new(106-124)get_test_yaml(429-491)crates/common/src/raindex_client/local_db/query/mod.rs (3)
execute_query_text(65-90)create_success_callback(110-124)create_success_callback(344-358)
crates/common/src/raindex_client/local_db/fetch.rs (2)
crates/common/src/raindex_client/local_db/mod.rs (3)
new(162-165)invalid_block_number(153-158)new_with_client(172-174)crates/common/src/hyper_rpc.rs (1)
new(80-94)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (14)
- GitHub Check: standard-tests (ubuntu-latest, rainix-wasm-test)
- GitHub Check: test
- GitHub Check: git-clean
- GitHub Check: standard-tests (ubuntu-latest, rainix-sol-legal)
- GitHub Check: build-tauri (ubuntu-22.04, true)
- GitHub Check: standard-tests (ubuntu-latest, rainix-sol-artifacts)
- GitHub Check: standard-tests (ubuntu-latest, rainix-rs-static)
- GitHub Check: standard-tests (ubuntu-latest, rainix-sol-test)
- GitHub Check: standard-tests (ubuntu-latest, ob-rs-test, true)
- GitHub Check: standard-tests (ubuntu-latest, rainix-sol-static)
- GitHub Check: test
- GitHub Check: standard-tests (ubuntu-latest, rainix-rs-artifacts, true)
- GitHub Check: Deploy-Preview-Push
- GitHub Check: Deploy-Docs-Preview
🔇 Additional comments (5)
flake.nix (2)
48-50: Whitespace tidy acknowledgedTrimmed trailing space after the path only; behavior remains unchanged.
119-123: No-op formatting cleanupTrailing spaces removed from the copy command; logic unaffected.
packages/webapp/package.json (1)
25-25: tarball dependency artifact confirmed
The file packages/webapp/sqlite-web-0.1.0.tgz exists; install should succeed.crates/common/src/raindex_client/local_db/query/mod.rs (2)
12-28: LGTM!The error enum is well-structured with clear, descriptive variants covering all failure modes. The use of
#[from]forJsonErroris idiomatic.Based on the previous review clarification,
Erroris correctly in scope viause super::*;.
93-325: Excellent test coverage!The test suite is comprehensive and well-structured:
- Success cases (standard and empty results)
- All error variants (DatabaseError, JsonError, InvalidResponse, CallbackError, PromiseError)
- Edge cases (callback throws, promise rejection)
- Parameter passing verification
The tests provide strong confidence in the query execution logic.
| fn create_success_callback(json_data: &str) -> Function { | ||
| let success_result = WasmEncodedResult::Success::<String> { | ||
| value: json_data.to_string(), | ||
| error: None, | ||
| }; | ||
| let js_value = serde_wasm_bindgen::to_value(&success_result).unwrap(); | ||
|
|
||
| Function::new_no_args(&format!( | ||
| "return {}", | ||
| js_sys::JSON::stringify(&js_value) | ||
| .unwrap() | ||
| .as_string() | ||
| .unwrap() | ||
| )) | ||
| } |
There was a problem hiding this comment.
🧹 Nitpick | 🔵 Trivial
Eliminate duplicate test helper function.
create_success_callback is defined twice:
- Lines 110-124 (private, inside
wasm_testsmodule) - Lines 344-358 (public, at module level)
The implementations are identical.
Remove the private version and use the public one:
- fn create_success_callback(json_data: &str) -> Function {
- let success_result = WasmEncodedResult::Success::<String> {
- value: json_data.to_string(),
- error: None,
- };
- let js_value = serde_wasm_bindgen::to_value(&success_result).unwrap();
-
- Function::new_no_args(&format!(
- "return {}",
- js_sys::JSON::stringify(&js_value)
- .unwrap()
- .as_string()
- .unwrap()
- ))
- }Then update the tests to use super::create_success_callback instead.
Also applies to: 344-358
🤖 Prompt for AI Agents
In crates/common/src/raindex_client/local_db/query/mod.rs around lines 110-124,
there is a duplicate private create_success_callback function that is identical
to the public one at lines ~344-358; delete the private definition (lines
110-124) and update any tests in the wasm_tests module to call the existing
public function via super::create_success_callback instead (adjust
visibility/imports if necessary so tests reference the module-level function).
| const syncResult = await raindexClient.syncLocalDatabase( | ||
| queryFn, | ||
| () => {}, | ||
| '0x2f209e5b67A33B8fE96E28f24628dF6Da301c8eB' // contract address | ||
| ); |
There was a problem hiding this comment.
Auto-sync never succeeds without a valid RPC token
syncLocalDatabase fetches from HyperSync using the token baked into the WASM side ("41e50…"). That token is invalid, so this call always returns an error and the auto-sync loop is effectively dead on arrival. Please plumb a real token (or let users configure one) and pass it here; otherwise the feature can’t function.
🤖 Prompt for AI Agents
In packages/webapp/src/routes/db-playground/+page.svelte around lines 149 to
153, the call to raindexClient.syncLocalDatabase is failing because the WASM
side uses a hard-coded invalid RPC token; replace the baked token with a real
token or surface a user-configurable token and pass that token into the
syncLocalDatabase flow. Specifically, add a way to obtain a valid token (read
from environment/config, or a UI input saved in app state), propagate that token
into the raindexClient/WASM initialization or as an explicit parameter to
syncLocalDatabase, and ensure the token is forwarded into the HyperSync request
so the auto-sync loop can authenticate and succeed.
There was a problem hiding this comment.
this page is used for testing purposes and it's fine with this logic now
There was a problem hiding this comment.
@findolor, understood! I've noted that this page is for testing purposes and the current implementation is acceptable.
✏️ Learnings added
Learnt from: findolor
PR: rainlanguage/rain.orderbook#2111
File: packages/webapp/src/routes/db-playground/+page.svelte:149-153
Timestamp: 2025-10-06T10:23:57.444Z
Learning: The db-playground page at packages/webapp/src/routes/db-playground/+page.svelte is used for testing purposes, and production-level concerns (such as invalid RPC tokens) are acceptable in this context.
Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.
| } | ||
| } | ||
|
|
||
| // TODO: Write test for this function |
| use reqwest::Client; | ||
| use std::io::Read; | ||
|
|
||
| const DUMP_URL: &str = "https://raw.githubusercontent.com/rainlanguage/rain.strategies/46a8065a2ccbf49e9fc509cbc052cd497feb6bd5/local-db-dump.sql.gz"; |
| } | ||
| } | ||
|
|
||
| async fn download_and_decompress_dump() -> Result<String, LocalDbError> { |
There was a problem hiding this comment.
this fn takes no parameters and deals with hardcoded constants, very fragile
|
@coderabbitai assess this PR size classification for the totality of the PR with the following criterias and report it in your comment: S/M/L PR Classification Guidelines:This guide helps classify merged pull requests by effort and complexity rather than just line count. The goal is to assess the difficulty and scope of changes after they have been completed. Small (S)Characteristics:
Review Effort: Would have taken 5-10 minutes Examples:
Medium (M)Characteristics:
Review Effort: Would have taken 15-30 minutes Examples:
Large (L)Characteristics:
Review Effort: Would have taken 45+ minutes Examples:
Additional Factors to ConsiderWhen deciding between sizes, also consider:
Notes:
|
|
Tip For best results, initiate chat on the files or code changes. SIZE=L 🧠 Learnings used |
3 participants
Motivation
This PR adds a new function to the raindex client to sync local db data using the previously create fetch/decode/insert loop. We also update the db-playground route to test out the functionality of this new function. With this various queries are added to try out the syncing process.
Solution
Checks
By submitting this for review, I'm confirming I've done the following:
Summary by CodeRabbit
New Features
Bug Fixes
Chores
Style