perf: optimize database refresh with drop-and-rebuild indexes by digizeph · Pull Request #130 · bgpkit/monocle

digizeph · 2026-06-30T04:42:28Z

Summary

Drop and rebuild indexes around bulk inserts (all 4 repositories)
Remove redundant idx_pfx2as_prefix_str and low-value idx_pfx2as_validation indexes; rewrite lookup_exact to use the BLOB range index
Fix PRAGMA restore bug: store() no longer toggles synchronous/journal_mode, preserving WAL/NORMAL connection defaults
Switch INSERT OR REPLACE → plain INSERT (tables cleared before insert)
Add db_refresh_bench example for measuring refresh with real data

Performance (real data, release build)

Repository	Rows	Before	After	Improvement
asinfo	121,463	988 ms	330 ms	67% faster
as2rel	910,280	2,230 ms	1,765 ms	21% faster
rpki	340,000	71 ms	66 ms	— (already fast)
pfx2as	1,628,039	8,631 ms	4,433 ms	49% faster

Query performance (pfx2as, 1.6M rows)

All queries well under the 1-second target:

lookup_exact("1.1.1.0/24")          90 ms
lookup_longest("1.1.1.128/32")     388 ms
lookup_covering("1.1.1.0/24")      210 ms
get_by_asn(13335)                    4 ms
validation_stats()                346 ms
record_count()                      0 ms

PRAGMA state

after open:    journal_mode=memory, synchronous=NORMAL
after store(): journal_mode=memory, synchronous=NORMAL
✓ preserved across refresh

Test coverage

8 new lookup_exact tests (IPv4/IPv6, multi-ASN, no-match, prefix-length distinction, consistency cross-check, error handling)
4 test_store_rebuilds_indexes_correctly tests (verify indexes exist after store)
3 test_store_idempotent_index_rebuild tests (no duplicate indexes on re-store)
All 361 tests pass; clippy + fmt clean

How to benchmark

# Fetch real data
curl -sL -o /tmp/monocle-bench/asninfo.jsonl http://spaces.bgpkit.org/broker/asninfo.jsonl
curl -sL -o /tmp/monocle-bench/as2rel.json.bz2 https://data.bgpkit.com/as2rel/as2rel-latest.json.bz2
curl -sL -o /tmp/monocle-bench/pfx2as.json.bz2 https://data.bgpkit.com/pfx2as/pfx2as-latest.json.bz2

# Run
cargo run --example db_refresh_bench --features lib --release

Full investigation details in docs/db-refresh-perf-investigation.md.

Three optimizations cut database refresh (bulk insert) time by 21–49% across all repositories: 1. Drop and rebuild indexes around bulk inserts — indexes are dropped before the insert loop and rebuilt in one efficient pass afterward, instead of being updated per-row. This is the primary win. 2. Remove redundant pfx2as indexes — idx_pfx2as_prefix_str (redundant with the BLOB range index) and idx_pfx2as_validation (3-value enum not worth indexing). lookup_exact rewritten to use prefix_start/ prefix_end/prefix_length BLOB range index. 3. Fix PRAGMA restore bug — store() methods no longer toggle synchronous/journal_mode PRAGMAs, preserving the connection's WAL/NORMAL defaults. Previously every refresh left the connection in DELETE/FULL mode, degrading query performance until restart. Also switches INSERT OR REPLACE to plain INSERT (tables are cleared before insert) and adds a db_refresh_bench example for measuring refresh performance with real production data. Performance (real data, release build): asinfo: 988ms → 330ms (67% faster) as2rel: 2230ms → 1765ms (21% faster) pfx2as: 8631ms → 4433ms (49% faster) rpki: 71ms → 66ms (already fast) All queries remain well under 1 second on 1.6M rows.

Copilot

Pull request overview

Improves SQLite refresh performance in the database/monocle repositories by dropping/rebuilding indexes around bulk inserts, removing low-value pfx2as indexes (and updating lookup_exact accordingly), and adding a benchmark + investigation notes to quantify the impact.

Changes:

Drop indexes before bulk insert and rebuild after across ASInfo / AS2Rel / RPKI / Pfx2as refresh paths.
Remove idx_pfx2as_prefix_str + idx_pfx2as_validation and rewrite lookup_exact to use (prefix_start, prefix_end, prefix_length) matching.
Add db_refresh_bench example plus documentation and changelog entry for the performance work.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 13 comments.

Show a summary per file

File	Description
src/database/monocle/rpki.rs	Drops/rebuilds RPKI indexes around refresh; adds tests asserting indexes exist after store.
src/database/monocle/pfx2as.rs	Removes two indexes, updates `lookup_exact` to use BLOB/range columns, drops/rebuilds remaining indexes around refresh, adds tests.
src/database/monocle/asinfo.rs	Drops/rebuilds ASInfo indexes around JSONL refresh; switches to plain `INSERT`; adds tests for index rebuild.
src/database/monocle/as2rel.rs	Drops/rebuilds AS2Rel indexes around refresh; switches to plain `INSERT`; adds an index rebuild test (but not exercising `load_from_path`).
examples/db_refresh_bench.rs	Adds a refresh/query benchmark example (needs updates to match new `pfx2as` defaults and PRAGMA behavior).
docs/db-refresh-perf-investigation.md	Adds written investigation notes and before/after measurements.
CHANGELOG.md	Documents refresh performance improvements and PRAGMA behavior fix.
Cargo.toml	Registers the new `db_refresh_bench` example target.

Comments suppressed due to low confidence (2)

src/database/monocle/pfx2as.rs:285

Manual BEGIN ... / COMMIT means any early-return error before COMMIT will leave the connection inside an open transaction (potentially locking the DB for subsequent calls). Using a rusqlite::Transaction (e.g., unchecked_transaction()) would ensure an automatic rollback on drop when errors propagate.

        // Begin transaction - all changes are atomic, data remains accessible until commit
        self.conn.execute("BEGIN IMMEDIATE TRANSACTION", [])?;

src/database/monocle/rpki.rs:371

Manual BEGIN TRANSACTION / COMMIT can leave the connection in an open transaction if an error occurs before COMMIT (locking the DB for future operations). Prefer a rusqlite::Transaction so rollback happens automatically on error paths.

        // Begin transaction for batch insert
        self.conn.execute("BEGIN TRANSACTION", [])?;

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

- Move DROP INDEX + CREATE INDEX inside the transaction in all 4 repositories so refresh is atomic: either data+indexes commit together, or everything rolls back to the previous indexed state. - Switch pfx2as and rpki from manual BEGIN/COMMIT to unchecked_transaction() for automatic rollback on error. - Fix as2rel test to call load_from_path() instead of manually dropping/recreating indexes. - Update benchmark comments to reflect current code (3 indexes is the default, no PRAGMA toggles, lookup_exact uses BLOB range index).

Copilot

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.

digizeph · 2026-07-01T02:53:16Z

Addressed the review comments in f1dfd0f:

Kept index drop/clear/insert/reindex inside one transaction for atomic refreshes.
Dropped managed indexes before table clears to avoid per-row index maintenance during DELETE.
Updated CHANGELOG entry for the review fixes.

Validation run locally: cargo fmt, targeted index rebuild tests, and cargo check --all-features. Ready for re-review.

Copilot

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 6 comments.

digizeph · 2026-07-01T03:03:39Z

Addressed the latest review comments in a16b9a0:

Added removed legacy pfx2as indexes to the refresh drop list so upgraded databases shed them on the next refresh.
Added test coverage for removing legacy pfx2as indexes during store().
Gated db_refresh_bench tracing setup behind cfg(feature = "cli") so the example builds with --no-default-features --features lib.
Replaced refresh metadata INSERT OR REPLACE statements with plain INSERT after clearing meta tables.
Updated CHANGELOG.md.

Validation run locally:

cargo fmt
cargo check --no-default-features --features lib --example db_refresh_bench
cargo test --features lib store_rebuilds -- --nocapture
cargo test --features lib idempotent_index_rebuild -- --nocapture
cargo check --all-features

Ready for re-review.

Copilot

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

+        // Everything below is in one transaction so failures roll back to the
+        // previous state (with original indexes intact).
+        let tx = self
+            .conn
+            .unchecked_transaction()
+            .map_err(|e| anyhow!("Failed to begin transaction: {}", e))?;


+    /// Index creation SQL (rebuilt after bulk insert)
+    const INDEX_SQL: [&'static str; 2] = [
+        "CREATE INDEX IF NOT EXISTS idx_as2rel_asn1 ON as2rel(asn1)",
+        "CREATE INDEX IF NOT EXISTS idx_as2rel_asn2 ON as2rel(asn2)",
+    ];


Resolve merge conflicts between SSE service overhaul and db-refresh-optimization: Cargo.toml: keep database example from sse, add db_refresh_bench from main, remove stale ws_client_all reference (file deleted in sse overhaul). CHANGELOG.md: merge Breaking Changes (SSE) + Performance/Code Improvements (refresh) + New Features (SSE) under Unreleased changes. Review fix: remove unnecessary As2relSortOrder re-sort in AS2REL search endpoint that overrode the user's sort_by_asn flag. Review fix: remove misleading /rpki/aspa/validate endpoint from rpki module docs (deferred feature). Review fix: handle pfx2as refresh early in database_refresh with a clear message instead of returning a 500 InternalError from spawn_blocking. Review fix: install curl in Docker runtime image for healthcheck. Review fix: use constant-time byte comparison for bearer token auth to reduce timing side-channels. Review fix: update database README to note pfx2as HTTP API refresh is not yet implemented.

digizeph requested a review from Copilot June 30, 2026 15:46

Copilot started reviewing on behalf of digizeph June 30, 2026 15:46 View session

Copilot AI reviewed Jun 30, 2026

View reviewed changes

digizeph requested a review from Copilot July 1, 2026 02:44

Copilot started reviewing on behalf of digizeph July 1, 2026 02:45 View session

Copilot AI reviewed Jul 1, 2026

View reviewed changes

Comment thread src/database/monocle/asinfo.rs Outdated

Comment thread src/database/monocle/as2rel.rs Outdated

Comment thread src/database/monocle/rpki.rs Outdated

fix: address database refresh review comments

f1dfd0f

digizeph requested a review from Copilot July 1, 2026 02:53

Copilot started reviewing on behalf of digizeph July 1, 2026 02:53 View session

Copilot AI reviewed Jul 1, 2026

View reviewed changes

fix: clean up refresh metadata inserts

a16b9a0

digizeph requested a review from Copilot July 1, 2026 03:04

Copilot started reviewing on behalf of digizeph July 1, 2026 03:04 View session

Copilot AI reviewed Jul 1, 2026

View reviewed changes

fix: address remaining refresh review comments

7703000

digizeph merged commit 1d3b6e3 into main Jul 1, 2026
1 check passed

digizeph deleted the perf/db-refresh-optimization branch July 1, 2026 03:13

Uh oh!

Conversation

digizeph commented Jun 30, 2026

Summary

Performance (real data, release build)

Query performance (pfx2as, 1.6M rows)

PRAGMA state

Test coverage

How to benchmark

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

digizeph commented Jul 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

digizeph commented Jul 1, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

digizeph commented Jul 1, 2026 •

edited

Loading