Skip to content

[Enhancement] Speed up advisory sync ingestion for growing OSV datasets#86

Merged
sonukapoor merged 1 commit intomainfrom
feature/issue-78-sync-performance
Apr 4, 2026
Merged

[Enhancement] Speed up advisory sync ingestion for growing OSV datasets#86
sonukapoor merged 1 commit intomainfrom
feature/issue-78-sync-performance

Conversation

@sonukapoor
Copy link
Copy Markdown
Collaborator

Summary

Improve local advisory DB sync performance by reducing SQLite write overhead during OSV dump ingestion.

Closes #78.

What Changed

  • reuse prepared SQLite statements across the advisory DB lifecycle
  • add a bulk advisory upsert path that writes all parsed advisories in a single outer transaction
  • parse OSV advisories first, then perform one coordinated DB write pass instead of one transactional upsert per record

Benchmark

Local benchmark on the same machine using the same command and OSV npm dump:

  • before: 87.53s real
  • after: 8.84s real
  • improvement: about 9.9x faster end-to-end

Command used:

authoritative
/usr/bin/time -p node dist/index.js advisories sync --output /tmp/advisories.db

Verification

  • npm test
  • npm run build

@sonukapoor sonukapoor merged commit 6528e03 into main Apr 4, 2026
4 checks passed
@sonukapoor sonukapoor deleted the feature/issue-78-sync-performance branch April 24, 2026 19:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Enhancement] Improve advisory sync performance as the OSV dump grows

1 participant