Skip to content

Re-implement local queue on Zeno v2 using sqlite - #243

Merged
CorentinB merged 6 commits into
internetarchive:dev/v2from
saveweb:v2/local-queue
Mar 7, 2025
Merged

CorentinB merged 6 commits into
internetarchive:dev/v2from
saveweb:v2/local-queue

Conversation

@yzqzss

@yzqzss yzqzss commented Mar 5, 2025

Copy link
Copy Markdown
Collaborator

Notes:

  • Uses a cgo-free SQLite client: https://github.com/ncruces/go-sqlite3
  • sqlc_model package is generated by sqlc
  • To ensure that seeds with the same URL.Value are only queued once (deduplication), I revertd the local seencheck part of 2a6535d to perform seencheck on top level seeds. (cc: @equals215)

Future Work:

  • We need a "globalStopper" that calls controler.Stop() when everything is finished.
  • Most of local queue code is Ctrl-CV from the hq package, maybe it/they could be simplified a bit.
  • The batchsize and workersize could to be fine-tuned for sqlite.

@yzqzss
yzqzss marked this pull request as ready for review March 7, 2025 10:58
@CorentinB
CorentinB merged commit 160d17a into internetarchive:dev/v2 Mar 7, 2025
@yzqzss
yzqzss deleted the v2/local-queue branch March 11, 2025 07:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants