Skip to content

chore(infra): cut the Cloudflare Worker's D1 binding over to a fresh loopover database - #9502

Merged
JSONbored merged 1 commit into
mainfrom
chore/d1-cutover-loopover-database
Jul 27, 2026
Merged

chore(infra): cut the Cloudflare Worker's D1 binding over to a fresh loopover database#9502
JSONbored merged 1 commit into
mainfrom
chore/d1-cutover-loopover-database

Conversation

@JSONbored

@JSONbored JSONbored commented Jul 27, 2026

Copy link
Copy Markdown
Owner

Why

The Worker's D1 database (b2c79dd6-7771-4b1f-aa6b-085d5f3e9528, internally still named gittensory from before the 2026-07-14 repo/app rename) had grown to 8.96 GB of its 10 GB cap. #9459 found and fixed the dominant cause — contributor-decision-pack signal snapshots accumulating unbounded (6.3 GB / 71% of the file) — and I pruned the already-accumulated 18,336 stale rows directly against the live database.

That prune shrank the live size (8.96 GB → 2.24 GB, confirmed via wrangler d1 info), which was better than expected — but D1 does not run VACUUM (cloudflare/workerd#1618), so the file was never guaranteed to fully return to its logical size, and any future large one-off bloat has the same risk. Since a rename was overdue anyway (gittensoryloopover, matching the 2026-07-14 app rename), this cuts over to a fresh, correctly-named database rather than just living with the old one.

What I verified before opening this

  • Full 103-table row-count sweep, source (gittensory) vs new (loopover): every mismatch is explained by ordinary live-write drift on actively-written tables (webhook/audit/usage logs — the source database never stopped taking writes between the export and the comparison). One delta (github_rate_limit_observations, exactly −250,000) is the automatic hourly retention.prune job's per-table cap firing once on the source after the export — not data loss.
  • d1_migrations: 202/202 rows, same max id, on both databases.
  • contributor-decision-pack (the table fix(retention): dedupe contributor-decision-pack to latest-only, not a preserved series #9459 fixed): 213/213 rows on the new database, exactly one row per contributor.
  • 206 oversized rows (payload_json large enough to exceed D1's ~100KB single-statement text limit, SQLITE_TOOBIG) were re-imported via a chunked INSERT + payload_json ||= UPDATE sequence instead of a raw literal insert. I verified this chunking byte-identical against the source data via a local SQLite reconstruction test before running it against the real database.
  • cf-typegen:check: unaffected — the binding name/shape is unchanged, only the database_id.
  • typecheck: clean.
  • Grepped the whole repo: wrangler.jsonc is the only file referencing the old database_id.

What this does NOT do

  • Does not delete the old database. It stays intact and readable until this is confirmed working in production.
  • Does not touch the ORB server (edge-nl-01) — that deployment uses its own Postgres, unrelated to this Cloudflare D1.
  • Does not update CLOUDFLARE_D1_MONITOR_DATABASE_ID (the fix(selfhost): alert explicitly when the monitored D1 approaches its 10 GB cap #9458 size-alerting probe) — that's an env var on the self-host side, currently unset, and is a separate follow-up once this cutover is confirmed.

Deploy mechanics — please read before merging

This repo's Worker (loopover-api) deploys via Cloudflare Workers Builds, git-connected to main — merging this PR triggers a live production deploy pointing the Worker at the new database. There is a brief window during the deploy where in-flight webhook-relay writes could be affected. I'm holding this for an explicit go rather than merging it myself, given it's the highest-blast-radius change in this batch.

Refs #9435

…loopover database

The previous database (b2c79dd6-7771-4b1f-aa6b-085d5f3e9528, internally still named
"gittensory" from before the 2026-07-14 rename) had grown to 8.96 GB of its 10 GB cap.
D1 never runs VACUUM (cloudflare/workerd#1618), so the batch of now-pruned
contributor-decision-pack rows (#9459 -- 18,336 rows / ~6.3 GB deleted) freed pages
that get reused, not returned: the file was never going to shrink back down on its own.

Cuts DB over to a freshly created, correctly-named "loopover" database
(da7537cc-ab54-4dc1-8c38-2713f03f1130), populated from a verified export of the
pruned source database:

- Every table's row count matches the source exactly, except ordinary live-write
  drift on actively-written tables (webhook/audit/usage logs, which never stop
  changing between an export and a later comparison) -- confirmed via a full
  103-table sweep.
- d1_migrations matches exactly (202/202 rows, same max id).
- signal_snapshots' contributor-decision-pack (the table #9459 fixed) is 213/213,
  exactly one row per contributor, confirming the dedup carried over correctly.
- 206 rows whose payload_json exceeded D1's ~100KB single-statement text limit
  (SQLITE_TOOBIG) were re-imported via a chunked INSERT + payload_json ||= UPDATE
  sequence, verified byte-identical against the source before import.

No other file in the repo references the old database_id. cf-typegen is unaffected
(binding name/shape unchanged, only the id).

Refs #9435
@loopover-orb

loopover-orb Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Warning

⏸️ LoopOver review result - manual review recommended

Review updated: 2026-07-27 22:15:07 UTC

1 file · 1 AI reviewer · 1 blocker · CI green · clean

⏸️ Suggested Action - Manual Review

Review summary
This is a config-only change swapping the Cloudflare Worker's D1 database_id from the old bloated/mis-named database to a freshly provisioned and verified replacement, with database_name already 'loopover' and migrations_dir unchanged. The PR description documents a thorough verification process (row-count sweep, migration table parity, chunked re-import of oversized rows) which is exactly the kind of rigor a live data-cutover like this needs, and the diff itself is minimal and consistent with that intent. No code paths change; the only risk is entirely operational (did the cutover actually happen correctly), which is outside what the diff can prove but is well-supported by the stated verification work.

Nits — 4 non-blocking
  • The comment block in wrangler.jsonc:293-301 is quite long for an inline JSONC comment; consider moving the detailed verification narrative into the PR description/CHANGELOG only and keeping the inline comment to a one-line pointer plus issue references, matching the terser style of most other inline comments in this file.
  • No rollback note: if the new database_id turns out to have a problem post-deploy, it'd help future readers to record the old database_id somewhere retrievable (e.g. this PR link) rather than only in comment history — worth confirming it's captured somewhere durable.
  • Consider adding a short migration/runbook doc (or linking one) for future D1 cutovers so the row-count-sweep + chunked-reimport process in the description isn't lost to PR history.
  • If not already done, confirm the old database_id b2c79dd6-... is scheduled for decommission/deletion on a delay (not immediately) in case rollback is needed.

Concerns raised — review before merging

  • No linked issue detected: No closing reference or linked issue number was found in the PR metadata/body. — If this PR is intended to solve an issue, link it explicitly in the PR body.
📋 Copy for AI agents — paste into your coding agent
Fix the following blocker(s) from this PR review:

1. No linked issue detected: No closing reference or linked issue number was found in the PR metadata/body. — If this PR is intended to solve an issue, link it explicitly in the PR body.

Decision drivers

  • ❌ Code review — 1 blocker (1 reviewer)
  • ❌ Gate result — Blocking (Repo-configured hard blocker found.)
Context & advisory signals — never blocks the verdict
Signal Result Evidence
Linked issue ✅ No-issue rationale PR body explains why no issue is linked.
Related work ✅ No active overlap found No same-issue or scoped active PR overlap found.
Change scope ✅ 20/20 Low review scope from cached public metadata (no linked issue context).
Validation posture ✅ 25/25 PR body includes validation/test evidence.
Contributor workload ✅ 10/10 Author activity: 13 registered-repo PR(s), 13 merged, 306 issue(s).
Contributor context ✅ Confirmed Gittensor contributor JSONbored; Gittensor profile; 13 PR(s), 306 issue(s).
Improvement ℹ️ Insufficient signal risk: clean · value: insufficient-signal
Review context
  • Author: JSONbored
  • Role context: owner (maintainer lane)
  • Public audience mode: oss maintainer
  • Lane context: Repository is configured for direct PR review.
  • Public profile languages: Python, TypeScript, Ruby, Go, MDX, Shell, Solidity, JavaScript
  • Official Gittensor activity: 13 PR(s), 306 issue(s).
  • PR-specific overlap: none found.
Contributor next steps
  • Start here: Treat this as maintainer-lane context rather than normal contributor-lane activity.
Signal definitions
  • Related work = same linked issue, overlapping active PRs, or title/path similarity.
  • Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
  • Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
  • Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
  • Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.
🧪 Chat with LoopOver

Ask LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.

  • @loopover ask <question> answers contribution-quality Q&A with source citations and freshness.
  • @loopover chat <question> answers in natural prose from cached decision-pack facts via local inference (maintainer/collaborator; read-only).
  • A plain-language @loopover mention with a real question is routed to the closest matching read-only command automatically — no exact syntax required.

Full command reference: https://loopover.ai/docs/loopover-commands

🧪 Experimental — new and may change.

Decision record
  • action: hold · clause: missing_linked_issue
  • config: 829f02175ea5e915b20bdd3b518867c79f6eab272951b20f1ee746e42e22110b · pack: oss-anti-slop · ci: passed
  • record: b829e1fb15eda705ade408740280055c3227b8590eb5606b0d9be4fa6f547ad4 (schema v5, head aecc460)

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed


💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.

  • Re-run LoopOver review

@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@codecov

codecov Bot commented Jul 27, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 88.65%. Comparing base (6571a1b) to head (aecc460).
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #9502      +/-   ##
==========================================
- Coverage   89.55%   88.65%   -0.91%     
==========================================
  Files         843      843              
  Lines      110073   110073              
  Branches    26194    26194              
==========================================
- Hits        98573    97582     -991     
- Misses      10238    11520    +1282     
+ Partials     1262      971     -291     
Flag Coverage Δ
backend 93.62% <ø> (-1.65%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 3 files with indirect coverage changes

@loopover-orb loopover-orb Bot added the manual-review Gittensor contributor context label Jul 27, 2026
@JSONbored
JSONbored merged commit 0f6c6b8 into main Jul 27, 2026
7 checks passed
@JSONbored
JSONbored deleted the chore/d1-cutover-loopover-database branch July 27, 2026 23:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

manual-review Gittensor contributor context

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant