Skip to content

Storage cleanup tools

Jump to bottom
Jens Luebeck edited this page Nov 11, 2026 · 3 revisions

What this page is

The procedure for the tools that measure and clean up storage: which one answers which question, how to run it against dev or production, and what to do with what it tells you.

It is the procedure. The design behind it — why authority runs documents to files, why deletion writes the tombstone before purging the payload — is in docs/backend-architecture.md in the repository. Read that if you want to know why; read this if you have to run something.

This page deliberately carries no current-state figures — no document counts, no storage totals. Any such number is wrong as soon as somebody uploads or deletes something, and a stale number in a runbook gets quoted. Where you would want one, the page names the command that prints it.

The one rule

A tool either reports or it acts, and you must know which one you are running before you run it.

Every tool here falls into one of four kinds:

Kind Writes? Examples
Surveyors Never report_gridfs_orphans.py, manage.py ownership_survey, snapshot_storage.py
Validators Never validate_project_lineage.py, verify_indexes.py
Sweepers Yes, with --execute sweep_gridfs_unreferenced.py, sweep_s3_unreferenced.py, cleanup_failed_upload_residue.py
Backfills / repairs Yes backfill_project_status.py, recover_deleted_version.py

Every sweeper is report-only by default. Adding --execute is the moment the run changes from a question into an action, and there is no tool here that deletes without it.

The admin page File Ownership & Orphans has no delete control at all, and that is deliberate rather than unfinished: the two production incidents behind this work were both a count exactly like the one on that page being believed and acted on. The page reports; a sweeper acts, from a snapshot somebody has reviewed.

Which tool answers which question

The question Run this
Are there orphaned GridFS files? How much storage is residue? report_gridfs_orphans.py
Which project owns each file? (the admin page's data) manage.py ownership_survey
How much storage is in each state, and which way is it going? Admin ▸ Statistics, backed by snapshot_storage.py
Is the version history internally consistent? validate_project_lineage.py
Did the storage I deleted actually come back? No; see How backups work
Which S3 objects can nothing ever ask for again? sweep_s3_unreferenced.py (report-only)
A failed upload left files behind cleanup_failed_upload_residue.py
An upload crashed and left a placeholder project clear_stale_uploads.py
What URLs must keep working if we rebuild? Admin ▸ Download Backups ▸ URL manifest

Running anything against a server

The scripts live in the checkout under scripts/diagnostics/ (read-only) and scripts/maintenance/ (they write), and the checkout is bind-mounted into the container at /srv. scripts/README.md in the repository lists them all. Run them inside the container, so they get the environment from config.sh:

sudo docker exec -w /srv amplicon-prod bash -c \
  'set -a; source /srv/caper/config.sh; set +a; \
   exec /opt/venv/bin/python /srv/scripts/<diagnostics|maintenance>/<script>.py <args>'

Substitute amplicon-dev for the dev box. Three things to know:

  • Production and dev share one DocumentDB cluster, so the database name alone identifies nothing. Every tool here takes --expect-db and refuses to run if DB_NAME says otherwise. Always pass it.
  • The dev application credential is scoped to caper-dev. Trying to read caper from the dev container raises OperationFailure: Authorization failure. Production measurements must run from the production container. This is a guard rail, not an obstacle.
  • Never print DB_URI_SECRET, and remember config.sh also holds mail keys. Redaction has to cover quoted forms, not just bare ones.

Tools that write refuse to run on dev or prod unless you hold the operator lock. Report-only runs don't need it.

Reading production is fine and needs no ceremony. Writing to it does — see below.

Reading a validator failure

/opt/venv/bin/python /srv/scripts/diagnostics/validate_project_lineage.py --expect-db caper

It prints one line per invariant with the full sentence it asserts, so you do not need a table of what I1, I7 and the rest mean here — the tool tells you. That is deliberate: a second copy of the list kept on this page would drift from the one in the code, and a list maintained in two places is the exact defect this whole area was dug out of. The same goes for the transition labels T1–T9 you will see in code comments: every one of them states the fact beside the label, and the definitions live in the version-history spec in the repository. An operator running these tools never needs to look one up.

Add --skip-gridfs for a fast structural run; it skips I12, I13 and I14, which are the ones that walk fs.files, and the summary line says how many were not checkable rather than silently passing them.

A clean production run reports every invariant ok, with none listed as not checkable. Anything else is worth reading in full before acting.

Dev normally reports a handful of failures against its lineage-exercise fixtures. That is expected; dev deliberately carries awkward historical states so they can be exercised somewhere that is not production.

What to do with the common ones:

  • I7 — a LIVE document listed in another chain's history. Two chains claim one document: its own page renders it as current while another project's history renders it as a past version. Worth investigating; it is also what produces the confusing "you are viewing an older version" banner on a page that is current.
  • I12 / I13 — documents and GridFS disagree. The metadata backlink is provenance, never authority. On disagreement the backlink is rebuilt from the documents; nothing deletes a file because its own metadata says it is orphaned.

Writing to production

Every write to production is a separate, explicit decision. Approval for one write is not approval for the next. When one is authorised, the procedure is four steps and skipping any of them has cost us before:

  1. Capture a state probe first, covering both what the write should change and what it must leave alone. Run it twice and diff the two runs — a probe that is not stable is measuring something else.
  2. Write the predicted numbers down before running. Which counts move, by how much, and which must not move.
  3. Stage it. Every tool here takes --limit N. Run a few, re-run the probe, check it against the predictions, then do the rest. A staged run whose prediction misses is a stop, not a retry.
  4. Keep the undo record. Sweepers write one with --undo-record; put it somewhere durable afterwards, not in a scratch directory.

Pin the reads to the primary

The cluster URI ends readPreference=secondaryPreferred, so an unpinned client reads a replica — including a script's own verification of what it just wrote. This is not intermittent: in 40 write-then-immediately-read trials on dev, 40 of 40 missed on the default preference and 0 of 40 missed pinned to PRIMARY.

For most writers this is a hazard rather than a fault, because their update_one filter re-asserts what the plan saw, so a stale plan is a no-op. It is a genuine fault for anything that writes a derived value: the retired rebuild_version_chains.py stored a digest computed from what it read, so a lagging replica made it record a state that no longer existed, and report success. If you write a new script that touches this cluster, pin it.

Sweeping S3

amprepo-private is a cache of prepared project downloads, not a tier of storage: the payload always exists in GridFS as well, and a deleted object costs one slow download, not data. Regeneration happens inside the request at about 21 MiB/s, so the largest cached project is roughly 277 seconds of a blocked worker against a 900-second limit — which is why nothing here expires objects by age or size, and why a sweep only ever touches objects nothing can ask for.

The bucket is shared by production, dev and individual developer laptops, and nothing has ever expired it. Two consequences:

  • Attribution needs a join against every database that writes to it. The script takes one --ids-file per database and --require-databases N refuses to run with fewer, because a database you forgot to query makes its objects look like garbage.
  • No id set is authoritative for the shared root, where production, dev and laptops all write with no key prefix. A developer prefix is different: it is written by exactly one machine, so that machine's ids are authoritative for it. --key-prefix scopes a sweep to those namespaces and can only narrow, so a scoped run cannot reach the root.

Objects are listed before the id files are read, deliberately: a project created in between then counts as referenced, which is the safe direction.

A sweep does not reduce the bill for 90 days. The bucket is versioned. Deleting an object writes a delete marker and turns the previous version noncurrent; a lifecycle rule expires noncurrent versions 90 days later. So the live namespace shrinks immediately and the billed footprint does not — at any given moment the bucket can be billing roughly twice what a plain object listing shows. Check both: list_objects_v2 for what is live, list_object_versions for what is billed. This is the same trap as the cluster volume, in a different service.

The upside of that window is that a sweep is reversible inside it, by removing the delete marker; after it, the object regenerates from GridFS on the next download anyway.

A note for developers: setting S3_FILE_DOWNLOADS=TRUE locally does not put your project payloads in AWS — those go to your own machine's MongoDB, like all GridFS. What reaches the shared bucket is only the prepared download. Setting S3_DOWNLOADS_BUCKET_PATH to something identifying yours is what makes it possible to clean up later; leaving it empty puts your objects in the one namespace nobody can attribute. S3_FILE_DOWNLOADS=FALSE works fine locally and needs no AWS credentials at all.

Health check

Five minutes, all read-only, all from inside the container:

# 1. Is the lineage consistent?  Expect: 0 failing.
python /srv/scripts/diagnostics/validate_project_lineage.py --expect-db caper

# 2. Is there any GridFS residue?  Expect: 100% owned, every residue bucket 0.
python /srv/scripts/diagnostics/report_gridfs_orphans.py --expect-db caper

Then look at Admin ▸ Statistics for the storage trend and Admin ▸ File Ownership & Orphans for the most recent survey.

What a healthy result looks like: every invariant ok with none listed as not checkable, and every residue bucket in the orphan report at zero with 100% of fs.files owned by a live document. Both tools print their own totals, so the numbers come from the run in front of you rather than from this page.

Gotchas that have cost time

  • A bare 403 from the public sites is not an outage. An AWS WAF with Bot Control fronts both load balancers and blocks curl's default user agent on page URLs; the allow-rule only covers /api/v1/. Health-check with curl localhost:8000/ inside the box, or send a browser user agent. A healthy ALB target plus a 403 means the WAF, not the application.
  • Deleting data does not shrink the cluster volume. Hundreds of gigabytes have been removed across both databases without VolumeBytesUsed moving. collStats free-list figures are worse than useless here — unusedStorageSize froze byte-identical across a multi-gigabyte write and drifted on quiet days. Read VolumeBytesUsed as described in How backups work, and expect the same behaviour from versioned S3 (see above).
  • A large payload can take longer to delete than a request may live (roughly 96.4 × GiB + 0.035 × files seconds against a 900-second worker timeout). Deletion now writes the tombstone first and purges afterwards, off the request thread, so this no longer leaves a half-deleted version — but a purge interrupted by the container going away resumes from ids recorded on the tombstone, so look for those rather than reaching for a global sweep.
  • private is not a boolean. It holds 'private', 'public' or 'hidden_public', with some legacy booleans still present. {'private': False} matches nothing, silently.
  • A field's name does not tell you what it holds. cnvkit directory holds both file paths and GridFS ids on production, because a version of the upload code that stopped writing ids there was deployed in March 2026 and everything written before that still has them. Read the current writer to form a hypothesis; count the values to settle it.

Clone this wiki locally