Skip to content

Off site backups

Jump to bottom
Jens Luebeck edited this page Nov 11, 2026 · 5 revisions

Why this page exists

Everything AmpliconRepository is backed up by lives in the same AWS account as the thing it protects. The DocumentDB cluster snapshots, the nightly caper.sqlite3 upload to s3://amprepo-backups, and the reference genome bucket are all one account and one region. That survives a dropped collection, a bad migration and a deleted instance. It does not survive an account-level event, and no rehearsal has ever tested one.

This page is the procedure for the copies that leave AWS. All of it is manual and infrequent by design — the site does not change often, and a copy on somebody's own disk is the only kind that survives losing the account.

DocumentDB snapshots cannot be exported. There is no start-export-task in the DocumentDB API — only RDS has one. A snapshot can be copied to another region and nowhere else. So the plan below does not try to move the 362 GiB of GridFS payload; it keeps the current version of every public project, plus enough metadata to rebuild around it.

What to take, and how often

What Size Where from How often
Current version of every public project varies the API (below) every few months
Metadata dump ~39 MiB Admin ▸ Download Backups every few months
caper.sqlite3 ~105 KB Admin ▸ Download Backups every few months
Project URL manifest a few hundred KB Admin ▸ Download Backups every few months
AmpliconArchitect reference builds 26.3 GiB scripts/maintenance/download_refs.py when they change

Put them somewhere that is not AWS. Anything works for the small three — they total well under 100 MB.

1. The three files from the admin menu

Log in as a staff user and open Admin ▸ Download Backups. Three buttons:

  • Accounts database — caper.sqlite3, sessionless and gzipped. Accounts, email and OAuth links, and the CMS pages. This is the only site data that is in no DocumentDB snapshot.
  • Metadata dump — every collection except fs.chunks: the whole catalogue of what exists, who owns it and what it is called, without the payload the snapshots already hold. Takes a minute or two to build.
  • Project URL manifest — one CSV row per URL that resolves to a project today, and what it resolves to.

Each download records who took it and what the totals were at the time, and the page shows how much has changed since. A copy whose age nobody knows is not a backup, which is the whole reason the page tracks it.

Verifying the metadata dump

The dump ships its own manifest, so the file you downloaded can be checked with the same command that checks the six-monthly one:

tar xzf metadata-caper-<stamp>.tar.gz
cd <your clone>/caper
python -m caper.dump_metadata --verify-only /path/to/metadata-caper-<stamp>

That re-reads every file, checks its sha256, and counts its lines against what the manifest claims. A corrupted file has the wrong hash; a truncated one has the right hash for fewer lines. Both are checked because they fail differently.

What the URL manifest is for

Rebuilding the site from re-uploaded projects mints new ids. Without this file, every link ever published — in a paper, in an email, in someone's notes — resolves to nothing, and there is no record of what it used to mean.

The columns that matter in a recovery:

  • url / identifier — what a published link contains.
  • project_id, project_name, version_chain_id, version_ordinal — which project and which version the old link meant.
  • current_version_id — the version that link should land on after a rebuild.
  • redirect_to_project — for tombstones. That target is also a new id after a rebuild, so these redirects have to be re-pointed rather than recreated.

The identifiers are generated from the resolver's own query definitions, not from a list maintained beside them, so a lookup added to get_one_project() cannot silently drop URLs from this file.

2. Current version of every public project, via the API

The payload is the part no download button can reasonably carry. Pull it with the public API instead — no credentials needed for public projects.

# 1. Every public project, with its id, name, visibility and versions.
curl -s https://ampliconrepository.org/api/v1/projects/ > projects.json

# 2. Resolve the ids you want into download URLs, in one call.
python - <<'EOF' > ids.json
import json
projects = json.load(open('projects.json'))
print(json.dumps({'ids': [p['id'] for p in projects
                          if p.get('visibility') == 'public']}))
EOF

curl -s -X POST https://ampliconrepository.org/api/v1/projects/download/ \
     -H "Content-Type: application/json" --data @ids.json > downloads.json

# 3. Fetch each one. download_url is pre-signed and expires, so do this now.
python - <<'EOF'
import json, subprocess
for row in json.load(open('downloads.json'))['downloads']:
    subprocess.run(['curl', '-sL', '-o', row['project_name'] + '.tar.gz',
                    row['download_url']], check=True)
EOF

downloads.json also carries a skipped list — ids that do not exist, have no archive, or are not accessible. Check it is empty, or know why it is not.

A token is only needed for private projects; add -H "Authorization: Token <token>" from Profile ▸ API token if you want those too.

Take the current version of each — that is the id in current_version_id in the URL manifest. Older versions are recoverable from the cluster snapshots and are not worth carrying off-site: they are 64% of production storage, measured 2026-09-02, and nobody browses them.

Keep projects.json beside the tarballs. Together with the URL manifest it is what tells a rebuild which tarball belongs at which old URL.

What this does not recover

State this plainly to yourself before relying on it:

  • Private and restricted projects. The API fetch is public-only.
  • Audit log and provenance, and the download counters.
  • Project membership lists. Accounts survive in the sqlite backup; who could see what does not.

Those are all in the metadata dump as data, so they can be read afterwards — they just would not come back automatically.

3. The reference builds

refs.ampliconrepository.org holds the AmpliconArchitect reference builds: GRCh37, GRCh38, GRCh38_viral, hg19 and mm10, indexed and plain — ten tarballs, 26.3 GiB, measured 2026-09-02. Every analysis this site has ever served was produced against them, and until 2 Sep 2026 they were in no backup. A verified copy has since been taken off AWS: all ten files downloaded and verified, 28,234,799,306 bytes, about twelve minutes on a home connection.

The bucket is public, so the script needs no credentials and runs from any machine:

python scripts/maintenance/download_refs.py --out ~/amplicon-refs

It checks each file against the md5sum published beside it in the bucket and skips anything that already matches, so an interrupted transfer resumes for the cost of hashing what is already there. To re-check an existing copy without downloading:

python scripts/maintenance/download_refs.py --out ~/amplicon-refs --verify-only

Exit status is non-zero if anything is missing or unverified.

Recovering from a total loss

Rough order, assuming a new AWS account and a fresh deployment:

  1. Stand up the site from the repository, and restore caper.sqlite3 so the accounts, email links and CMS pages come back.
  2. Re-upload the project tarballs. Each gets a new id.
  3. Use the URL manifest to map every old identifier to the project it named, and configure redirects from the old URLs to the new ones. Tombstone rows need their redirect_to_project re-pointed at the new id of the target.
  4. Restore the reference builds to a bucket the aggregator can reach.
  5. Read the metadata dump for anything that has to be reconstructed by hand: membership lists, featured flags, download history.

The manifest is what makes step 3 possible at all. Take it whenever you take the projects, and keep the two together.

Clone this wiki locally