Archive photo posts and carousels with gallery-dl

[zachlatta]

Problem

Instagram feed posts (/p/, /tv/) were routed to yt-dlp, which cannot download photos — it fails with There is no video in this post. In production 354 of 407 /p/ captures failed (87%).

Worse, the viewer chose its default tab by type preference without checking status, so every one of those captures opened on a red "Archive Failed" page while a perfectly good screenshot sat one tab over.

orpheus-engine is the main consumer of those captures.

What this does

Adds a gallery-dl archiver — the image-world counterpart to yt-dlp. gallery-dl supports ~300 sites; this routes the post-shaped URLs of 15 of them: Instagram /p/ and /tv/, X/Twitter, Reddit, Tumblr, Bluesky, Flickr, Imgur, DeviantArt, ArtStation, Pixiv, Pinterest, Newgrounds, VSCO. Adding a site is one entry in galleryDLSites.

Profile and feed URLs are deliberately not routed — gallery-dl would happily download an entire account. Instagram reels stay on yt-dlp, which handles them well (~83%).

Output is a streamed ZIP: every downloaded file, gallery-dl's raw per-file metadata sidecars, and a normalized metadata.json (author, caption, date, likes, tags, per-file dimensions). A new viewer renders it as the original post, served from /gallery/:shortid/list and /gallery/:shortid/file/*.

Verified end-to-end against a real 10-slide Instagram carousel: all 10 images at full resolution, plus author, caption, hashtags and like count.

The rename

youtube → yt-dlp, so a stored artifact names the tool that produced it (alongside the new gallery-dl).

That string lives in ~100k rows, in permalinks handed out long ago, and in API clients — so the old name keeps resolving forever. Requests and River job args are normalized on the way in, lookups match both spellings, and a startup migration rewrites existing rows.

The migration moves archive_items and queued river_job args in one transaction. The cleanup worker pairs an item with its job by comparing those two values; renaming only the items would make every in-flight item look orphaned and get force-failed while its job was still queued and about to succeed.

Decisions worth reviewing

• No --sleep-request override. gallery-dl already waits a randomized 6–12s between Instagram API calls. Because that flag is a root config key, gallery-dl resolves it ahead of per-extractor settings — so setting it replaces those tuned defaults rather than acting as a floor, and any value below Instagram's own would make throttling more likely.
• Exit status is a bitmask, not an enum (4 extraction, 8 challenge, 16 auth, 64 no-extractor). Decoded as flags; one run can report several.
• Exit 0 does not imply files were written — "no results" also exits 0 — so success requires at least one downloaded media file.
• gallery-dl installs into the same Python env as yt-dlp: its default Instagram video path hands DASH manifests to yt-dlp as an importable module, and silently degrades to lower-quality pre-merged MP4 otherwise.

Incidental fixes

Found while building this; each is independently defensible:

• A failed archive no longer wins the default tab over a completed one.
• Screenshots that fell back to JPEG (pages over 16383px) were served as image/webp.
• itch inherited the 2-minute browser timeout; it now has its own.
• s3SeekableReader.Seek dropped its open body even when seeking to the current offset — one ranged GetObject per ~32KB for random-access reads.
• saveArchiveData could return without closing the archiver's reader, leaking the producing goroutine and its temp directory.
• An unreadable cookies file was log.Fatalf at startup. Losing logged-in archiving is bad; crash-looping the whole server is worse (this caused a ~6 minute outage in July).

Testing

43 new test cases: type normalization and legacy-permalink resolution, URL routing (including host-substring and query-string traps), metadata extraction from a real Instagram carousel sidecar, ZIP construction, exit-bitmask decoding, tab selection, and the viewer including path traversal, Range requests, and incomplete-archive rejection.

go build, go vet, gofmt clean; full suite green under -race.

Deploy note

Instagram posts additionally require YTDLP_COOKIES_FILE — gallery-dl gets redirected to the login page when logged out. Every other routed site works anonymously.