A personal media archive for public Instagram, Facebook, and TikTok posts and galleries. Users can paste an individual post URL, find recent media by platform and username, or upload a device gallery of images, GIFs, and videos. The application uses deterministic command-line extractors (yt-dlp, then gallery-dl) and does not include browser automation.
apps/web— React and Vite galleryapps/api— Fastify REST API and OpenAPI documentationapps/worker— BullMQ download and media-processing workerpackages/shared— normalized schemas, types, and constantspackages/database— PostgreSQL schema and Drizzle clientpackages/extractors— isolated extractor adapterspackages/config— validated environment configuration
- Node.js 24+
- pnpm 11+
- PostgreSQL and Redis
yt-dlpandgallery-dlfor a locally-run workerffmpegfor locally-run API and worker media processing
The Docker setup provisions all runtime dependencies automatically.
Railway should be configured as four services in one project:
backend— deploys the repository root with the defaultDockerfile. The final image runs both the API and worker, which lets them share one persistent volume.web— deploys the repository root withRAILWAY_DOCKERFILE_PATH=/docker/web.Dockerfile.Postgres— add Railway's managed PostgreSQL service.Redis— add Railway's managed Redis service.
Create the database services first, then configure the backend variables. Use Railway reference variables for the service names if you choose different names:
DATABASE_URL=${{Postgres.DATABASE_URL}}
REDIS_URL=${{Redis.REDIS_URL}}
API_ACCESS_TOKEN=<your access password>
API_HOST=0.0.0.0
COOKIE_SECURE=true
PROFILE_DISCOVERY_TIMEOUT_MS=45000
# Use http://localhost:5173 until the web service has a generated domain.
WEB_ORIGIN=http://localhost:5173
MEDIA_ROOT=/data/media
CREDENTIALS_ROOT=/data/credentials
# Optional: override resource-aware collection concurrency (1-8). Automatic sizing caps at 2.
# COLLECTION_CONCURRENCY=2
MAX_ASSET_BYTES=104857600
MAX_COLLECTION_BYTES=524288000
MAX_MEDIA_STORAGE_BYTES=4294967296
MEDIA_RETENTION_TRIGGER_PERCENT=80
MEDIA_RETENTION_TARGET_PERCENT=70
VIDEO_MAX_DIMENSION=1280
Set the backend service's pre-deploy command to pnpm db:migrate, healthcheck path to /health, attach a Railway volume mounted at /data, and generate its public domain. Do not scale this volume-backed service horizontally.
On the web service, set API_PROXY_URL=https://${{backend.RAILWAY_PUBLIC_DOMAIN}}, set RAILWAY_DOCKERFILE_PATH=/docker/web.Dockerfile, and generate a public domain. The web service uses that backend URL as its server-side proxy target while browser requests stay on the web origin, allowing session cookies to remain first-party. Finally, change the backend's WEB_ORIGIN to https://${{web.RAILWAY_PUBLIC_DOMAIN}} and redeploy it. Railway provides PORT automatically; the API and web preview are configured to listen on it.
Railway's private networking supplies the database and Redis connections through the reference variables. Keep the /data volume attached for extraction workspace and platform credentials; with local storage, it also holds all downloaded media.
You can connect the GitHub repository to both application services so pushes to the selected branch redeploy them automatically. Set watch paths if desired; changes under packages/** should trigger both services.
cp .env.example .env
# Set API_ACCESS_TOKEN in .env to any non-empty password.
docker compose up --buildThe one-shot migrate service applies committed database migrations before the API and worker start.
Open the gallery at http://localhost:5173, sign in with API_ACCESS_TOKEN, and view API documentation at http://localhost:3000/docs. The API stores the token in an HTTP-only session cookie. Set COOKIE_SECURE=true when serving it over HTTPS. The web service proxies /api requests so the cookie remains first-party even though the API runs as a separate service.
Start PostgreSQL and Redis, install the extractor tools on your host, then run:
cp .env.example .env
pnpm install
pnpm db:migrate
pnpm devDownloaded files are stored under MEDIA_ROOT. Relative storage paths are resolved from the workspace root so the API and worker always share the same files. Collection jobs retry three times with exponential backoff, and failed jobs remain visible for diagnostics and manual retry.
Profile discovery uses gallery-dl in metadata-only mode and returns at most 24 items per request. Instagram discovery includes posts and reels, with Story Highlights available as an opt-in source; TikTok includes posts, and Facebook includes profile photos. Discovery metadata is fetched in bounded 24-item snapshots and cached in Redis for 10 minutes by default. When more media is available, an opaque continuation cursor loads and caches another snapshot without imposing a total browsing limit. Cold profile extraction is limited to one active and one queued lookup, runs at most one gallery-dl process at a time, and has a 45-second total deadline by default so it cannot starve normal API traffic. PROFILE_DISCOVERY_CACHE_TTL_SECONDS controls the cache lifetime, and PROFILE_DISCOVERY_TIMEOUT_MS controls the deadline up to a maximum of 90 seconds. Up to 100 selected items are accepted in one batch API request and their canonical post URLs are queued through the same collection pipeline and limits as manually pasted links.
Automatic collection can watch a platform profile and run an uncached metadata check every 15 minutes through 7 days. BullMQ stores each recurring schedule in Redis, while PostgreSQL remains the source of truth and restores schedules when the worker starts. Instagram posts and reels, TikTok posts, and Facebook profile photos are watched by default. Instagram and TikTok current stories, plus Instagram Story Highlights, can be explicitly included per watched profile; valid platform cookies are recommended because these endpoints require authentication more often. Checks run one at a time with rate limiting, use the latest 24 profile items, queue only source IDs that have not already been collected or scheduled, and refresh completed Highlights when their child-media revision changes. Failed profile checks use bounded exponential backoff and remain visible in the dashboard. The first check runs immediately when automatic collection is enabled.
POST /profile-archives starts a background archive and enables automatic collection in one request. POST /profile-archives/:id/retry, where id is the watched profile ID, restores a failed archive's schedule and requeues its current page. The archive follows the profile continuation cursor one bounded page at a time until all available media has been discovered, persists its progress in PostgreSQL, and resumes queued or interrupted runs when the worker starts. Archives and scheduled checks share the same deduplication path, so they can safely overlap. Profile discovery is paced across both queues within each worker process; run one worker replica when a globally strict upstream request interval is required. Stories mean stories that are currently available; expired stories cannot be archived.
Extraction defaults to a 100 MiB asset limit and a 500 MiB collection limit. Images are preserved exactly as downloaded, without resizing or re-encoding. Videos are downloaded at up to 720p when that source is available, then normalized to H.264/AAC with a maximum 1280px dimension and 30 FPS. An optimized video replaces its source only when it is smaller or the source exceeds the configured dimensions. A 480px JPEG derivative is generated for every newly collected or uploaded image and video; gallery cards and the media selector load that derivative, while the full asset is reserved for the modal and downloads. OPTIMIZATION_TIMEOUT_MS bounds each FFmpeg operation.
Manual uploads accept up to 100 images, GIFs, and videos as one gallery. Files are streamed rather than buffered in API memory and use the same MAX_ASSET_BYTES, MAX_COLLECTION_BYTES, storage backend, deletion path, and retention policy as collected media. The username and platform are optional; an omitted platform is labeled Manual upload, and an omitted username is grouped as Unknown username.
MAX_MEDIA_STORAGE_BYTES defaults to 4 GiB. After a collection completes, oldest media is removed when database-tracked usage exceeds MEDIA_RETENTION_TRIGGER_PERCENT (80% by default) until it reaches MEDIA_RETENTION_TARGET_PERCENT (70%). Retention is serialized independently from collection processing, so collections can run concurrently without overlapping quota decisions.
Collection concurrency is automatically sized when the worker starts. It budgets two CPU threads and 1.5 GiB per collection, reserves 1.5 GiB for the co-located API and worker overhead, and caps automatic concurrency at two to preserve capacity for the API and media streaming. The worker logs rounded detected limits and the selected concurrency at startup. Set COLLECTION_CONCURRENCY to a value from 1 through 8 only when an explicit override is needed. Metadata probing within each collection remains bounded by METADATA_CONCURRENCY.
Local volume storage remains the default. For a larger archive, the API and worker can store final media in a private S3-compatible bucket while continuing to use MEDIA_ROOT as temporary extraction space and CREDENTIALS_ROOT for platform credentials.
For a Railway Bucket, add its credentials to the backend service with Railway variable references:
MEDIA_STORAGE_DRIVER=s3
S3_BUCKET=${{media-bucket.BUCKET}}
S3_ENDPOINT=${{media-bucket.ENDPOINT}}
S3_REGION=${{media-bucket.REGION}}
S3_ACCESS_KEY_ID=${{media-bucket.ACCESS_KEY_ID}}
S3_SECRET_ACCESS_KEY=${{media-bucket.SECRET_ACCESS_KEY}}
S3_FORCE_PATH_STYLE=false
S3_PRESIGNED_URL_TTL_SECONDS=900
Replace media-bucket with the bucket service name. Object keys include the content hash for traceability and a unique ownership suffix so cleanup can never remove another asset's media. The API authenticates each media request, then redirects S3 previews and downloads to short-lived presigned URLs so media bytes bypass the web and backend proxy services. S3 previews are privately cached in the browser for slightly less than the presigned URL lifetime; downloads are not cached. Configure the bucket to allow the web origin with GET and HEAD CORS access; this preserves browser video-frame export. Local assets are served by the API with private, immutable browser caching. Assets created before thumbnails were introduced can be backfilled without recollecting them. After deploying this version, run the worker command below once; it is safe to rerun and supports both local and S3 storage. It logs individual FFmpeg/download failures and exits non-zero if any assets could not be processed.
railway ssh --service backend pnpm --filter @media-scraper/worker thumbnails:backfillAfter deploying and applying migrations, move existing local assets into the configured bucket with:
railway ssh --service backend pnpm --filter @media-scraper/worker storage:migrateThe migration uploads each object before atomically updating its database location and queuing the local copy for durable cleanup. Failed file/object cleanup and quota enforcement are retried by the worker's database-backed maintenance loop. Keep the volume attached afterward because extraction and credentials still use it. Railway Buckets do not currently provide lifecycle rules, so application retention remains enabled.
Open the relevant platform access panel in the gallery and either paste a Cookie request header or select a Netscape-format cookies.txt export. The input must contain the platform's authentication cookies:
- Instagram:
sessionid - Facebook:
c_userandxs - TikTok:
sid_tt
Each normalized credential is stored separately in a private Docker volume, mounted read-only by the worker, and never returned by the API or placed in queue payloads. A configured status only confirms that a credential is stored; platforms can invalidate sessions at any time.
Treat cookie files as passwords. Remove them from the application when they are no longer needed, and replace them after logging out, changing a password, or when the platform invalidates the session.
Only collect media you are authorized to access. Preserve attribution, source links, privacy, copyright, and platform terms.