Development

Development Guide

This document explains how GalleryVault is structured. The project is organized as a monorepo containing the backend (backend/: FastAPI JSON API + PostgreSQL), the frontend (frontend/: vanilla-JS static SPA served by nginx), and documentation. This file covers the backend.

Architecture overview

Browser ── :8000 ──▶ nginx (frontend/: static SPA, proxies /api,/login,/logout)
                          │
Browser ── :8001 ──▶ FastAPI app (backend/galleryvault/app/main.py + routers) ──▶ PostgreSQL
                          │
                    /api/* JSON routes (galleryvault.app.routers)

• Frontend (frontend/) is a build-free vanilla-JS SPA served by nginx on port 8000. nginx reverse-proxies /api, /login and /logout to the backend service (http://backend:8001).
• Backend (backend/) is a pure JSON API on port 8001 (container port 8001): no HTML pages, no static files.
• Database: PostgreSQL 18 runs alongside the backend in the root docker-compose.yml; its data persists in ./db-data (next to the compose file).

Key points:

• Authentication is cookie-based. The auth_and_csrf_middleware (galleryvault/app/middleware.py) returns 401 JSON for unauthenticated /api/* requests; the SPA detects the 401 and shows the login form. Login is performed by POST /login (form-encoded) which sets the session cookie, and POST /logout clears it. /healthz, /metrics, /login, /logout are exempt.
• CSRF: /api/ mutations check Origin → Referer → X-CSRF-Token (cookie galleryvault_csrf, set on GET, 30-day lax, readable by JS). The SPA (frontend/assets/core.js) sends X-CSRF-Token on POST/PUT/DELETE/PATCH. Sec-Fetch-Site: cross-site is rejected. HTML form POSTs still use the double-submit cookie.
• Observability: GET /metrics (exempt from auth) exposes Prometheus-style counters (requests, download/scan/tag-sync activity); every request carries a X-Request-ID correlation header (echoed in responses and structured logs), so a failing request can be traced across the nginx / backend / DB boundary.
• Favorites skip heuristic: FAVORITES_SKIP_LIMIT (default 5) — after five consecutive checks report an unchanged cloud folder count, the full re-scan is skipped (count-only checks continue); the first changed count re-enables full checks.

Project layout

galleryvault/          # Monorepo root
backend/
  galleryvault/
    app/
      main.py          # FastAPI application factory and assembly (<=100 lines)
      state.py         # AppState runtime container and service factories
      middleware.py    # Authentication and CSRF protection middleware
      lifespan.py      # Lifespan startup/shutdown and background workers
      dependencies.py  # FastAPI dependencies and injection helpers
      routers/         # route handlers split by domain
        auth.py core.py tasks.py settings.py downloads.py favorites.py
        galleries.py tags.py duplicates.py updates.py
    auth.py            # password hashing + session cookies
    config.py          # Settings model + DB persistence
    db/                # SQLAlchemy models, session, Unit of Work
      models.py repositories/  # galleries/favorites/downloads/updates/jobs/settings
      repository.py    # backward-compatible re-exports
    logging.py         # structured log formatter, ring buffer, httpx 2xx filter
    observability.py   # request-id middleware + /metrics counters
    secrets.py         # at-rest encryption (ENCRYPTION_KEY)
    scanners/          # ehviewer / zip / rar / folder scanners + registry
    services/
      downloader.py    # ExHentai download engine (concurrent pages, resume, max_pages, cancel)
      eh_client.py     # ExHentai HTML scraper (httpx) + GalleryGoneError + favorites paging/sizes
      favorites.py     # favorite-folder monitor + download queue (check-only when disabled)
      ingest.py        # metadata ingestion from scan results
      library.py       # filesystem scanning / expiry
      tag_sync.py      # per-gallery tag & category sync, category backfill
      tag_translation.py # EhTagTranslation database loading + translation lookups
      telegram.py      # TelegramNotifier (async client)
      telegram_bot.py  # long-poll bot for incoming commands
      thumbnails.py    # static JPEG thumbnail generation + on-disk cache
  alembic/             # database migrations (0001..0025)
  tests/               # pytest suite
  docs/                # this guide, API.md, openapi.json
  Dockerfile
  pyproject.toml
  requirements.txt

frontend/
  index.html           # SPA shell (hash-routed)
  assets/
    app.js             # vanilla-JS SPA bootstrap
    components.js      # reusable UI components
    core.js            # router & API client
    events.js          # global events & shortcuts
    i18n.js            # localization loader
    locales/           # translation dictionary files (en.js, zh.js)
    state.js           # reactive state management
    styles.css         # CSS tokens & styles
    utils.js           # helper functions
    views/             # modular view modules
  nginx.conf           # static serving + /api proxy to backend:8001
  Dockerfile

docs/wiki/             # Canonical bilingual wiki docs
docs-site/             # VitePress documentation site
docker-compose.yml     # Production compose
docker-compose.dev.yml # Development hot-reload compose

The frontend is intentionally build-free: app.js is plain ES2020 and styles.css is plain CSS, loaded directly by the browser. This keeps the project free of any Node toolchain and works fully offline.

Running locally (without Docker)

1. Create a virtualenv with Python 3.12 and install:

   cd backend
   python -m venv .venv && source .venv/bin/activate
   pip install -e ".[dev]"

2. Start PostgreSQL and export the connection string, e.g.:

   export DATABASE_URL=postgresql+asyncpg://galleryvault:galleryvault@localhost:5432/galleryvault
   export AUTH_SECRET=dev-secret
   export AUTH_PASSWORD_HASH=$(python -c "from galleryvault.auth import hash_password; print(hash_password('changeme'))")

3. Apply migrations and run the API:

   alembic upgrade head
   uvicorn galleryvault.app.main:app --reload --port 8001

4. Serve the frontend separately from frontend/ in this monorepo, pointing /api, /login, /logout at http://localhost:8001. Prefer docker compose -f docker-compose.dev.yml up -d --build for hot reload.

Building the container

From the monorepo root (galleryvault/):

docker compose build
docker compose up -d

This builds backend/ and frontend/ images, starts PostgreSQL with persistence in ./db-data, and maps frontend to host port 8000 and backend to host port 8001.

Published images are residualblood/galleryvault-backend and residualblood/galleryvault-frontend. Runtime containers are authoritative: docker cp edits do not persist across recreation.

Testing

# inside the container (or a venv with the same DB configured)
python -m pytest tests -q -p no:cacheprovider

tests/test_auth.py exercises the auth gate, the JSON API contract, and the SPA fallback page. test_scanners.py, test_p1_services.py, and test_latest_requirements.py cover the library/scanning services. The tests run against a real PostgreSQL (the container's DATABASE_URL); the db_isolated autouse fixture stubs the DB-backed auth bootstrap and disables the background worker loops (including _thumbnail_worker_loop) so they don't interfere with the test event loop.

Database migrations

Schema changes go through Alembic:

alembic revision --autogenerate -m "describe change"
alembic upgrade head

Migrations are applied automatically on container boot (CMD runs alembic upgrade head before uvicorn), so docker compose pull && up -d is a complete upgrade. Notable migrations: 0009 added the category_refreshed_at column (one-time category backfill), 0010 merged other into misc and moved coordinate-less galleries to deleted, 0011 added download_tasks.max_pages so partial/sample downloads survive the worker, 0012 added favorite_items.file_size for exact cloud-size estimates, 0015 added pg_trgm title indexes, 0018 added favorite_items.thumb (cover URLs captured from the favorites listing), 0021 added the background_jobs table backing the thumbnail / tag-sync queues, and 0025 added gallery_metadata versioning columns (parent_gid, newer_gid, is_replaced) for multi-chapter update tracking.

Background job queues

Thumbnail generation and tag sync use a persistent queue (background_jobs, one row per (job_type, gallery_id) with a pending/claimed status) instead of an in-memory asyncio.Queue, so queued work survives a restart and a future multi-process deployment can claim safely. BackgroundJobsRepository exposes enqueue/enqueue_many (idempotent via a unique constraint), claim (FOR UPDATE SKIP LOCKED + a lease_until that expires a row back to pending when the claiming worker died — recovered at worker start by mark_stale), requeue (retry, bumping attempts; next_attempt_at defers), complete (deletes the row) and clear (used by the cancel API). Tag-sync network-failure retries count down against the persisted attempts column, so a restart does not reset a poisoned gallery's retry budget.

Related throughput improvements: download progress is batched to at most one download_tasks write every 20 pages / 5 s instead of once per page, the Chinese tag autocomplete (search_zh) and translation-table rebuilds run off the event loop, and the library scan's heavy work already runs in the threadpool via run_in_threadpool.

Thumbnails

services/thumbnails.py renders static JPEG thumbnails (max 240px wide) into a dedicated cache dir (thumbnail_cache_dir, default /gv-cache/thumbs, a volume mounted at /gv-cache) keyed by gallery id + page index. Animated formats become static first frames. Nothing is ever written into the gallery archives. A background worker (_thumbnail_worker_loop, 4 concurrent) claims galleries from background_jobs (seeded at boot and after each download) and generates every missing page; progress is exposed via GET /api/thumbs/status. Gallery cover art and the detail-page thumbnail grid load /api/galleries/{id}/thumb/{page} instead of the full-size page.

Favorites

• fetch_favorites walks ExHentai's next cursor (not page=), calling a progress callback with the walked count; per-folder check progress lives in favorites_check_state (see GET /api/favorites/check-status).
• A disabled folder runs check-only (monitor_only): it records items and fetches sizes but never downloads. Only enabled folders download.
• remember_many upserts in 500-row batches (a single INSERT for a folder with thousands of galleries exceeds the asyncpg parameter limit). A successful full check also prunes favorite_items rows for gids no longer in the cloud folder (unfavorited / expunged), keeping the recorded set in sync so the scheduled "cloud count unchanged" skip keeps working.
• cloud_size = local real size + fetched sizes of missing galleries (favorite_items.file_size, via _favorite_size_sync) + an average estimate for the unfetched tail.

Building & deploying

Prefer pulling published images (docker compose pull && docker compose up -d). To build locally from the monorepo root:

docker build -t residualblood/galleryvault-backend:latest ./backend
docker build -t residualblood/galleryvault-frontend:latest ./frontend
docker compose up -d backend frontend

git push to dev runs CI (test + lint + build-push :dev to Docker Hub). Runtime containers are authoritative: docker cp edits do not persist across recreation.

Tag translations

Implemented in the backend; the frontend only switches the display by language.

• The backend module galleryvault/services/tag_translation.py calls load_translations() at startup to load the translation database into the in-memory _TRANSLATIONS dict.
• Load order (later sources override earlier ones):
  1. The bundled galleryvault/data/tag_translations.json (~2.1 MB, exported from EhTagTranslation / ehsyringe in the {"data":[{"namespace","data":[{"key","name"}]}]} format).
  2. A user override file galleryvault/tag_translations.json (if present).
  3. An external file pointed to by the TAG_TRANSLATIONS_FILE environment variable (recommended under Docker, e.g. by mounting a host file).
  4. An explicit path argument (used by the unit tests).
• translate_tag() / translated_tag() query the in-memory table; on a hit, clean_display() strips any nested markdown icon syntax ![alt](https://...webp) the translation database may contain (69 such records, e.g. ![贝合图标](...tribadism.webp)贝合 → 贝合) and truncates the result to 60 characters.
• The API returns Chinese display fields in three places: GET /api/galleries per-item tags[], GET /api/galleries/{id} tags[], and GET /api/tags/search items[].
• The frontend frontend/assets/app.js tagText(tag) shows tag.display when localStorage.gv_lang==='zh' and tag.name otherwise; the namespace labels NAMESPACE_LABELS_ZH also come from the backend constants.

Syncing with EhTagTranslation / ehsyringe:

• Built-in auto-update (recommended): a backend background task parses the latest db.text.json asset from the EhTagTranslation/Database release every TAG_TRANSLATION_UPDATE_INTERVAL_MINUTES (default 720, 0 disables it) and hot-reloads it (load_translations(reset=True)). You can trigger it manually with Update now in Settings or check its status there. Note that db.text.json's data is a {name: zh} dict whose values are {name,intro,links} objects; merge_translation_data is compatible with that format and with the older {key,name} list / flat mapping as well.
• Manual mount (offline environments): put the downloaded JSON in galleryvault/data/tag_translations.json or mount it at TAG_TRANSLATIONS_FILE; rebuild to bundle it, or hot-load it via the env var.

# fetch the latest translation database manually
curl -L https://github.com/EhTagTranslation/Database/releases/latest/download/db.text.json \
  -o ./tag_translations.json
# docker-compose.yml:
#   environment:
#     TAG_TRANSLATIONS_FILE: /app/tag_translations.json
#   volumes:
#     - ./tag_translations.json:/app/tag_translations.json:ro
docker compose up -d backend

You can confirm loading in the container logs: docker logs galleryvault-backend | grep "tag translations".

Local Development

For zero-build development with live reloading for both frontend and backend:

cd galleryvault  # monorepo root
docker compose -f docker-compose.dev.yml up -d --build

• Frontend static assets are mounted live (./frontend/assets), browser refresh updates instantly;
• Backend runs with uvicorn --reload --reload-dir /app/galleryvault --proxy-headers --forwarded-allow-ips="172.16.0.0/12,127.0.0.1" (proxy flags ensure real client IP is resolved behind nginx for rate limiting and logging);
• Development database runs isolated on ./db-data-dev; host data directories ./library, ./downloads, and ./cache are mapped locally.

Conventions

• Keep route handlers in galleryvault/app/routers/ (one module per domain). Handlers reference state via app_state or FastAPI Depends providers. Production code directly imports source modules; tests use dependency overrides or app_state. Each handler is a plain async def.
• All errors returned to the client are HTTPException (or _db_error for SQLAlchemy failures). Never leak raw exception text – secrets/cookies are already scrubbed in logging.py.
• Do not commit cookies, .env, TEMP/*, or media/. (There is no longer a config.json; settings persist in the database.)

Production Log Monitoring & Analysis Scripts

The scripts/ directory provides command-line tools for inspecting and diagnosing production container logs:

1. scripts/monitor_prod_logs.sh

Remotely streams and collects logs from production containers (galleryvault-backend, galleryvault-frontend, galleryvault-db) over SSH, bundles them into a compressed archive, transfers them locally, and triggers automated analysis.

• Syntax:

  ./scripts/monitor_prod_logs.sh [duration]

• Arguments:
  • duration: Collection window duration (default 30m). Supports suffixes s, m, h (e.g. 10m, 1h).
• Environment variables:
  • PROD_HOST: Remote host SSH target (default: root@192.168.1.123).
  • CHECK_INTERVAL: Polling interval in seconds (default: adaptive, up to 300s).
• Execution flow:
  1. Generates and transfers a self-contained collector script to ${PROD_HOST}:/tmp/prod_collector.sh;
  2. Runs concurrent log streaming (docker logs -f) for backend, frontend, and db into /tmp/galleryvault-prod-logs/;
  3. Periodically monitors collector process health and archive disk usage;
  4. Automatically downloads the resulting tar.gz to local /tmp/galleryvault-prod-logs-<timestamp>/;
  5. Cleans up remote files and runs scripts/analyze_prod_logs.py on the extracted logs.

2. scripts/analyze_prod_logs.py

Parses extracted container log files (or .tar.gz log bundles) and generates a structured, colorized terminal report.

• Syntax:

  python3 scripts/analyze_prod_logs.py <log_dir_or_archive>

• Key Metrics Analyzed:
  • HTTP Status Code Breakdown: Total requests, 2xx/3xx/4xx/5xx counts, and error percentages;
  • Top Routes & Error Hotspots: Identifies high-traffic API routes and paths triggering 4xx/5xx responses;
  • Exception Clustering: Groups Python exception types (e.g. IntegrityError, TimeoutException, KeyError) with occurrence counts;
  • Component Health Overview: Segregates line counts, warnings, and errors across backend, frontend (nginx access/error), and db (PostgreSQL);
  • Structured Error Tracing: Highlights critical traceback entries with timestamps and logger names.