Skip to content

api-v6.0.0

Latest
Latest

Choose a tag to compare

View all tags
@pashidlos pashidlos released this 12 Nov 07:24
api-v6.0.0
1f0426b
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Backend 6.0.0: PostgreSQL 18 and Prisma 7, Node 26, a non-root image, a worker pool for image diffs, a typed error model, authenticated socket.io, and the rendering-noise hint.

This release covers the images visualregressiontracker/api:6.0.0, visualregressiontracker/migration:6.0.0 (tag migration-v6.0.0) and visualregressiontracker/dinov2:6.0.0 (tag dinov2-v6.0.0; optional service, compose profile dinov2).

Upgrade steps and the user-facing overview are in the umbrella release: https://github.com/Visual-Regression-Tracker/Visual-Regression-Tracker/releases/tag/6.0.0

Requires UI 6.0.0: https://github.com/Visual-Regression-Tracker/frontend/releases/tag/v6.0.0

⚠️ Breaking changes

Install and runtime

  • The bundled PostgreSQL moves to 18, which cannot read a PostgreSQL 12 data directory: dump and restore once (see the umbrella release). An external database is unaffected; 12 still works but is end of life, so move to 15+.
  • The api and migration images run as uid 1000 (USER 1000:1000). On Linux, chown -R 1000:1000 the imageUploads volume and any mounted TLS files, or the API stops with EACCES.
  • The image starts node dist/main under tini (ENTRYPOINT ["/usr/bin/tini", "--"]). Check this if you override command or entrypoint.
  • JWT_SECRET is required. Public sample values (jwtPrivateKey, the one in .env) stop startup when NODE_ENV=production, as in the Docker image. Everyone signs in again once; API keys keep working.
  • Tokens are resolved by sub only; email-only tokens from old releases are no longer accepted.
  • Node >=26 and Docker Compose v2.
  • UI and API ship together: UI 5.x gets no live updates from API 6.0 and misreads its bulk results.
  • DIFF_WORKERS_COUNT and DIFF_QUEUE_LIMIT are no longer read; the pool size and queue are fixed (see Performance). DIFF_WORKER_MAX_HEAP_MB stays.

API

  • Errors are 4xx instead of 200/500 (see API changes). Check clients that treat any 2xx as success.
  • Uploads are stricter: non-PNG screenshots get 400, multipart uploads are capped at MULTIPART_FILE_SIZE_LIMIT, GET /images/:name refuses unexpected names with 400.
  • socket.io: connections need auth: { token } and must join a project or build room to receive events. There are no opt-out flags.
  • Removed endpoints: GET /test-variations/merge (use POST /test-variations/merge), PATCH /builds/:id/approve (use POST /test-runs/approve), POST /ollama/compare (GET /ollama/models stays), and the build noise-summary endpoint.
  • PATCH /builds/:id rejects unknown fields; PATCH /test-runs/update/:id accepts only comment. Unknown body fields are dropped everywhere (global ValidationPipe with whitelist).
  • POST /test-runs/approve|reject|delete answer 200 with a per-id result body instead of 201 with an empty body, and no longer stop at the first failing id.
  • apiKey is removed from GET /users/all and PATCH /users/assignRole responses.

Database

  • TLS to PostgreSQL is used only when DATABASE_URL asks for it (?sslmode=require). Without sslmode the connection is plain.
  • sslmode=prefer now behaves as require.
  • Prisma 7 ignores the pgbouncer, statement_cache_size, socket_timeout, sslidentity and sslpassword connection parameters.
  • A failing TLS probe at startup now stops the API instead of continuing.

Other

  • OIDC_REDIRECT_URI must not contain a query string.
  • Comparison results can change (see Comparison & noise).
  • noiseAnalysisConfig reads only blur.enabled and dinov2.enabled; threshold keys are accepted and ignored.

API changes

New

  • GET /projects/favorites, PUT|DELETE /projects/:id/favorite: per-user favourite projects.
  • POST /test-runs/revert: undo an approve or reject; 204, or 409 if the status changed since. Only the user who made the decision gets Undo; a decision made by someone else is reported as failed. Undoing a feature-branch approval restores that branch's ignore areas and comment.
  • POST /test-variations/merge with { projectId, fromBranch, toBranch }: merges branches, answers 201 with BuildDto.
  • GET /statistics/results: run counts per status for each build.
  • POST /test-runs/noise-analysis: queue runs for the noise hint (see Comparison & noise).
  • GET /dinov2/status: whether DINOv2 is configured.
  • GET /users/registration/enabled.
  • User summaries include createdAt.

Changed

  • Bulk POST /test-runs/approve|reject|delete answer 200 { succeeded: string[], failed: { id, message }[] }, ids as sent. An HTTP error or Prisma "record not found" for one id lands in failed; other errors (for example database down) still fail the request.
  • GET /test-variations returns variations in a stable order.
  • /projects responses mask the Gemini key as •••• plus its last 4 characters (fully masked below 12 characters). PUT /projects keeps the stored key when it receives a value starting with ••••.
  • GET /images/:name streams with Cache-Control: immutable.
  • /ollama/* requires an admin or editor login.
  • LDAP: an empty password is rejected, a directory outage answers 503, changePassword answers 400, PUT /users ignores the email in the body.
  • /health answers 503 without the database.

Error model

  • Body: { statusCode, path, name, message }; exception and stack are gone.
  • Validation failures return their reasons.
  • Statuses: 400 validation and non-PNG upload (also negative take/skip, duplicate records), 404 missing record, 409 conflict (foreign-key violation, run without a variation, undo after a status change), 413 body too large, 422 image over MAX_IMAGE_PIXELS, 503 diff queue full (with Retry-After).

socket.io

  • The handshake carries auth: { token }, checked like the HTTP JWT (signature, expiry, user still exists). A refusal is unauthorized; a failed user lookup (for example database down) uses a different message so clients do not log the user out.
  • Clients send join / leave with { projectId?, buildId? } to enter the project:<id> and build:<id> rooms (non-uuid ids are ignored; the ack returns the room names).
  • build_created, build_updated and build_deleted go to the project room; testRun_* go to the build room.
  • testRun_* payloads are TestRunDto; empty batches are not sent; build_created goes out once; every update is testRun_updated.
  • A socket is closed when its token expires or its user is deleted.
  • Pending event batches are dropped on shutdown.

Configuration

New, all optional:

Variable Default Purpose
REGISTRATION_ENABLED true false disables sign-up
LOG_LEVEL log,error,warn,fatal comma-separated Nest log levels
MAX_IMAGE_PIXELS 100 MP (0 = no limit) larger images get 422
MULTIPART_FILE_SIZE_LIMIT 100 MiB multipart upload cap
DIFF_WORKER_MAX_HEAP_MB 512 heap limit per diff worker
DINOV2_URL unset DINOv2 service, second noise stage; unset means blur only
DINOV2_TIMEOUT_MS 60000 timeout of one DINOv2 request
HTTPS_KEY_PATH, HTTPS_CERT_PATH unset now read by the API; when both files exist the server serves HTTPS
SERVER_HEADERS_TIMEOUT Node default HTTP headers timeout

DINOV2_MODEL_PATH, DINOV2_MODEL_ID, DINOV2_THREADS, DINOV2_REGIONS, DINOV2_MAX_QUEUE, DINOV2_MAX_PIXELS, DINOV2_MAX_BODY_BYTES and DINOV2_PATCH configure the dinov2 service itself.

Removed: DIFF_WORKERS_COUNT, DIFF_QUEUE_LIMIT. WS_AUTH_REQUIRED and WS_BROADCAST appeared in release candidates and no longer exist.

Behaviour worth knowing:

  • If Redis is unreachable at startup, the API exits with code 1 after 30 s. REDIS_URL is needed with more than one API instance so live updates reach every client.
  • The .env and compose file still ship sample DEFAULT_USER_PASSWORD and DEFAULT_USER_API_KEY; change both before exposing VRT. When the environment sets neither, the seed generates the first admin's password and key and prints them once. The seed no longer logs users, hashes or keys, exits non-zero on error, and never promotes an existing user to admin.
  • Timeouts for Ollama and Gemini (5 min), S3, LDAP, Redis and a diff job (120 s) are fixed in code.

Details: README "Environment variables".

Database & migrations

  • Prisma 7 with the pg driver adapter and NestJS 12. One PrismaClient is shared across the app; the default pool is sized by physical cores, and DATABASE_URL TLS settings are interpreted as Prisma 6 did, with the exceptions above.
  • The bundled database is PostgreSQL 18 (volume ./db:/var/lib/postgresql); dev compose and CI run 18.
  • Four migrations are new since 5.7.0 (6.0.0 ships ten in total). The migration service applies them before the API starts:
    • 20261024174953_add_test_run_test_variation_index: TestRun_testVariationId_idx
    • 20261024175021_add_baseline_test_variation_index: Baseline_testVariationId_createdAt_idx
    • 20261024175038_add_test_run_name_baseline_name_index: TestRun_name_baselineName_idx
    • 20261109000000_upgrade_to_6_0_0: Baseline.commentBeforeApproval and ignoreAreasBeforeApproval, Project.noiseAnalysisConfig (default: both stages off), TestRun.noiseAnalysis and statusBeforeDecision, and the _UserFavoriteProjects join table.
  • The three indexes use CREATE INDEX CONCURRENTLY, so writes are not blocked. A cancelled build (for example a statement_timeout) leaves an invalid index and a failed migration (P3009); the recovery steps are in the umbrella release. Avoid deploying around 00:00, when the nightly cleanup runs.
  • upgrade_to_6_0_0 squashes the schema migrations added during the 6.0.0 release candidates and is guarded (IF NOT EXISTS, constraint checks), so rc databases upgrade in place. migrate deploy ignores applied names no longer on disk. It sets the noiseAnalysisConfig default to off; stored project configs are kept. Databases from rc1-rc3 get off for all projects. prisma migrate dev against an rc database offers a reset: recreate dev databases.
  • CI checks that migrations match the schema and that databases at api-v5.7.0, api-v6.0.0-rc4 and api-v6.0.0-rc7 upgrade without drift.

Performance

  • Image diffing runs in DiffWorkerPool worker threads, off the event loop: availableParallelism() - 1 workers (1 to 8), up to 256 queued jobs. A full queue answers 503 with Retry-After. A diff job times out after 120 s. Ignore areas that are huge or negative no longer hang a worker.
  • Auto-approve compares each baseline once and rejects size mismatches from the PNG header.
  • Three indexes (see Database & migrations) speed up large projects.
  • The nightly cleanup runs on one instance only and ages branches by last use. Build retention skips builds with uploads in the last 60 minutes and runs in the background; it never removes the reused build.
  • Repeated approve no longer duplicates the baseline; deletes remove rows before files and no longer swallow errors.
  • Downloads are streamed; DINOv2 normalises only the tiles it encodes.
  • Compile with SWC (about 5x faster builds and watch rebuilds).

Security

Warning

From api 4.16.0 to 5.7.0, GET /test-variations/details/:id leaked approvers' API keys and password hashes to any signed-in user, guests included. If untrusted users could sign in, change those users' passwords and regenerate their API keys (profile page, Regenerate).

  • GET /images/%2F... no longer redirects to another site; image names are validated.
  • Editors' API keys can no longer rewrite a build's branchName, number, status or projectId.
  • /ollama/* requires an admin or editor login.
  • GET /users/all and role changes no longer return other users' API keys.
  • The Gemini API key is masked in project responses.
  • Live updates require a valid token and reach only joined rooms.
  • JWT_SECRET is required and public samples are refused; JWTs resolve by sub.
  • Containers run unprivileged (uid 1000; dinov2 as nobody); NODE_TLS_REJECT_UNAUTHORIZED=0 is removed from both Dockerfiles.
  • The backend's dev docker-compose.yml binds PostgreSQL to 127.0.0.1 only.
  • OIDC trusts the issuer the provider advertises, as openid-client 5 did (now openid-client 6).
  • Dependencies: uuid-apikey dropped and ldapts 9 (no vulnerable uuid left), follow-redirects 1.16.2, js-yaml 3.15.2; CI actions pinned by SHA and release tags checked out explicitly without token persistence.

Comparison & noise

  • A project's comparison config is applied over the comparator defaults.
  • pixelmatch never treats screenshots of different sizes as equal.
  • looks-same honours the run's diffTollerancePercent; with a refused project config it compares with the defaults.
  • odiff is 4.5.0 and every odiff result is handled.
  • Ignore areas are rounded outwards and clipped to the image; fractional areas (Playwright) now mask pixels. They are also masked in images sent to the VLM.
  • A run that cannot be compared is refused before it is stored. A stale VLM analysis is cleared when a recomputed diff has none.
  • The rendering-noise check is a hint on unresolved runs: it never changes a status and never approves anything. It is off by default, for new and upgraded projects, and is switched per project in noiseAnalysisConfig (blur.enabled, dinov2.enabled). The ignoreRenderingNoise key from rc1-rc3 is ignored. VLM projects are excluded.
  • Stage one is the blur amplitude check in the API process (thresholds are constants: amplitude 80, solid share 0.01). Stage two is DINOv2 (threshold 0.15), used when DINOV2_URL is set.
  • Runs are analysed one at a time from an in-memory queue of up to 256 (POST /test-runs/noise-analysis). The queue is not persisted: pending analyses are dropped on restart and can be requested again.
  • The dinov2 service (dinov2/, Python, ONNX Runtime, no PyTorch) pins facebook/dinov2-base at a fixed revision; the model is exported at image build. It answers 503 past DINOV2_MAX_QUEUE and 413 past DINOV2_MAX_PIXELS, and honours a caller deadline. Start it with docker compose --profile dinov2 up -d.

Fixed

  • Editing a feature-branch run no longer changes the main branch's variation; merging into a branch whose variation is older than main's works.
  • An empty or missing PATCH /builds/:id body stops the build again.
  • A build-wide approval records who approved.
  • GET /users/oidc/enabled answers JSON.
  • POST /test-runs to a missing project answers 404.
  • The maxBuildAllowed cleanup never removes the reused build.
  • Duplicate-record errors answer 400 again under Prisma 7.
  • Graceful shutdown with Prisma 6 connection semantics.
  • An odiff project on S3 cannot be saved until its comparator changes (it never worked).
  • A foreign buildId on upload is accepted with a warning.
  • Seed, log level and libssl warnings in the migration image.

Runtime & build

  • Node 26 (node:26-bookworm-slim), NestJS 12, TypeScript 6 in full strict mode (noImplicitAny, useUnknownInCatchVariables, strictNullChecks), Jest 30 on @swc/jest, Redis client 6, openid-client 6.
  • api and migration images are built for amd64 and arm64, each on a native runner, and published from release tags only. The Docker build is gated on the type check.
  • tini is PID 1 so SIGTERM is handled.
  • Compose: api starts after migration has finished and has a healthcheck.
  • Test suite: unit, e2e on real worker threads and sockets, a noise regression bench on real browser screenshots; coverage thresholds are enforced.

What's Changed

  • fix(vlm): mask ignore areas in images sent to the VLM by @pashidlos in #408
  • docs(readme): fix local setup steps that fail at prisma migrate deploy by @pashidlos in #409
  • fix(builds): stop build on PATCH with empty or missing body by @pashidlos in #410
  • fix(odiff): handle every odiff result, bump odiff-bin to 4.5.0 by @pashidlos in #411
  • ci(docker): build and publish api and migration images for amd64 and arm64 by @pashidlos in #412
  • fix(prisma): share a single PrismaClient across the app by @pashidlos in #413
  • fix(vlm): require admin/editor auth for Ollama endpoints by @pashidlos in #415
  • ci(docker): check out release tags explicitly and drop token persistence by @pashidlos in #416
  • Backend review: security, correctness and performance fixes by @pashidlos in #443
  • API stage F1–F4, F6: POST merge, websocket JWT and rooms, Gemini key mask, per-id bulk results by @pashidlos in #446
  • chore(deps): Node 26 images, follow-redirects 1.16.2 by @pashidlos in #449
  • fix(prisma): fail startup when the TLS probe cannot reach the database by @pashidlos in #450
  • refactor: behaviour-neutral cleanup before 6.0.0 by @pashidlos in #451
  • chore(deps): dependency refresh for 6.0 (safe majors), risky majors evaluated by @pashidlos in #452
  • fix(dinov2): make the image build's ONNX check deterministic by @pashidlos in #453
  • ci(docker): build each platform on a native runner by @pashidlos in #454
  • fix(test-variations): return the variation list in a stable order by @pashidlos in #455
  • feat!: strict socket defaults and no GET merge in 6.0 by @pashidlos in #456
  • chore: run and test on Node 26 by @pashidlos in #457
  • feat!: refuse a public JWT_SECRET in production by @pashidlos in #458
  • feat!: resolve JWTs by sub only by @pashidlos in #459
  • fix(test-runs): revert gives a feature variation back its ignore areas and comment by @pashidlos in #460
  • feat(compare)!: one rendering-noise switch, blur and DINOv2 together by @pashidlos in #461
  • feat(socket): close a socket when its token expires or its user is deleted by @pashidlos in #462
  • feat!: the api image always runs as uid 1000 by @pashidlos in #464
  • chore: PostgreSQL 18 in the dev compose and CI by @pashidlos in #465
  • chore(deps): redis 4 → 6 by @pashidlos in #466
  • chore(deps): openid-client 5 → 6 by @pashidlos in #467
  • fix(oidc): accept an advertised issuer that differs from OIDC_ISSUER_URL by @pashidlos in #470
  • refactor: drop dead code and duplicated tests by @pashidlos in #471
  • fix: keep feature-branch run edits out of the main branch variation by @pashidlos in #472
  • fix(dinov2): bounded queue, caller deadline and input size guard by @pashidlos in #473
  • fix(test-runs): only the maker of a decision gets Undo by @pashidlos in #474
  • feat!: connect without TLS when DATABASE_URL names no sslmode by @pashidlos in #475
  • test: rendering-noise regression bench on real browser screenshots by @pashidlos in #476
  • feat(compare): rendering-noise check as a hint on unresolved runs, replacing the filter by @pashidlos in #478
  • feat(statistics): results per build endpoint (B4) by @pashidlos in #479
  • feat(users): return createdAt in user summaries (B5) by @pashidlos in #480
  • test(noise): reference solid-mask script for F7 parity (P6a-be) by @pashidlos in #481
  • feat: rendering-noise check off by default for new projects by @pashidlos in #484
  • docs(agents): user-visible changes update the umbrella docs/wiki by @pashidlos in #485
  • refactor(test-runs): split TestRunsService, log swallowed errors, fix lint by @pashidlos in #486
  • feat(prisma): squash the 6.0.0 schema migrations into one by @pashidlos in #487
  • fix: default log level and Prisma libssl warning in the migration image by @pashidlos in #488
  • feat!: live updates always need a login and go only to joined rooms by @pashidlos in #489
  • refactor!: remove the build noise-summary endpoint by @pashidlos in #490
  • refactor!: remove unused PATCH /builds/:id/approve by @pashidlos in #491
  • refactor!: cut niche pre-6.0 options (noise thresholds, queue, /ollama/compare, diff env vars) by @pashidlos in #492
  • Dependency updates: #434, #440, #444, #445, #482

Full Changelog: api-v5.7.0...api-v6.0.0

Contributors

pashidlos
Assets 2
Loading