Skip to content

Test Environments and Quality Gates

MAVERICK edited this page Nov 13, 2026 · 3 revisions

Test Environments and Quality Gates

Where tests run, what each environment may contain, and which checks run and block at each stage. Part of the Testing Strategy. The deployment environments are defined on Spec: CI/CD and Release section 2; this page adds the testing view. Requirements: TST-010, TST-011, TST-017, TST-018, TST-022.

1. Environments

Environment Purpose in testing Lifetime Data Keys Network
Local development; every functional level on demand generator tiny and ci profiles in a throwaway PostGIS generated per run developer's choice, but tests deny egress
CI ephemeral every pull-request and main gate up to end to end created per job, destroyed on every exit path ci profile generated per run outbound denied after dependency install
Test stack performance, resilience, chaos, DAST full scan created per nightly or weekly run benchmark profile generated per run isolated from production traffic
Staging release acceptance; the same image digest that production will run per confirmed release production-shaped fixture set vf_test_ as production, but separate
Sandbox client trials, not a test environment of ours with each release fixtures vf_test_ issued to clients public
Production never a test target; observed by its own SLOs only — the read contract vf_live_ public

Environment readiness checklist

An environment is ready for a test run when:

  • it was created from the committed definition, not modified by hand;
  • the database was built from the read-contract stub and loaded from a generator set whose manifest checksum matches;
  • the image under test is identified by digest;
  • the target guard allow-list is present and readable, and the guard refuses a host not on it (checked at the start of each non-functional run);
  • no production credential is present in the environment (a sweep for the vf_live_ prefix finds nothing).

Isolation rules

  • No test environment holds a production credential or a connection string to production storage.
  • Load, chaos, fuzzing and DAST runners refuse any target not on the allow-list (TST-010).
  • CI ephemeral environments are destroyed on every exit path, including failure and cancellation; nothing is shared between runs (TST-022).

2. Stages

flowchart TB
  subgraph Dev[Developer machine]
    PC[Pre-commit hooks]
  end
  subgraph CI[CI ephemeral]
    PR[Pull-request gate]
    MN[Main gate]
  end
  subgraph TS[Test stack]
    NI[Nightly]
    WK[Weekly]
  end
  subgraph ST[Staging]
    RL[Release gate]
  end
  PC --> PR
  PR -->|merge after review| MN
  MN --> NI
  NI --> WK
  MN -->|release tag| RL
  NI -.->|open issues block the next release| RL
  WK -.->|open issues block the next release| RL
Loading

Staging runs the published image of a confirmed release, so a failed release acceptance is answered by yanking that release and cutting a new patch, never by replacing the image.

Text description of the diagram: pre-commit hooks run on the developer machine; the pull-request and main gates run in CI ephemeral environments; nightly and weekly runs use the test stack; a release tag starts the release gate on staging, and open issues from nightly or weekly runs block the next release.

3. Gate definitions

Stage Trigger Blocking checks Non-blocking (report or open issue) Time budget (design)
Pre-commit local commit none enforced remotely; format, lint, fast unit tests, secret scan as local hooks — under 30 s
Pull request every push to a pull request format, lint, type checks, scope and configuration lints, unit and property-based, handler, integration, coverage, mutation on changed in-scope code, contract both directions, breaking-change diff, MCP parity, short fuzz, end-to-end smoke, SAST, dependency, secret and container scans, DAST baseline, docs examples and links when docs change short load smoke, requirement report (reports) under 15 minutes with a warm cache (REL-014)
Main merge to main everything of the pull-request gate, full end-to-end set, requirement report for in-scope rows — under 20 minutes
Nightly schedule — long fuzz, deep schema-driven runs, load, polling, single-flight, overload, soak, chaos; each breach opens or updates an issue under 3 hours
Weekly schedule — full DAST, stress, spike, scheduled dependency scans, Scorecard; each finding opens an issue under 4 hours
Release release tag on staging everything of main, plus the full mutation run, load, stress, spike, soak and chaos results of the candidate, full DAST of staging, TLS and header checks, docs accessibility, no open S1 or S2, the completion report and conformance claim — as needed

4. Gate rules

  • Fail closed. Every gate fails when its input is empty, its tool is missing, its result is unreadable or a step was skipped. Every gate prints what it checked: files, tests, operations, requirements, mutants, requests (DEV-021, TST-011).
  • Declared, not implied. Each gate is listed here with its stage and whether it blocks. Changing a blocking gate to non-blocking needs a recorded decision (ADR or owner approval) (TST-018).
  • No retry to green. CI test steps run without automatic retries (TST-017); flaky tests follow the quarantine policy of the Testing Strategy section 6.
  • Planted-fault proof. Each blocking gate is shown red once by a planted fault before it is trusted, and again after any change to its configuration (TST-006).
  • Bounded. Every job and every test has a timeout; a hung job fails at its budget.
  • Required in branch protection. Every blocking pull-request check is a required status check on main (DEV-020).

Home

About

Design

Data Model

Technical specification

Testing

Engineering

Technology choice

Reference

Clone this wiki locally