Repository navigation
Test Environments and Quality Gates
Where tests run, what each environment may contain, and which checks run and block at each stage. Part of the Testing Strategy. The deployment environments are defined on Spec: CI/CD and Release section 2; this page adds the testing view. Requirements: TST-010, TST-011, TST-017, TST-018, TST-022.
| Environment | Purpose in testing | Lifetime | Data | Keys | Network |
|---|---|---|---|---|---|
| Local | development; every functional level | on demand | generator tiny and ci profiles in a throwaway PostGIS |
generated per run | developer's choice, but tests deny egress |
| CI ephemeral | every pull-request and main gate up to end to end | created per job, destroyed on every exit path |
ci profile |
generated per run | outbound denied after dependency install |
| Test stack | performance, resilience, chaos, DAST full scan | created per nightly or weekly run |
benchmark profile |
generated per run | isolated from production traffic |
| Staging | release acceptance; the same image digest that production will run | per confirmed release | production-shaped fixture set | vf_test_ |
as production, but separate |
| Sandbox | client trials, not a test environment of ours | with each release | fixtures |
vf_test_ issued to clients |
public |
| Production | never a test target; observed by its own SLOs only | — | the read contract | vf_live_ |
public |
An environment is ready for a test run when:
- it was created from the committed definition, not modified by hand;
- the database was built from the read-contract stub and loaded from a generator set whose manifest checksum matches;
- the image under test is identified by digest;
- the target guard allow-list is present and readable, and the guard refuses a host not on it (checked at the start of each non-functional run);
- no production credential is present in the environment (a sweep for the
vf_live_prefix finds nothing).
- No test environment holds a production credential or a connection string to production storage.
- Load, chaos, fuzzing and DAST runners refuse any target not on the allow-list (TST-010).
- CI ephemeral environments are destroyed on every exit path, including failure and cancellation; nothing is shared between runs (TST-022).
flowchart TB
subgraph Dev[Developer machine]
PC[Pre-commit hooks]
end
subgraph CI[CI ephemeral]
PR[Pull-request gate]
MN[Main gate]
end
subgraph TS[Test stack]
NI[Nightly]
WK[Weekly]
end
subgraph ST[Staging]
RL[Release gate]
end
PC --> PR
PR -->|merge after review| MN
MN --> NI
NI --> WK
MN -->|release tag| RL
NI -.->|open issues block the next release| RL
WK -.->|open issues block the next release| RL
Staging runs the published image of a confirmed release, so a failed release acceptance is answered by yanking that release and cutting a new patch, never by replacing the image.
Text description of the diagram: pre-commit hooks run on the developer machine; the pull-request and main gates run in CI ephemeral environments; nightly and weekly runs use the test stack; a release tag starts the release gate on staging, and open issues from nightly or weekly runs block the next release.
| Stage | Trigger | Blocking checks | Non-blocking (report or open issue) | Time budget (design) |
|---|---|---|---|---|
| Pre-commit | local commit | none enforced remotely; format, lint, fast unit tests, secret scan as local hooks | — | under 30 s |
| Pull request | every push to a pull request | format, lint, type checks, scope and configuration lints, unit and property-based, handler, integration, coverage, mutation on changed in-scope code, contract both directions, breaking-change diff, MCP parity, short fuzz, end-to-end smoke, SAST, dependency, secret and container scans, DAST baseline, docs examples and links when docs change | short load smoke, requirement report (reports) | under 15 minutes with a warm cache (REL-014) |
| Main | merge to main | everything of the pull-request gate, full end-to-end set, requirement report for in-scope rows | — | under 20 minutes |
| Nightly | schedule | — | long fuzz, deep schema-driven runs, load, polling, single-flight, overload, soak, chaos; each breach opens or updates an issue | under 3 hours |
| Weekly | schedule | — | full DAST, stress, spike, scheduled dependency scans, Scorecard; each finding opens an issue | under 4 hours |
| Release | release tag on staging | everything of main, plus the full mutation run, load, stress, spike, soak and chaos results of the candidate, full DAST of staging, TLS and header checks, docs accessibility, no open S1 or S2, the completion report and conformance claim | — | as needed |
- Fail closed. Every gate fails when its input is empty, its tool is missing, its result is unreadable or a step was skipped. Every gate prints what it checked: files, tests, operations, requirements, mutants, requests (DEV-021, TST-011).
- Declared, not implied. Each gate is listed here with its stage and whether it blocks. Changing a blocking gate to non-blocking needs a recorded decision (ADR or owner approval) (TST-018).
- No retry to green. CI test steps run without automatic retries (TST-017); flaky tests follow the quarantine policy of the Testing Strategy section 6.
- Planted-fault proof. Each blocking gate is shown red once by a planted fault before it is trusted, and again after any change to its configuration (TST-006).
- Bounded. Every job and every test has a timeout; a hung job fails at its budget.
- Required in branch protection. Every blocking pull-request check is a required status check on main (DEV-020).
ViaFrei API: a public, keyed, read-only REST API to live German mobility data. Design phase; nothing on these pages is a measurement of a running service. Data on viafrei.de and in every API answer is attributed to its publishers under their licences.
About
Design
- Domain Model
- API Design
- Architecture Overview
- Security and Access Keys
- Rate Limits and Priority
- Performance and Caching
- Configuration
- Observability
Data Model
Technical specification
-
Technical Specification
- Interface
- Scope, Conventions and References
- Protocol
- Common Envelope
- Resource Model
- Endpoints, Places
- Endpoints, Roads
- Endpoints, Energy and Parking
- Endpoints, Public Transport and Accessibility
- Endpoints, Weather Risk, Driving Rules and Composed Features
- Service Endpoints
- Errors
- Authentication and Authorisation
- Rate Limiting, Quotas and Priority
- Data Licensing and Attribution
- Compatibility and Versioning
- OpenAPI Skeleton
- Quality
- Delivery and operations
- Interface
Testing
- Testing Strategy
- Testing and Quality (design summary)
Engineering
Technology choice
- Technology Evaluation
- Architecture: Go
- Architecture: Rust
- Architecture: TypeScript
- Stack Comparison
- Benchmark Plan
Reference