Tiered CI pipeline: unit → integration → e2e with self-hosted runner

[jh-lee-cryptolab]

Problem

The only CI workflow today is docker-publish.yml, triggered on release to build and push the Docker image. There is no automated test execution on pull requests — tests are run manually before merging.

This means regressions can slip through, and contributors get no feedback until someone remembers to run pytest locally.

Goal

A tiered CI pipeline that runs the right level of testing at the right time:

Tier	Tests	Trigger	Runner	~Duration
1	Unit tests	Every PR push	ubuntu-latest	~1 min
2	Integration + Docker build	Every PR push (vault/ changed)	ubuntu-latest	~3 min
3	E2E (live container + gRPC)	e2e label / push to main / manual	Self-hosted vault-ci	~5 min

Phase 0 — Self-Hosted CI Runner (OCI)

Provision a dedicated CI runner on OCI, reusing existing deployment/oci/ Terraform patterns.

Infrastructure

• deployment/ci/oci/main.tf — OCI Compute instance
  • Shape: VM.Standard.E5.Flex (2 OCPU / 8 GB RAM)
  • VCN + subnet + security list (SSH only, no public service ports)
  • Ubuntu 22.04
• deployment/ci/oci/startup-script.sh — bootstrap script
  • Docker CE + compose plugin
  • GitHub Actions self-hosted runner (auto-registered)
  • grpcurl binary
  • Python 3.12 + pip
• Runner labels: self-hosted, vault-ci
• enVector credentials stored on-runner (env file), never in GitHub Secrets

Why self-hosted

• enVector API credentials stay within controlled infrastructure
• Consistent environment (Docker, grpcurl, Python pre-installed)
• No per-minute billing for e2e runs

Phase 1 — CI Workflows

.github/workflows/ci.yml — Tier 1 + Tier 2

Triggers on pull_request targeting main.

concurrency:
  group: ci-${{ github.event.pull_request.number || github.ref }}
  cancel-in-progress: true

Path filter: skip entirely if only docs/, *.md, deployment/, .github/ changed.

Job: unit-tests

• Runner: ubuntu-latest
• Steps:
  1. Checkout
  2. Setup Python 3.12 + pip cache (actions/cache keyed on tests/requirements.txt + vault/requirements.txt)
  3. Install deps: pip install -r vault/requirements.txt -r tests/requirements.txt
  4. pytest tests/unit/ -v --tb=short --cov=vault --cov-report=xml -x
  5. Upload coverage.xml artifact

Job: integration-tests (needs: unit-tests)

• Runner: ubuntu-latest
• No ENVECTOR_ENDPOINT set → vault_core.ensure_vault() runs in offline mode (local FHE keys only)
• Steps:
  1. Checkout + setup Python + pip cache
  2. pytest tests/integration/ -v --tb=short --cov=vault --cov-report=xml
  3. Upload coverage artifact

Job: build-image (needs: unit-tests)

• Runner: ubuntu-latest
• Steps:
  1. Checkout
  2. Setup Docker Buildx
  3. docker buildx build --load -t rune-vault:ci-${{ github.sha }} vault/ (amd64 only)
  4. docker save rune-vault:ci-${{ github.sha }} | gzip > /tmp/rune-vault-ci.tar.gz
  5. Upload image artifact via actions/upload-artifact

Job: coverage-report (needs: unit-tests, integration-tests)

• Merge coverage XMLs → upload to Codecov (or post PR comment)

.github/workflows/e2e.yml — Tier 3

Triggers:

• pull_request with label e2e
• push to main
• workflow_dispatch (manual)

Job: e2e-tests

• Runner: self-hosted + vault-ci
• Needs the Docker image — either build inline or download from ci.yml artifact
• Steps:
  1. Checkout
  2. Build (or load) Docker image
  3. Start container:

     docker run -d --name vault-ci \
       -p 50051:50051 -p 9090:9090 \
       -e VAULT_TLS_DISABLE=true \
       -e VAULT_AUDIT_LOG=stdout \
       -e ENVECTOR_ENDPOINT=... \
       -e ENVECTOR_API_KEY=... \
       -e VAULT_TEAM_SECRET=$(openssl rand -hex 32) \
       rune-vault:ci-${{ github.sha }}

  4. Poll http://localhost:9090/health until ready (timeout 120s)
  5. Issue test token via admin API
  6. Run scripts/e2e-test.sh (see Phase 2)
  7. Tear down container

Phase 2 — E2E Test Script

• scripts/e2e-test.sh — standalone, runnable locally or in CI

Test cases

#	Test	Method	Pass criteria
1	Health check	curl localhost:9090/health	{"status": "healthy"}
2	Metrics endpoint	curl localhost:9090/metrics	Contains vault_requests_total
3	gRPC reflection	grpcurl -plaintext localhost:50051 list	Lists rune.vault.v1.VaultService
4	gRPC health	grpcurl ... grpc.health.v1.Health/Check	SERVING
5	GetPublicKey (valid token)	grpcurl ... VaultService/GetPublicKey	Response contains key_bundle_json
6	Auth rejection (bad token)	grpcurl ... VaultService/GetPublicKey	UNAUTHENTICATED error code
7	Audit log emitted	docker logs vault-ci | grep '"method"'	JSON with required fields (timestamp, user_id, method, status, source_ip)
8	Audit alert on auth failure	Send 5 bad tokens	AUDIT_ALERT.*auth_failure_threshold in logs

Dependencies

• Per-user token auth (already merged) — needed for meaningful e2e auth tests
• Structured audit logging (Structured audit logging for Vault operations #19) — needed for audit log verification tests

Priority

Medium — Prevents regressions, enables confident merges. Phase 0 (runner) can be done independently of workflow authoring.

[jh-lee-cryptolab]

Related: #45 — Harness engineering foundation (CLAUDE.md, linter, ADR, PR template). Phase 4 of #45 depends on CI gates established by this issue.

[m13v]

the tiered approach with self-hosted runners for E2E is the way to go. one thing we learned the hard way: make sure Tier 2 (integration) can run without the vault credentials being present, even if it means mocking the vault responses. otherwise a credential rotation or runner setup issue blocks all PRs.

for the gRPC E2E tier, consider recording and replaying gRPC interactions as your baseline test rather than always hitting the live service. tools like grpcurl with response recording let you snapshot known-good responses and diff against them. saves you from needing the full vault infrastructure running for every test.

also on the self-hosted runner, we found that a dedicated runner per repo is overkill for small teams. a shared runner with job isolation (fresh docker network per job, pruned volumes between runs) works fine and cuts your OCI bill in half.

[m13v]

we open sourced a framework that handles the E2E test generation tier. it auto-discovers test scenarios by crawling the running app and outputs real Playwright test files. the agent that orchestrates the discovery and test execution: https://github.com/assrt-ai/assrt-freestyle/blob/main/src/core/agent.ts

could plug into your Tier 3 as the test generation step before running the specs.

[jh-lee-cryptolab]

Thanks for the feedback!

Credential isolation — Integration tests (Tier 2) already run without external credentials. vault_core.ensure_vault() enters offline FHE mode when ENVECTOR_ENDPOINT is absent, so credential rotation or runner misconfiguration won't block PRs.

gRPC response recording — Understood the appeal, but the whole point of our Tier 3 is to verify actual end-to-end behavior: container startup, TLS bootstrap, gRPC health, token auth, and audit logging against a live instance. Replacing that with recorded responses would defeat the purpose.

Shared runner across repos — Good point. We only have a single repo today, but will keep job-level isolation (fresh Docker network + volume pruning) in mind if we expand to multiple repos.

Re: assrt-freestyle — The linked repository (assrt-ai/assrt-freestyle) returns a 404. Also, our service is a gRPC server, not a web application, so a Playwright-based tool wouldn't be applicable here.