ci: real-storage live test lane (TIN-1421) - #377
Merged
Conversation
Brings up SeaweedFS + NATS via the existing docker-compose stack inside the job (subset: masters + volume + filer + nats — no prometheus/grafana since those aren't on the test path), waits for health, creates the tcfs S3 bucket, then runs `cargo test -p tcfs-e2e --test fleet_live` with TCFS_E2E_LIVE=1 + endpoints pointing at localhost forwards. Closes the trust-boundary gap from the 2026-05-18 audit: PRs touching sync code paths now exercise real S3 + JetStream behavior in CI, not just inline unit tests. fleet_live's five tests (push/pull roundtrip, storage outage recovery, NATS pub/sub, durable consumer replay, neo-honey two-device smoke) all run. Triggers on PR + push to main with path filters scoped to relevant crates/tests/config/workflow paths. workflow_dispatch for manual runs. On failure, uploads `docker compose logs` for each service as an artifact for triage. Refs TIN-1421, related to TIN-133 (eliminates the recurring external-endpoint dependency that burned 5 smoke dispatches today).
curl -sf suppresses --write-out on HTTP error responses, so the 403-from-/-without-creds case (SeaweedFS's correct "service up" signal) printed nothing and the grep never matched. First CI run on this branch timed out after 3 minutes despite seaweed-filer logs showing "Start Seaweed S3 API Server ... at http port 8333" within seconds of compose up. Drop -f, capture the code into a shell var, case-match accepted responses (200/301/302/307/403/404). Echo the last code on each retry so failures are diagnosable from the log.
Shared docker-compose.yml passes `--max_payload 8MB` to nats-server, which is rejected on current nats:2.10-alpine (CLI flag not defined; must be configured via -c config file). Locally `task dev` works against a cached older image; CI pulls fresh and the nats container exits with usage help instead of starting. CI workflow now writes .github/compose-overrides/ci.yml on the fly with a minimal nats command (no --max_payload), and brings up the stack with both docker-compose.yml + the override. Does not touch the shared compose so local dev is unaffected. Followup TIN: decide whether to fix the shared compose (mount a config file with max_payload=8MB) or accept the default 1MB payload limit globally.
Owner
Author
|
Review pass after the CI peel-back:
No blocking review findings from this pass. Still waiting on the standard Nix/Nix-CI checks before marking ready/mergeable. After landing, we should run one deliberate broken-push canary to prove the lane fails red for the right class of sync break. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
First concrete step on TIN-1421 (Urgent): brings up SeaweedFS + NATS in CI via the project's existing docker-compose stack and runs
cargo test -p tcfs-e2e --test fleet_livewithTCFS_E2E_LIVE=1.This closes the trust-boundary gap from the 2026-05-18 audit: until now, the live-fleet tests only ran on operator-driven runs against neo/honey/yoga, so PRs could land broken sync paths and
ci.ymlwould stay green.Design choices
services:) keepsdocker-compose.ymlas the local + CI source of truth.TCFS_S3_ENDPOINT=http://127.0.0.1:8333,TCFS_NATS_URL=nats://127.0.0.1:4222.Validation
~/.cargo/bin/cargo test -p tcfs-e2e --test fleet_live --no-runfleet_live tests against compose-managed SeaweedFS + NATSpassed in 6m34sFollow-up