Skip to content

v0.18.0

Choose a tag to compare

@avatar29A avatar29A released this 14 Aug 14:38
· 12 commits to main since this release

Seven issues from a field evaluation against v0.17.0 — the first onboarding of a large production Go service, on Uber Fx with ~30 gateways, MySQL, Redis and Kafka — plus one found while building the regression corpus for the first of them.

The headline

Two of the reports — a host-binary SUT losing every outbound socket, and the proxy data plane freezing at the test-phase boundary — were one defect, and it was not the proxy.

SECCOMP_IOCTL_NOTIF_RECV returns ENOENT when a notification has already been discarded: the notifying thread died or took a signal before the supervisor reached it. That says nothing about the listener fd. The classifier matched it anyway by substring-searching the errno text, so the notification loop returned nil on the first one and stopped supervising for good — silently, because a nil error reads as a clean shutdown. The child kept its seccomp filter with nobody to answer it, and every intercepted syscall from that point blocked forever.

That accounts for the whole reported shape: reads and new dials both failing, never recovering, the container staying healthy while only the SUT was affected, and a busy connection pool always losing while a quiet client survived — ENOENT probability scales with concurrent notification volume, and Go's SIGURG preemption interrupts syscalls constantly.

A loop that ends while its target is still running now fails the test loudly, outranking every other verdict: a filtered process with no supervisor produces no evidence, so a pass would mean "it never got far enough to fail".

Fixed

  • A fault written the normal way installs a filter. Multi-line calls and spaces around = matched neither of the literal substrings the scan looked for, so the fault silently did nothing. Where the scan still cannot see a fault, the runtime now refuses it (FAULT_NOT_FILTERABLE) rather than installing something inert — a fault either fires or the run fails saying why.
  • mock_service() is reachable from a containerized SUT. It bound host loopback, and was injected as localhost:<port>, which resolves to the SUT's own container.
  • The Docker network survives the whole run. It was destroyed after each test and never recreated, so from test 2 every container landed on the default bridge, which has no embedded DNS for container names.
  • Timeouts are no longer blamed on faults that were not there. TIMEOUT_NO_FAULT_FIRED and TIMEOUT_NO_FAULTS cover what TIMEOUT_DURING_FAULT used to absorb.
  • replay loads specs in a subdirectory, and the Replay: hint it prints is a command replay accepts.
  • Packet faults reach a containerized consumer. The gateway address was allocated behind a proxy-fault gate that packet faults are invisible to.
  • grpc.call() completes a round trip against a real server for the first time: it handed []byte to grpc-go's proto codec and failed client-side before reaching the wire. All 13 step protocols now have a real-server spec.

Changed

  • faultbox report <bundle.fb> derives its output name from the bundle (report_<ts>-<seed>.html), so reporting a second bundle no longer silently overwrites the first. --output still pins a fixed name.
  • Mock listeners bind 0.0.0.0 on Linux, as proxies already did. Overridable with FAULTBOX_PROXY_BIND.

Upgrading

A minor rather than a patch, because three changes alter what an existing spec does:

  • Faults that were silently inert now fire, so a suite that was green for the wrong reason can go red. That is the fix working.
  • The report filename changed; anything scripted against report.html needs --output.
  • TIMEOUT_DURING_FAULT no longer fires on fault-free runs.

Full notes in CHANGELOG.md.

What's Changed

  • docs: v0.17.0 follow-ups — usage banner, diagnostic cross-links by @avatar29A in #155
  • fix(netfault): unify the test wait budgets — the flake I half-fixed came back by @avatar29A in #156

Full Changelog: release-0.17.0...release-0.18.0