Feature request: request-scoped failure injection (Netflix FIT) on OpenTelemetry baggage

[Zaba505]

Note

The API and wire format in this post are superseded by the review comment below. This draft forces wire conventions on its users: the fit baggage key constant, the value grammar, the HMAC envelope, the required expiry and the character set for injection point names. The revision moves all of them into a caller-supplied Codec, so the organisation's edge can keep its own baggage keys and formats. The FIT background below still applies. The rest of this post is unchanged and kept for context.

Summary

Add a small package for request-scoped failure injection, modelled on Netflix's FIT (Failure Injection Testing). It uses OpenTelemetry baggage as the failure context.

The package does only two things:

• stores a failure scenario in an OTel context.Context and reads it back;
• evaluates that scenario at a named injection point.

Moving the OTel context between services is the caller's job, using their own propagators. The package assumes nothing about the architecture: no gateway, no control plane, no transport and no naming scheme. Later, humus could provide protocol clients (HTTP, gRPC, Kafka) that standardize how injection points use it.

Background: how FIT works

From the 2014 post by Kolton Andrus, Naresh Gopalani and Ben Schmaus:

1. The FIT service pushes failure metadata to Zuul, Netflix's edge gateway. Zuul checks each request against the failure scope. On a match it "decorates the request with a failure context, which is then propagated to all dependent services".
2. The scope widens step by step: first a test account or device, then a small percentage of traffic, then up to 100%.
3. Injection points live in the shared client libraries (Hystrix, Ribbon, EVCache, Astyanax). Each one checks the context for a failure aimed at its own component. "The failure behavior is provided to that layer, which determines how to emulate that failure in a realistic fashion: sleep for a delay period, return a 500, throw an exception."
4. A failure scenario is a "set of injection points which should or should not fail". One example is the "critical services" scenario: everything fails except the services needed to stream.
5. Injection points are found by tracing requests through the system.

Two later papers point the same way:

• Automating Failure Testing Research at Internet Scale (SoCC '16, §8) suggests using tracing baggage "to decorate requests with failure metadata, much as Zuul did".
• ChAP (2019) says FIT has two fault types, failure and latency, with one error type per injection point.

Proposed API

package fit // github.com/z5labs/bedrock/fit

const BaggageKey = "fit"
const MaxEncodedLen = 1024
var ErrInvalid, ErrUntrusted, ErrExpired, ErrInjected error

type Scenario struct {
    Rules   []Rule    // tried in order; the first match wins
    Expires time.Time // required
}
func (s Scenario) Match(point string) (Rule, bool)

type Rule struct {
    Point string // "grpc:/billing.v1.Billing/Charge", a prefix ending in "*", or "*"
    Fault Fault  // the zero Fault spares matching points
}

type Fault struct {
    Delay time.Duration
    Abort bool
}
func (f Fault) Wait(ctx context.Context) error   // waits out Delay, or until ctx is done
func (f Fault) Inject(ctx context.Context) error // Wait, then ErrInjected if Abort

type Key struct{ /* id, secret */ }
func NewKey(id string, secret []byte) (Key, error)

type Injector struct { // every field is required
    Keys     []Key            // Keys[0] signs; any of them verifies, which allows rotation
    MaxDelay time.Duration    // upper bound on any Delay written or honoured
    Now      func() time.Time // the clock Expires is checked against
}
func (in Injector) ContextWithScenario(ctx context.Context, s Scenario) (context.Context, error)
func (in Injector) ScenarioFromContext(ctx context.Context) (Scenario, bool, error)
func (in Injector) Eval(ctx context.Context, point string) (Fault, error)
func ContextWithoutScenario(ctx context.Context) context.Context

Usage at the edge and at an injection point:

// Edge: whatever decides that this request is under test.
ctx, err := in.ContextWithScenario(ctx, fit.Scenario{
    Rules: []fit.Rule{
        {Point: "grpc:/catalog.v1.Catalog/*"},       // spared
        {Point: "*", Fault: fit.Fault{Abort: true}}, // everything else fails
    },
    Expires: now.Add(time.Minute),
})

// Injection point, for example inside a protocol client.
f, err := in.Eval(ctx, "http:ratings.internal")
if err != nil {
    return nil, err // or record it and carry on; that is the client's choice
}
if err := f.Inject(ctx); err != nil {
    return nil, err
}

Design decisions

• One ordered list of rules, where the first match wins and a rule with the zero Fault spares its points. That covers both of FIT's scenario shapes: "fail X" and "fail everything except X".
• Injection points are plain strings. A rule matches a name exactly, or by a prefix ending in *. The package imposes no naming scheme; protocol clients choose one.
• A Fault is only Delay plus Abort, FIT's two fault types. Making an abort look real is the injection point's job, as it was in FIT. An HTTP client might return a 503 or a transport error, and a gRPC client might return codes.Unavailable. A client calls Wait and checks Abort to do this itself; Inject is the generic fallback.
• The scenario itself travels with the request, not an ID. Signadot, Gremlin Failure Flags and Filibuster send only an ID or key, and look up the scenario in a control plane or sidecar. That assumes an architecture.
• Names follow OTel Go's context functions (baggage.ContextWithBaggage, baggage.FromContext) rather than Inject/Extract. In OTel, Inject and Extract are what propagators do to carriers, and that is the part this package leaves to the caller. "Inject" also already means something else in a fault-injection package.
• Each evaluation on a request under test adds a fit.evaluation event to the current span. It is modelled on OTel's feature_flag.evaluation event. A scenario with no rules injects nothing, yet the request's trace still lists every injection point it reached. This gives FIT's tracing-based discovery of injection points for free.
• No global state. The span comes from trace.SpanFromContext rather than a global tracer provider, and the clock is Injector.Now.
• Errors versus panics. Keys and scenarios come from config or from the network, so problems with them are returned as errors. An Injector missing a required field is a programming error, so it panics.
• Out of scope:
  • deciding which requests get a scenario (by account, device or percentage);
  • the network hop between services;
  • a control plane;
  • per-call probabilities, which FIT also left to the edge.

Trust

OTel baggage has no integrity check, and services commonly accept baggage from any caller. For example, humus's REST server registers the W3C baggage propagator (rest/rest.go#L256-L261) and wraps every handler in otelhttp.NewHandler (rest/rest.go#L286-L288). With an unauthenticated format, any external caller could send baggage: fit=…*=delay:… and tie up goroutines and connections at every hop. Envoy's fault filter docs give the same warning about its x-envoy-fault-* headers. So the proposal adds four safeguards:

• A signature. Every scenario carries a 16-byte HMAC-SHA256 tag and the ID of the key that signed it. Forging a scenario takes the key. Keys can be rotated, because Keys[0] signs and any key verifies.
• A required Expires. OTel auto-instrumentation forwards baggage to third-party APIs, so scenarios will leak, and a leaked scenario has to stop working.
• MaxDelay. Even a validly signed scenario can only cost a bounded amount.
• ContextWithoutScenario. Call it wherever requests cross a trust boundary, in either direction.

All of this is about 80 lines of standard-library crypto.

Wire format

fit=1~k1~1792065660~grpc:/catalog.v1.Catalog/*|grpc:/playback.v1.Playback/*|*=delay:250+abort~49yBG3zn7ROBHL1empzp_g

• Layout. The value is version~key id~expiry (unix seconds)~rules~mac. A rule is a point pattern, optionally followed by =abort, =delay:<ms> or =delay:<ms>+abort.
• No escaping. Every character is one that W3C Baggage allows in a value unescaped, so otel-go sends the value without percent-encoding. The Example test below checks this.
• A 1024-byte cap. W3C only guarantees 64 entries and 8192 bytes, shared by every baggage user. otel-go's baggage.Parse returns empty baggage for a header over 8192 bytes, so an oversized scenario would wipe every other baggage entry downstream.
• A cross-language contract. The format is versioned and has to be implementable in other languages, which is why it stays this simple.

Where it should live: bedrock or humus?

The proposal is to make it a nested module in the bedrock repository, github.com/z5labs/bedrock/fit, with its own go.mod.

• For bedrock: this is a foundation piece, not a framework piece.
  • It assumes no transport and no topology.
  • Teams building their own framework on bedrock should be able to use it without pulling in humus's dependencies (docker, testcontainers, pgx, franz-go, grpc).
  • Putting the core in bedrock and the protocol clients in humus mirrors how FIT was layered: one shared failure context, with injection points in the client libraries everyone used.
• Against bedrock's root module: the process-model reset (story(repo): land the reset on main as a licensed, CI-green Go 1.27 module #553) made "library code imports only the standard library" a convention. It deliberately removed runtime/otel and the Renovate rules for OTel (see story(otel): inline OpenTelemetry setup in humus humus#623). An OTel package in the root module would undo that in three ways:
  • every bedrock user's module graph would include the OTel API again, even for teams that only use the process model;
  • OTel's frequent releases would come back to bedrock's dependency updates;
  • the package has nothing to do with the "an application is an OS process" idea bedrock is now built around.
• A nested module gets both. The root module stays standard-library only, and fit/go.mod requires only go.opentelemetry.io/otel and go.opentelemetry.io/otel/trace. It has three costs:
  • CI, lint and coverage each need a step per module, because go build/vet/test ./... in .github/workflows/build.yaml does not descend into nested modules.
  • The module is versioned on its own (fit/vX.Y.Z tags).
  • The convention becomes "the bedrock module imports only the standard library; fit imports only the OTel API".
• Fallback: if multi-module plumbing isn't wanted, put it in humus as a transport-agnostic humus/fit rather than loosening bedrock's standard-library-only rule.

What humus would build on top

• Injection point names per protocol, ideally taken from OTel semantic-convention attributes so they line up with trace data: grpc:<rpc method>, http:<server.address>, kafka:<messaging.destination.name>.
• What an abort looks like in each protocol.
• Stripping scenarios from requests that come in from untrusted callers and from calls out to third parties.
• Building the Injector from config (keys and MaxDelay) through bedrock's Environment.
• Carrying baggage in Kafka message headers, so that scenarios survive async hops.

Open questions

1. Should signing be mandatory? The proposal says yes. An opt-out is how a deployment ends up exposed to the attack described under Trust.
2. Is a 1024-byte cap right? It fits about 30 rules, enough for a "critical services"-sized scenario.
3. Should there be error kinds (throttled versus unavailable)? The proposal leaves them out of v1. A version 2 of the format could add something like +abort:throttled.
4. What should the package be called? fit reads as fit.Fault and fit.Scenario; fault would stutter as fault.Fault.

Prototype

A working prototype is below.

• Builds against: OpenTelemetry Go v1.47.0 and Go 1.27.
• Checks: go vet, go test -race and golangci-lint with bedrock's .golangci.yaml all pass.
• Coverage: 86.9% of statements, short of the 100% library packages need.
• Dependencies: its compiled code imports packages from three modules only: go.opentelemetry.io/otel, go.opentelemetry.io/otel/trace and github.com/cespare/xxhash/v2.

go.mod

module github.com/z5labs/bedrock/fit

go 1.27.0

require (
	github.com/stretchr/testify v1.12.1
	go.opentelemetry.io/otel v1.47.0
	go.opentelemetry.io/otel/sdk v1.47.0
	go.opentelemetry.io/otel/trace v1.47.0
)

require (
	github.com/cespare/xxhash/v2 v2.3.0 // indirect
	github.com/go-logr/logr v1.4.4 // indirect
	github.com/go-logr/stdr v1.2.2 // indirect
	github.com/google/uuid v1.6.0 // indirect
	go.opentelemetry.io/auto/sdk v1.2.1 // indirect
	go.opentelemetry.io/otel/log v1.47.0 // indirect
	go.opentelemetry.io/otel/metric v1.47.0 // indirect
	go.yaml.in/yaml/v3 v3.0.5 // indirect
	golang.org/x/sys v0.48.0 // indirect
)

fit.go

// Package fit injects request-scoped failures, after Netflix's Failure
// Injection Testing, using OpenTelemetry baggage as the failure context.
//
// A Scenario names the injection points that should or should not fail.
// Whoever decides that a request is under test stores a Scenario in the
// request's context with Injector.ContextWithScenario. The Scenario then
// travels wherever the context's OTel baggage is propagated, which is the
// caller's responsibility. Each injection point asks Injector.Eval which
// Fault, if any, it should emulate for the request, and emulates it.
package fit

import (
	"context"
	"errors"
	"strings"
	"time"
)

// Fault is a failure an injection point emulates. The zero Fault emulates
// nothing.
type Fault struct {
	// Delay is how long the injection point waits before it proceeds or
	// aborts.
	Delay time.Duration

	// Abort makes the injection point fail instead of proceeding.
	Abort bool
}

// ErrInjected is the error Fault.Inject returns for an aborting Fault.
var ErrInjected = errors.New("fit: injected failure")

// Wait waits out f.Delay. It returns context.Cause(ctx) if ctx is done
// first.
func (f Fault) Wait(ctx context.Context) error {
	if f.Delay <= 0 {
		return nil
	}
	t := time.NewTimer(f.Delay)
	defer t.Stop()
	select {
	case <-ctx.Done():
		return context.Cause(ctx)
	case <-t.C:
		return nil
	}
}

// Inject emulates f generically: it waits out f.Delay and then returns
// ErrInjected if f.Abort. An injection point that can fail more
// realistically, such as with a 503 response, calls Wait and inspects Abort
// itself.
func (f Fault) Inject(ctx context.Context) error {
	if err := f.Wait(ctx); err != nil {
		return err
	}
	if f.Abort {
		return ErrInjected
	}
	return nil
}

// Rule matches injection points by name and gives the Fault they emulate.
type Rule struct {
	// Point is the name of an injection point, or a prefix of one followed
	// by "*". A lone "*" matches every injection point.
	Point string

	// Fault is the failure matching injection points emulate. The zero
	// Fault spares them.
	Fault Fault
}

func (r Rule) matches(point string) bool {
	if prefix, ok := strings.CutSuffix(r.Point, "*"); ok {
		return strings.HasPrefix(point, prefix)
	}
	return r.Point == point
}

// Scenario is a set of injection points that should or should not fail.
type Scenario struct {
	// Rules are tried in order, and the first whose Point matches an
	// injection point decides its Fault. An injection point that no Rule
	// matches emulates nothing. A Scenario without Rules injects nothing,
	// but still marks a request as under test, so its trace records every
	// injection point the request reaches.
	Rules []Rule

	// Expires is when the Scenario stops being honoured. It is required,
	// so a Scenario that leaks out of the system cannot be replayed
	// indefinitely.
	Expires time.Time
}

// Match returns the first Rule of s that matches point.
func (s Scenario) Match(point string) (Rule, bool) {
	for _, r := range s.Rules {
		if r.matches(point) {
			return r, true
		}
	}
	return Rule{}, false
}

injector.go

package fit

import (
	"context"
	"crypto/hmac"
	"crypto/sha256"
	"encoding/base64"
	"errors"
	"fmt"
	"strconv"
	"strings"
	"time"

	"go.opentelemetry.io/otel/attribute"
	"go.opentelemetry.io/otel/baggage"
	"go.opentelemetry.io/otel/trace"
)

// BaggageKey is the key of the OTel baggage member that carries a Scenario.
const BaggageKey = "fit"

// MaxEncodedLen is the longest encoded Scenario, in bytes. It keeps a
// Scenario well inside the 8192 bytes of baggage the W3C Baggage
// specification guarantees, which every other baggage member shares.
const MaxEncodedLen = 1024

var (
	// ErrInvalid reports a Scenario that is malformed, or that asks for a
	// Fault the Injector does not honour.
	ErrInvalid = errors.New("fit: invalid scenario")

	// ErrUntrusted reports a Scenario that is not signed by any of the
	// Injector's keys.
	ErrUntrusted = errors.New("fit: untrusted scenario")

	// ErrExpired reports a Scenario whose Expires has passed.
	ErrExpired = errors.New("fit: expired scenario")
)

// Key is a secret shared by every service that writes or honours Scenarios.
type Key struct {
	id     string
	secret []byte
}

// NewKey returns a Key named id. The id travels with every Scenario signed
// by the Key, so that Keys can be rotated. It is 1 to 16 ASCII letters,
// digits, '-' or '_'. The secret is at least 32 bytes.
func NewKey(id string, secret []byte) (Key, error) {
	if len(id) == 0 || len(id) > 16 || strings.IndexFunc(id, func(r rune) bool { return !isKeyIDRune(r) }) >= 0 {
		return Key{}, fmt.Errorf("fit: invalid key id %q", id)
	}
	if len(secret) < 32 {
		return Key{}, fmt.Errorf("fit: key %q has a %d-byte secret, want at least 32", id, len(secret))
	}
	return Key{id: id, secret: secret}, nil
}

func isKeyIDRune(r rune) bool {
	return 'a' <= r && r <= 'z' || 'A' <= r && r <= 'Z' || '0' <= r && r <= '9' || r == '-' || r == '_'
}

// Injector writes Scenarios into, and evaluates them from, the OTel baggage
// of a context.Context. Every field is required.
type Injector struct {
	// Keys authenticate Scenarios. Keys[0] signs every Scenario the
	// Injector writes, and a Scenario signed by any of the Keys is
	// honoured.
	Keys []Key

	// MaxDelay is the longest Fault.Delay the Injector writes or honours.
	MaxDelay time.Duration

	// Now returns the current time, which Scenario.Expires is checked
	// against.
	Now func() time.Time
}

func (in Injector) check() {
	if len(in.Keys) == 0 || in.MaxDelay <= 0 || in.Now == nil {
		panic("fit: Injector requires Keys, a positive MaxDelay and Now")
	}
	for _, k := range in.Keys {
		if k.id == "" {
			panic("fit: Injector has a zero Key; build Keys with NewKey")
		}
	}
}

// ContextWithScenario returns a copy of ctx whose baggage carries s, signed
// with in.Keys[0]. It replaces any Scenario ctx already carries. On error it
// returns ctx unchanged.
func (in Injector) ContextWithScenario(ctx context.Context, s Scenario) (context.Context, error) {
	in.check()
	v, err := in.encode(s)
	if err != nil {
		return ctx, err
	}
	m, err := baggage.NewMemberRaw(BaggageKey, v)
	if err != nil {
		return ctx, err
	}
	b, err := baggage.FromContext(ctx).SetMember(m)
	if err != nil {
		return ctx, err
	}
	return baggage.ContextWithBaggage(ctx, b), nil
}

// ContextWithoutScenario returns a copy of ctx whose baggage carries no
// Scenario. Call it where requests cross a trust boundary, in either
// direction.
func ContextWithoutScenario(ctx context.Context) context.Context {
	return baggage.ContextWithBaggage(ctx, baggage.FromContext(ctx).DeleteMember(BaggageKey))
}

// ScenarioFromContext returns the Scenario carried by the baggage of ctx,
// and whether it carries one. It returns an error wrapping ErrInvalid,
// ErrUntrusted or ErrExpired if the Scenario cannot be honoured.
func (in Injector) ScenarioFromContext(ctx context.Context) (Scenario, bool, error) {
	in.check()
	m := baggage.FromContext(ctx).Member(BaggageKey)
	if m.Key() == "" {
		return Scenario{}, false, nil
	}
	s, err := in.decode(m.Value())
	if err != nil {
		return Scenario{}, true, err
	}
	return s, true, nil
}

// Eval returns the Fault the injection point named point emulates for the
// request ctx belongs to. It returns the zero Fault if ctx carries no
// Scenario or no Rule matches point, and the errors of ScenarioFromContext.
//
// When ctx carries a Scenario, Eval adds a "fit.evaluation" event to the
// span in ctx, so a request's trace shows every injection point it reached
// and the Fault each one emulated.
func (in Injector) Eval(ctx context.Context, point string) (Fault, error) {
	s, ok, err := in.ScenarioFromContext(ctx)
	if !ok || err != nil {
		return Fault{}, err
	}
	r, _ := s.Match(point)
	trace.SpanFromContext(ctx).AddEvent("fit.evaluation", trace.WithAttributes(
		attribute.String("fit.point", point),
		attribute.String("fit.rule", r.Point),
		attribute.Int64("fit.delay_ms", r.Fault.Delay.Milliseconds()),
		attribute.Bool("fit.abort", r.Fault.Abort),
	))
	return r.Fault, nil
}

// The encoded form of a Scenario is
//
//	1~<key id>~<expiry, unix seconds>~<rule>|<rule>|...~<mac>
//
// where a rule is a point pattern optionally followed by "=abort",
// "=delay:<ms>" or "=delay:<ms>+abort", and the mac is the first 16 bytes of
// the HMAC-SHA256 of everything before it, in unpadded base64url. Every
// character is a W3C baggage-octet, so the value is never percent-encoded.
const version = "1"

func (in Injector) encode(s Scenario) (string, error) {
	if s.Expires.IsZero() {
		return "", fmt.Errorf("%w: Expires is required", ErrInvalid)
	}
	rules := make([]string, len(s.Rules))
	for i, r := range s.Rules {
		if err := in.validate(r); err != nil {
			return "", err
		}
		rules[i] = r.Point + encodeFault(r.Fault)
	}
	k := in.Keys[0]
	payload := strings.Join([]string{version, k.id, strconv.FormatInt(s.Expires.Unix(), 10), strings.Join(rules, "|")}, "~")
	v := payload + "~" + mac(k, payload)
	if len(v) > MaxEncodedLen {
		return "", fmt.Errorf("%w: encodes to %d bytes, more than %d", ErrInvalid, len(v), MaxEncodedLen)
	}
	return v, nil
}

func encodeFault(f Fault) string {
	switch {
	case f.Delay > 0 && f.Abort:
		return "=delay:" + strconv.FormatInt(f.Delay.Milliseconds(), 10) + "+abort"
	case f.Delay > 0:
		return "=delay:" + strconv.FormatInt(f.Delay.Milliseconds(), 10)
	case f.Abort:
		return "=abort"
	}
	return ""
}

func (in Injector) decode(v string) (Scenario, error) {
	if len(v) > MaxEncodedLen {
		return Scenario{}, fmt.Errorf("%w: %d bytes, more than %d", ErrInvalid, len(v), MaxEncodedLen)
	}
	parts := strings.Split(v, "~")
	if len(parts) != 5 || parts[0] != version {
		return Scenario{}, fmt.Errorf("%w: not a version %s scenario", ErrInvalid, version)
	}
	k, ok := in.key(parts[1])
	if !ok {
		return Scenario{}, fmt.Errorf("%w: unknown key %q", ErrUntrusted, parts[1])
	}
	payload := v[:len(v)-len(parts[4])-1]
	if !hmac.Equal([]byte(parts[4]), []byte(mac(k, payload))) {
		return Scenario{}, fmt.Errorf("%w: bad signature for key %q", ErrUntrusted, k.id)
	}
	exp, err := strconv.ParseInt(parts[2], 10, 64)
	if err != nil {
		return Scenario{}, fmt.Errorf("%w: expiry: %w", ErrInvalid, err)
	}
	s := Scenario{Expires: time.Unix(exp, 0)}
	if !in.Now().Before(s.Expires) {
		return Scenario{}, fmt.Errorf("%w: at %s", ErrExpired, s.Expires.UTC().Format(time.RFC3339))
	}
	if parts[3] == "" {
		return s, nil
	}
	for _, rs := range strings.Split(parts[3], "|") {
		r, err := decodeRule(rs)
		if err != nil {
			return Scenario{}, err
		}
		if err := in.validate(r); err != nil {
			return Scenario{}, err
		}
		s.Rules = append(s.Rules, r)
	}
	return s, nil
}

func decodeRule(s string) (Rule, error) {
	point, fault, _ := strings.Cut(s, "=")
	r := Rule{Point: point}
	switch d, abort := strings.CutSuffix(fault, "+abort"); {
	case fault == "":
	case fault == "abort":
		r.Fault.Abort = true
	case strings.HasPrefix(d, "delay:"):
		ms, err := strconv.ParseInt(d[len("delay:"):], 10, 64)
		if err != nil || ms <= 0 {
			return Rule{}, fmt.Errorf("%w: rule %q: bad delay", ErrInvalid, s)
		}
		r.Fault = Fault{Delay: time.Duration(ms) * time.Millisecond, Abort: abort}
	default:
		return Rule{}, fmt.Errorf("%w: rule %q: unknown fault %q", ErrInvalid, s, fault)
	}
	return r, nil
}

func (in Injector) validate(r Rule) error {
	name, _ := strings.CutSuffix(r.Point, "*")
	if r.Point == "" || strings.IndexFunc(name, func(c rune) bool { return !isPointRune(c) }) >= 0 {
		return fmt.Errorf("%w: point %q: want ASCII letters, digits and ._:/- with an optional trailing *", ErrInvalid, r.Point)
	}
	if r.Fault.Delay < 0 || r.Fault.Delay%time.Millisecond != 0 {
		return fmt.Errorf("%w: point %q: delay %s is not a whole number of milliseconds", ErrInvalid, r.Point, r.Fault.Delay)
	}
	if r.Fault.Delay > in.MaxDelay {
		return fmt.Errorf("%w: point %q: delay %s exceeds %s", ErrInvalid, r.Point, r.Fault.Delay, in.MaxDelay)
	}
	return nil
}

func isPointRune(r rune) bool {
	return isKeyIDRune(r) || r == '.' || r == ':' || r == '/'
}

func (in Injector) key(id string) (Key, bool) {
	for _, k := range in.Keys {
		if k.id == id {
			return k, true
		}
	}
	return Key{}, false
}

func mac(k Key, payload string) string {
	h := hmac.New(sha256.New, k.secret)
	if _, err := h.Write([]byte(payload)); err != nil {
		panic(err) // hash.Hash.Write never returns an error
	}
	return base64.RawURLEncoding.EncodeToString(h.Sum(nil)[:16])
}

fit_test.go

package fit_test

import (
	"context"
	"fmt"
	"strings"
	"testing"
	"time"

	"github.com/stretchr/testify/require"
	"go.opentelemetry.io/otel/attribute"
	"go.opentelemetry.io/otel/baggage"
	"go.opentelemetry.io/otel/propagation"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
	"go.opentelemetry.io/otel/sdk/trace/tracetest"

	"github.com/z5labs/bedrock/fit"
)

var now = time.Date(2026, 10, 15, 12, 0, 0, 0, time.UTC)

func newInjector(t testing.TB, ids ...string) fit.Injector {
	var keys []fit.Key
	for _, id := range ids {
		k, err := fit.NewKey(id, []byte(strings.Repeat(id, 32)))
		require.NoError(t, err)
		keys = append(keys, k)
	}
	return fit.Injector{Keys: keys, MaxDelay: 5 * time.Second, Now: func() time.Time { return now }}
}

// Example shows a scenario crossing a service hop. The edge stores it, the
// caller's own OTel propagator carries it, and the downstream injection
// points evaluate it.
func Example() {
	k, _ := fit.NewKey("k1", []byte("0123456789abcdef0123456789abcdef"))
	in := fit.Injector{Keys: []fit.Key{k}, MaxDelay: 5 * time.Second, Now: func() time.Time { return now }}

	// Netflix's "critical services" scenario: everything fails except the
	// services needed to stream.
	critical := fit.Scenario{
		Rules: []fit.Rule{
			{Point: "grpc:/catalog.v1.Catalog/*"},
			{Point: "grpc:/playback.v1.Playback/*"},
			{Point: "*", Fault: fit.Fault{Delay: 250 * time.Millisecond, Abort: true}},
		},
		Expires: now.Add(time.Minute),
	}
	ctx, err := in.ContextWithScenario(context.Background(), critical)
	if err != nil {
		panic(err)
	}

	// The hop is the caller's business; here it is OTel's baggage propagator.
	carrier := propagation.MapCarrier{}
	propagation.Baggage{}.Inject(ctx, carrier)
	fmt.Println("baggage:", carrier.Get("baggage"))
	downstream := propagation.Baggage{}.Extract(context.Background(), carrier)

	for _, point := range []string{"grpc:/catalog.v1.Catalog/List", "http:ratings.internal"} {
		f, err := in.Eval(downstream, point)
		if err != nil {
			panic(err)
		}
		fmt.Printf("%s: %+v\n", point, f)
	}
	// Output:
	// baggage: fit=1~k1~1792065660~grpc:/catalog.v1.Catalog/*|grpc:/playback.v1.Playback/*|*=delay:250+abort~49yBG3zn7ROBHL1empzp_g
	// grpc:/catalog.v1.Catalog/List: {Delay:0s Abort:false}
	// http:ratings.internal: {Delay:250ms Abort:true}
}

func TestRoundTrip(t *testing.T) {
	in := newInjector(t, "k1")
	want := fit.Scenario{
		Rules: []fit.Rule{
			{Point: "kafka:orders", Fault: fit.Fault{Abort: true}},
			{Point: "redis:sessions", Fault: fit.Fault{Delay: time.Second}},
			{Point: "*"},
		},
		Expires: now.Add(time.Minute),
	}
	ctx, err := in.ContextWithScenario(context.Background(), want)
	require.NoError(t, err)
	got, ok, err := in.ScenarioFromContext(ctx)
	require.NoError(t, err)
	require.True(t, ok)
	require.Equal(t, want.Rules, got.Rules)
	require.True(t, want.Expires.Equal(got.Expires))
}

func TestNoScenario(t *testing.T) {
	in := newInjector(t, "k1")
	f, err := in.Eval(context.Background(), "http:anything")
	require.NoError(t, err)
	require.Equal(t, fit.Fault{}, f)
}

func TestKeyRotation(t *testing.T) {
	ctx, err := newInjector(t, "old").ContextWithScenario(context.Background(), fit.Scenario{Expires: now.Add(time.Minute)})
	require.NoError(t, err)

	_, _, err = newInjector(t, "new").ScenarioFromContext(ctx)
	require.ErrorIs(t, err, fit.ErrUntrusted)

	_, ok, err := newInjector(t, "new", "old").ScenarioFromContext(ctx)
	require.NoError(t, err)
	require.True(t, ok)
}

func TestTamperedScenarioIsUntrusted(t *testing.T) {
	in := newInjector(t, "k1")
	ctx, err := in.ContextWithScenario(context.Background(), fit.Scenario{
		Rules:   []fit.Rule{{Point: "http:payments", Fault: fit.Fault{Delay: time.Second}}},
		Expires: now.Add(time.Minute),
	})
	require.NoError(t, err)
	v := baggage.FromContext(ctx).Member(fit.BaggageKey).Value()
	m, err := baggage.NewMemberRaw(fit.BaggageKey, strings.Replace(v, "delay:1000", "delay:5000", 1))
	require.NoError(t, err)
	b, err := baggage.New(m)
	require.NoError(t, err)

	_, err = in.Eval(baggage.ContextWithBaggage(context.Background(), b), "http:payments")
	require.ErrorIs(t, err, fit.ErrUntrusted)
}

func TestExpiredScenario(t *testing.T) {
	in := newInjector(t, "k1")
	ctx, err := in.ContextWithScenario(context.Background(), fit.Scenario{Expires: now})
	require.NoError(t, err)
	_, err = in.Eval(ctx, "http:payments")
	require.ErrorIs(t, err, fit.ErrExpired)
}

func TestInvalidScenarios(t *testing.T) {
	in := newInjector(t, "k1")
	for name, s := range map[string]fit.Scenario{
		"no expiry":       {},
		"delay too long":  {Rules: []fit.Rule{{Point: "a", Fault: fit.Fault{Delay: time.Minute}}}, Expires: now.Add(time.Minute)},
		"sub-millisecond": {Rules: []fit.Rule{{Point: "a", Fault: fit.Fault{Delay: time.Microsecond}}}, Expires: now.Add(time.Minute)},
		"bad point":       {Rules: []fit.Rule{{Point: "a|b"}}, Expires: now.Add(time.Minute)},
		"star mid-point":  {Rules: []fit.Rule{{Point: "a*b"}}, Expires: now.Add(time.Minute)},
		"too long":        {Rules: []fit.Rule{{Point: strings.Repeat("x", fit.MaxEncodedLen)}}, Expires: now.Add(time.Minute)},
	} {
		t.Run(name, func(t *testing.T) {
			_, err := in.ContextWithScenario(context.Background(), s)
			require.ErrorIs(t, err, fit.ErrInvalid)
		})
	}
}

func TestEvalRecordsSpanEvent(t *testing.T) {
	in := newInjector(t, "k1")
	rec := tracetest.NewSpanRecorder()
	ctx, span := sdktrace.NewTracerProvider(sdktrace.WithSpanProcessor(rec)).Tracer("test").Start(context.Background(), "op")

	// A Scenario without Rules injects nothing but still traces every point.
	ctx, err := in.ContextWithScenario(ctx, fit.Scenario{Expires: now.Add(time.Minute)})
	require.NoError(t, err)
	_, err = in.Eval(ctx, "http:payments")
	require.NoError(t, err)
	span.End()

	events := rec.Ended()[0].Events()
	require.Len(t, events, 1)
	require.Equal(t, "fit.evaluation", events[0].Name)
	require.Contains(t, events[0].Attributes, attribute.String("fit.point", "http:payments"))
}

func TestInject(t *testing.T) {
	require.NoError(t, fit.Fault{}.Inject(context.Background()))
	require.ErrorIs(t, fit.Fault{Abort: true}.Inject(context.Background()), fit.ErrInjected)

	ctx, cancel := context.WithCancelCause(context.Background())
	cancel(context.DeadlineExceeded)
	require.ErrorIs(t, fit.Fault{Delay: time.Hour, Abort: true}.Inject(ctx), context.DeadlineExceeded)
}

References

• FIT: Failure Injection Testing, Netflix Tech Blog, 2014
• Automating Failure Testing Research at Internet Scale, Alvaro et al., SoCC '16
• Automating Chaos Experiments in Production (ChAP), Basiri et al., 2019
• Envoy fault injection filter
• W3C Baggage and OpenTelemetry baggage
• Gremlin Failure Flags for Go
• Filibuster, Meiklejohn et al., SoCC '21
• Signadot context propagation

[Zaba505]

Review: does the proposed API force a standard on its users?

This comment covers two things: an adversarial review of the proposal above, and one review comment raised separately, which is that the baggage key must not be a constant.

The question that matters most is whether bedrock forces wire-level conventions on the organisations that use it. FIT isn't an open standard, and bedrock is a foundation for teams that build their own frameworks.

Why this matters

An organisation's edge decides which requests are under test. It is often an API gateway such as Envoy, Zuul, Kong or a cloud load balancer, and a different team usually owns it. In a large organisation, the teams that own application services and the teams that own infrastructure such as edge gateways are not the same people. Infrastructure changes also get much harder scrutiny than application changes.

So any wire convention bedrock forces turns an application team's choice of library into a change request for the infrastructure team. The organisation should be able to standardize on its own baggage keys and values, and bedrock should work with them.

Verdict

Yes, the proposal forces a standard, and the baggage key is only one part of it. The proposal says the package "assumes nothing about the architecture" and "imposes no naming scheme". Neither is true.

The only way to write or evaluate a scenario is through Injector, and its wire format is unexported and fixed:

• the fit baggage entry;
• the 1~kid~unix-secs~rules~mac grammar;
• an HMAC-SHA256 envelope;
• a mandatory expiry;
• a size cap;
• a character set for injection point names.

An Envoy, Kong or ALB edge would have to produce all of these. Making the key configurable is necessary, but on its own it isn't enough. The edge team would still have to ship bedrock's grammar, an HMAC implementation and a shared secret, which is a bigger change request than picking a key name.

Only two things genuinely need to be fixed:

• the in-memory model (Scenario, Rule, Fault, first match wins, exact or prefix matching);
• OTel baggage as the carrier.

Nothing on the wire has to be fixed.

Forced conventions

Forced convention	Severity	Fix
Value grammar 1~kid~exp~rules~mac, its version prefix and the HMAC envelope	Blocks adoption. The edge team has to implement bedrock's grammar and crypto and hold a shared secret. An Envoy Lua filter, or a gateway that can only add static values, can't do this.	The caller supplies a Codec
const BaggageKey = "fit"	Blocks adoption. It is a generic name in the shared W3C namespace. Any other tool's fit entry makes every injection point fail, and ContextWithoutScenario deletes that tool's entry.	The Codec chooses the key or keys
HMAC signing as the only trust model	Blocks adoption. Every edge has to compute the HMAC, and every service holds the symmetric secret, so every verifier can also mint scenarios. It rules out mTLS plus stripping at the boundary, asymmetric signatures, and a registry of scenarios.	Trust moves into the Codec
Required Scenario.Expires, checked against each service's clock	Blocks adoption or forces glue code. It assumes synced clocks and breaks designs that send only a scenario ID.	Expiry moves into the Codec
Exactly one baggage entry holding one string	Forces glue code for gateways that can only set several static entries, such as chaos.target=payments,chaos.fault=abort	The Codec reads the whole baggage
Injection point names limited to [A-Za-z0-9._:/-]	Forces glue code. GET /users/{id}, http:[::1]:8080 and nats:orders.> are rejected.	Name rules move into the Codec
1024-byte cap, millisecond units, fit.* span event names	Cosmetic once the above is fixed. Even so, a 1.5ms time.Duration is rejected, because the wire unit leaks into the Go API.	Covered by the Codec and an Observe hook

The other side of the argument. Within one organisation, every service has to agree on a format, and a ready-made signed format is useful. Most teams don't want to design an HMAC envelope. The flaw is only that bedrock's format is the sole option, so bedrock chooses instead of the organisation.

The original reasoning contradicts itself. The proposal rejects ID-based designs because they "assume an architecture". Forbidding them is also an architecture choice. With a Codec, both designs work.

Bugs in the original prototype

Each of these is shown by a test in the collapsible section Tests against the original prototype below.

• An expired or invalid scenario fails the request instead of being ignored. The documented usage is return nil, err. A Kafka message consumed after its scenario expires would fail forever, like a poison message. A request carrying a stray fit entry from another tool fails too.
• Services with different MaxDelay settings reject whole scenarios. A 2s rule aimed at point a makes Eval fail at an unrelated point b in any service whose MaxDelay is 1s.
• The "cross-language contract" is really defined by Go's behaviour. A signed delay:288230376151712744 decodes to 1s because of int64 overflow, while another language would read it as about 9 million years. +0250 and an expiry of +1792065660 are also accepted.
• The grammar breaks with OTel Python. OTel Python's baggage propagator decodes values with unquote_plus, so the + in delay:250+abort arrives as a space and a Python verifier rejects the signature.
• Nothing on the verifying side caps a scenario's lifetime. A scenario with Expires in year 9999 is honoured, and an edge can mint one that has already expired.

Revised core

package fit

type Fault struct{ Delay time.Duration; Abort bool } // Wait, Inject and ErrInjected unchanged
type Rule struct{ Point string; Fault Fault }         // no character set
type Scenario struct{ Rules []Rule }                  // Expires removed
func (Scenario) Match(point string) (Rule, bool)

// Codec owns every wire convention: the baggage key or keys, the grammar,
// the version, the size limit and trust.
type Codec interface {
	Encode(b baggage.Baggage, s Scenario) (baggage.Baggage, error)
	Decode(b baggage.Baggage) (s Scenario, ok bool, err error)
	Strip(b baggage.Baggage) baggage.Baggage
}
func ContextWithScenario(ctx context.Context, c Codec, s Scenario) (context.Context, error)
func ScenarioFromContext(ctx context.Context, c Codec) (Scenario, bool, error)
func ContextWithoutScenario(ctx context.Context, c Codec) context.Context

type Evaluation struct{ Point string; Rule Rule; Matched bool; Fault Fault; Err error }

type Evaluator struct { // every field is required
	Codec    Codec
	MaxDelay time.Duration                     // clamps longer delays instead of rejecting them
	Observe  func(context.Context, Evaluation) // also receives scenarios the Codec rejected
}
func (Evaluator) Eval(ctx context.Context, point string) Fault // a rejected scenario injects nothing

Errors are no longer returned from Eval, but they aren't discarded either. They go to Observe, which is required, so this still follows bedrock's "every error is checked" rule.

The revised prototype's Example runs three different codecs through the same core:

• a JSON value under chaos.scenario, written by an Envoy edge that can't compute an HMAC;
• a scenario ID under acme-fault, resolved from the service's own registry;
• the original signed format, now one codec among others, under acme.fit, a key the organisation chose.

Recommendations beyond the review

• Take the conventions out of bedrock entirely. The revised prototype keeps the signed format as an optional fit/signed subpackage and SpanEvent as an optional Observe for the core. Even optional, they're bedrock conventions, and the signed format already has the cross-language bug above. humus is an opinionated framework and is allowed to set conventions. It could ship a signed codec and the fit.* span event, leaving bedrock to supply only the mechanism. The core would then import only go.opentelemetry.io/otel/baggage.
• SpanEvent sets error.type to the full error message. OTel's semantic conventions expect a low-cardinality error class there. This is moot if the observer moves to humus.
• Optional refinement: split Codec so that services implement only Decode and Strip, while edges implement Encode. Most services never write a scenario.

Placement

With the convention-setting parts removed, "assumes no transport and no topology" is finally true, so the case for bedrock is stronger than before. Because Codec uses baggage.Baggage, the core still imports the OTel API, so a nested module (github.com/z5labs/bedrock/fit) is still needed to keep bedrock's root module standard-library only.

There is an alternative. If Codec worked on context.Context instead, the core would import only the standard library and could live in the root module. But then nothing would guarantee that the carrier is OTel baggage, which is the point of the package. The recommendation is to keep baggage.Baggage.

Revised prototype

• Builds against OpenTelemetry Go v1.47.0 and Go 1.27.
• Checks: go vet, go test -race and golangci-lint with bedrock's .golangci.yaml all pass.
• Coverage: it is prototype quality. The core has 47.7% statement coverage, and signed is exercised only by the root package's Example.

fit.go (revised core)

// Package fit evaluates request-scoped failure scenarios carried in OTel
// baggage. It owns no wire convention: a Codec, supplied by the caller,
// decides which baggage members carry a Scenario, how they are encoded and
// whether they are trusted.
package fit

import (
	"context"
	"errors"
	"strings"
	"time"

	"go.opentelemetry.io/otel/attribute"
	"go.opentelemetry.io/otel/baggage"
	"go.opentelemetry.io/otel/trace"
)

// Fault is unchanged from the proposal.
type Fault struct {
	Delay time.Duration
	Abort bool
}

var ErrInjected = errors.New("fit: injected failure")

func (f Fault) Wait(ctx context.Context) error {
	if f.Delay <= 0 {
		return nil
	}
	t := time.NewTimer(f.Delay)
	defer t.Stop()
	select {
	case <-ctx.Done():
		return context.Cause(ctx)
	case <-t.C:
		return nil
	}
}

func (f Fault) Inject(ctx context.Context) error {
	if err := f.Wait(ctx); err != nil {
		return err
	}
	if f.Abort {
		return ErrInjected
	}
	return nil
}

// Rule is unchanged, except that Point has no charset: any restriction is
// the Codec's.
type Rule struct {
	Point string
	Fault Fault
}

func (r Rule) matches(point string) bool {
	if prefix, ok := strings.CutSuffix(r.Point, "*"); ok {
		return strings.HasPrefix(point, prefix)
	}
	return r.Point == point
}

// Scenario loses Expires: expiry is a trust concern of the Codec that
// carries it, not of evaluation.
type Scenario struct {
	Rules []Rule
}

func (s Scenario) Match(point string) (Rule, bool) {
	for _, r := range s.Rules {
		if r.matches(point) {
			return r, true
		}
	}
	return Rule{}, false
}

// Codec stores a Scenario in OTel baggage and reads it back. It owns every
// wire convention: member key or keys, value grammar, version, size and
// trust. Decode reports ok=false when b carries no Scenario.
type Codec interface {
	Encode(b baggage.Baggage, s Scenario) (baggage.Baggage, error)
	Decode(b baggage.Baggage) (s Scenario, ok bool, err error)
	Strip(b baggage.Baggage) baggage.Baggage
}

func ContextWithScenario(ctx context.Context, c Codec, s Scenario) (context.Context, error) {
	b, err := c.Encode(baggage.FromContext(ctx), s)
	if err != nil {
		return ctx, err
	}
	return baggage.ContextWithBaggage(ctx, b), nil
}

func ScenarioFromContext(ctx context.Context, c Codec) (Scenario, bool, error) {
	return c.Decode(baggage.FromContext(ctx))
}

func ContextWithoutScenario(ctx context.Context, c Codec) context.Context {
	return baggage.ContextWithBaggage(ctx, c.Strip(baggage.FromContext(ctx)))
}

// Evaluation is what Evaluator.Eval reports for one injection point.
type Evaluation struct {
	Point   string
	Rule    Rule // the matching Rule, if Matched
	Matched bool
	Fault   Fault // what the injection point emulates, after MaxDelay
	Err     error // why the carried Scenario was ignored, if it was
}

// Evaluator evaluates the Scenario in a context at named injection points.
// Every field is required.
type Evaluator struct {
	Codec Codec

	// MaxDelay bounds every Delay this service honours. Longer delays are
	// clamped, so services configured differently still agree on the rest
	// of a Scenario.
	MaxDelay time.Duration

	// Observe is called for every evaluation of a request that carries a
	// Scenario, including one the Codec rejected. SpanEvent is a default.
	Observe func(ctx context.Context, e Evaluation)
}

// Eval returns the Fault point emulates. A Scenario the Codec rejects is
// reported to Observe and otherwise ignored: test metadata never fails a
// request on its own.
func (e Evaluator) Eval(ctx context.Context, point string) Fault {
	if e.Codec == nil || e.MaxDelay <= 0 || e.Observe == nil {
		panic("fit: Evaluator requires Codec, a positive MaxDelay and Observe")
	}
	s, ok, err := ScenarioFromContext(ctx, e.Codec)
	if !ok && err == nil {
		return Fault{}
	}
	if err != nil {
		e.Observe(ctx, Evaluation{Point: point, Err: err})
		return Fault{}
	}
	r, matched := s.Match(point)
	f := r.Fault
	f.Delay = min(f.Delay, e.MaxDelay)
	e.Observe(ctx, Evaluation{Point: point, Rule: r, Matched: matched, Fault: f})
	return f
}

// SpanEvent is an optional Observe that adds a "fit.evaluation" event to the
// span in ctx. Organisations with their own telemetry names pass their own.
func SpanEvent(ctx context.Context, e Evaluation) {
	attrs := []attribute.KeyValue{
		attribute.String("fit.point", e.Point),
		attribute.String("fit.rule", e.Rule.Point),
		attribute.Float64("fit.delay", e.Fault.Delay.Seconds()),
		attribute.Bool("fit.abort", e.Fault.Abort),
	}
	if e.Err != nil {
		attrs = append(attrs, attribute.String("error.type", e.Err.Error()))
	}
	trace.SpanFromContext(ctx).AddEvent("fit.evaluation", trace.WithAttributes(attrs...))
}

signed/signed.go (the original format as one codec among others)

// Package signed is bedrock's optional default fit.Codec: the proposal's
// version-1 grammar under one baggage member, with an HMAC-SHA256 envelope.
// An organisation that has its own wire convention does not import it.
package signed

import (
	"crypto/hmac"
	"crypto/sha256"
	"encoding/base64"
	"errors"
	"fmt"
	"strconv"
	"strings"
	"time"

	"go.opentelemetry.io/otel/baggage"

	"github.com/z5labs/bedrock/fit"
)

const MaxEncodedLen = 1024

var (
	ErrInvalid   = errors.New("signed: invalid scenario")
	ErrUntrusted = errors.New("signed: untrusted scenario")
	ErrExpired   = errors.New("signed: expired scenario")
)

type Key struct {
	id     string
	secret []byte
}

func NewKey(id string, secret []byte) (Key, error) {
	if id == "" || len(id) > 16 || strings.ContainsFunc(id, func(r rune) bool { return !isIDRune(r) }) {
		return Key{}, fmt.Errorf("signed: invalid key id %q", id)
	}
	if len(secret) < 32 {
		return Key{}, fmt.Errorf("signed: key %q: secret shorter than 32 bytes", id)
	}
	return Key{id: id, secret: append([]byte(nil), secret...)}, nil
}

// Codec is a fit.Codec. Every field is required.
type Codec struct {
	// Member is the baggage key that carries a Scenario. The organisation
	// chooses it; bedrock does not.
	Member string

	// Keys[0] signs; any Key verifies.
	Keys []Key

	// TTL is how long an encoded Scenario lives. Encode stamps Now()+TTL,
	// and Decode rejects an expiry more than TTL ahead, so a leaked value
	// is bounded by the verifier, not the signer.
	TTL time.Duration

	Now func() time.Time
}

var _ fit.Codec = Codec{}

func (c Codec) check() {
	if c.Member == "" || len(c.Keys) == 0 || c.TTL <= 0 || c.Now == nil {
		panic("signed: Codec requires Member, Keys, a positive TTL and Now")
	}
}

func (c Codec) Strip(b baggage.Baggage) baggage.Baggage {
	c.check()
	return b.DeleteMember(c.Member)
}

func (c Codec) Encode(b baggage.Baggage, s fit.Scenario) (baggage.Baggage, error) {
	c.check()
	rules := make([]string, len(s.Rules))
	for i, r := range s.Rules {
		if err := validate(r); err != nil {
			return b, err
		}
		rules[i] = r.Point + encodeFault(r.Fault)
	}
	k := c.Keys[0]
	exp := c.Now().Add(c.TTL).Unix()
	payload := strings.Join([]string{"1", k.id, strconv.FormatInt(exp, 10), strings.Join(rules, "|")}, "~")
	v := payload + "~" + mac(k, payload)
	if len(v) > MaxEncodedLen {
		return b, fmt.Errorf("%w: %d bytes", ErrInvalid, len(v))
	}
	m, err := baggage.NewMemberRaw(c.Member, v)
	if err != nil {
		return b, err
	}
	return b.SetMember(m)
}

func (c Codec) Decode(b baggage.Baggage) (fit.Scenario, bool, error) {
	c.check()
	m := b.Member(c.Member)
	if m.Key() == "" {
		return fit.Scenario{}, false, nil
	}
	v := m.Value()
	if len(v) > MaxEncodedLen {
		return fit.Scenario{}, true, fmt.Errorf("%w: %d bytes", ErrInvalid, len(v))
	}
	parts := strings.Split(v, "~")
	if len(parts) != 5 || parts[0] != "1" {
		return fit.Scenario{}, true, fmt.Errorf("%w: not version 1", ErrInvalid)
	}
	k, ok := c.key(parts[1])
	if !ok {
		return fit.Scenario{}, true, fmt.Errorf("%w: unknown key %q", ErrUntrusted, parts[1])
	}
	if !hmac.Equal([]byte(parts[4]), []byte(mac(k, v[:len(v)-len(parts[4])-1]))) {
		return fit.Scenario{}, true, fmt.Errorf("%w: bad signature", ErrUntrusted)
	}
	exp, err := strconv.ParseUint(parts[2], 10, 63)
	if err != nil {
		return fit.Scenario{}, true, fmt.Errorf("%w: expiry: %w", ErrInvalid, err)
	}
	now := c.Now()
	if e := time.Unix(int64(exp), 0); !now.Before(e) || e.After(now.Add(c.TTL)) {
		return fit.Scenario{}, true, fmt.Errorf("%w: expiry %s", ErrExpired, e.UTC().Format(time.RFC3339))
	}
	var s fit.Scenario
	if parts[3] == "" {
		return s, true, nil
	}
	for rs := range strings.SplitSeq(parts[3], "|") {
		r, err := decodeRule(rs)
		if err != nil {
			return fit.Scenario{}, true, err
		}
		s.Rules = append(s.Rules, r)
	}
	return s, true, nil
}

func encodeFault(f fit.Fault) string {
	ms := strconv.FormatInt(f.Delay.Milliseconds(), 10)
	switch {
	case f.Delay > 0 && f.Abort:
		return "=delay:" + ms + "+abort"
	case f.Delay > 0:
		return "=delay:" + ms
	case f.Abort:
		return "=abort"
	}
	return ""
}

// maxDelayMS keeps the conversion to time.Duration from overflowing.
const maxDelayMS = int64(time.Duration(1<<63-1) / time.Millisecond)

func decodeRule(s string) (fit.Rule, error) {
	point, fault, _ := strings.Cut(s, "=")
	r := fit.Rule{Point: point}
	d, abort := strings.CutSuffix(fault, "+abort")
	switch {
	case fault == "":
	case fault == "abort":
		r.Fault.Abort = true
	case strings.HasPrefix(d, "delay:"):
		n := d[len("delay:"):]
		ms, err := strconv.ParseInt(n, 10, 64)
		if err != nil || ms <= 0 || ms > maxDelayMS || n[0] == '+' || n[0] == '0' {
			return fit.Rule{}, fmt.Errorf("%w: rule %q: bad delay", ErrInvalid, s)
		}
		r.Fault = fit.Fault{Delay: time.Duration(ms) * time.Millisecond, Abort: abort}
	default:
		return fit.Rule{}, fmt.Errorf("%w: rule %q: unknown fault", ErrInvalid, s)
	}
	return r, validate(r)
}

// validate is this grammar's restriction, not fit's.
func validate(r fit.Rule) error {
	name, _ := strings.CutSuffix(r.Point, "*")
	if r.Point == "" || strings.ContainsFunc(name, func(c rune) bool { return !isPointRune(c) }) {
		return fmt.Errorf("%w: point %q", ErrInvalid, r.Point)
	}
	if r.Fault.Delay < 0 || r.Fault.Delay%time.Millisecond != 0 {
		return fmt.Errorf("%w: point %q: delay %s", ErrInvalid, r.Point, r.Fault.Delay)
	}
	return nil
}

func isIDRune(r rune) bool {
	return 'a' <= r && r <= 'z' || 'A' <= r && r <= 'Z' || '0' <= r && r <= '9' || r == '-' || r == '_'
}

func isPointRune(r rune) bool { return isIDRune(r) || r == '.' || r == ':' || r == '/' }

func (c Codec) key(id string) (Key, bool) {
	for _, k := range c.Keys {
		if k.id == id {
			return k, true
		}
	}
	return Key{}, false
}

func mac(k Key, payload string) string {
	h := hmac.New(sha256.New, k.secret)
	if _, err := h.Write([]byte(payload)); err != nil {
		panic(err) // hash.Hash.Write never returns an error
	}
	return base64.RawURLEncoding.EncodeToString(h.Sum(nil)[:16])
}

example_test.go (three codecs, one core)

package fit_test

import (
	"context"
	"encoding/json"
	"errors"
	"fmt"
	"time"

	"go.opentelemetry.io/otel/baggage"
	"go.opentelemetry.io/otel/propagation"

	"github.com/z5labs/bedrock/fit"
	"github.com/z5labs/bedrock/fit/signed"
)

// jsonCodec is an org's own convention: an Envoy Lua filter sets
// baggage: chaos.scenario=<percent-encoded JSON>. Trust comes from the edge
// stripping inbound baggage, so there is no signature.
type jsonCodec struct{ key string }

type jsonRule struct {
	Point   string `json:"point"`
	DelayMS int64  `json:"delay_ms"`
	Abort   bool   `json:"abort"`
}

func (c jsonCodec) Encode(b baggage.Baggage, s fit.Scenario) (baggage.Baggage, error) {
	rs := make([]jsonRule, len(s.Rules))
	for i, r := range s.Rules {
		rs[i] = jsonRule{r.Point, r.Fault.Delay.Milliseconds(), r.Fault.Abort}
	}
	v, err := json.Marshal(rs)
	if err != nil {
		return b, err
	}
	m, err := baggage.NewMemberRaw(c.key, string(v))
	if err != nil {
		return b, err
	}
	return b.SetMember(m)
}

func (c jsonCodec) Decode(b baggage.Baggage) (fit.Scenario, bool, error) {
	m := b.Member(c.key)
	if m.Key() == "" {
		return fit.Scenario{}, false, nil
	}
	var rs []jsonRule
	if err := json.Unmarshal([]byte(m.Value()), &rs); err != nil {
		return fit.Scenario{}, true, err
	}
	var s fit.Scenario
	for _, r := range rs {
		s.Rules = append(s.Rules, fit.Rule{Point: r.Point, Fault: fit.Fault{Delay: time.Duration(r.DelayMS) * time.Millisecond, Abort: r.Abort}})
	}
	return s, true, nil
}

func (c jsonCodec) Strip(b baggage.Baggage) baggage.Baggage { return b.DeleteMember(c.key) }

// registryCodec resolves acme-fault=<scenario id> from the service's own
// registry; unknown ids are untrusted.
type registryCodec map[string]fit.Scenario

func (registryCodec) Encode(b baggage.Baggage, _ fit.Scenario) (baggage.Baggage, error) {
	return b, errors.New("scenarios are minted by the registry, not by services")
}

func (c registryCodec) Decode(b baggage.Baggage) (fit.Scenario, bool, error) {
	m := b.Member("acme-fault")
	if m.Key() == "" {
		return fit.Scenario{}, false, nil
	}
	s, ok := c[m.Value()]
	if !ok {
		return fit.Scenario{}, true, fmt.Errorf("unknown scenario %q", m.Value())
	}
	return s, true, nil
}

func (registryCodec) Strip(b baggage.Baggage) baggage.Baggage { return b.DeleteMember("acme-fault") }

func report(ctx context.Context, e fit.Evaluation) {
	fmt.Printf("%s: %+v err=%v\n", e.Point, e.Fault, e.Err)
}

func Example() {
	// An edge that is not Go and cannot HMAC: header written by Envoy.
	carrier := propagation.MapCarrier{"baggage": "chaos.scenario=" +
		"%5B%7B%22point%22%3A%22http%3Aratings%22%2C%22delay_ms%22%3A250%2C%22abort%22%3Atrue%7D%5D"}
	ctx := propagation.Baggage{}.Extract(context.Background(), carrier)
	ev := fit.Evaluator{Codec: jsonCodec{key: "chaos.scenario"}, MaxDelay: time.Second, Observe: report}
	ev.Eval(ctx, "http:ratings")

	// Scenario id resolved from a registry.
	reg := registryCodec{"scn-42": {Rules: []fit.Rule{{Point: "*", Fault: fit.Fault{Abort: true}}}}}
	carrier = propagation.MapCarrier{"baggage": "acme-fault=scn-42"}
	ctx = propagation.Baggage{}.Extract(context.Background(), carrier)
	fit.Evaluator{Codec: reg, MaxDelay: time.Second, Observe: report}.Eval(ctx, "grpc:/billing.v1.Billing/Charge")

	// bedrock's default, under a key the org chose; an expired value is inert.
	now := time.Date(2026, 10, 15, 12, 0, 0, 0, time.UTC)
	k, err := signed.NewKey("k1", []byte("0123456789abcdef0123456789abcdef"))
	if err != nil {
		panic(err)
	}
	edge := signed.Codec{Member: "acme.fit", Keys: []signed.Key{k}, TTL: time.Minute, Now: func() time.Time { return now }}
	ctx, err = fit.ContextWithScenario(context.Background(), edge, fit.Scenario{Rules: []fit.Rule{{Point: "*", Fault: fit.Fault{Delay: 5 * time.Second}}}})
	if err != nil {
		panic(err)
	}
	fit.Evaluator{Codec: edge, MaxDelay: time.Second, Observe: report}.Eval(ctx, "kafka:orders")
	late := edge
	late.Now = func() time.Time { return now.Add(time.Hour) }
	f := fit.Evaluator{Codec: late, MaxDelay: time.Second, Observe: report}.Eval(ctx, "kafka:orders")
	fmt.Println("late fault:", f)
	// Output:
	// http:ratings: {Delay:250ms Abort:true} err=<nil>
	// grpc:/billing.v1.Billing/Charge: {Delay:0s Abort:true} err=<nil>
	// kafka:orders: {Delay:1s Abort:false} err=<nil>
	// kafka:orders: {Delay:0s Abort:false} err=signed: expired scenario: expiry 2026-10-15T12:01:00Z
	// late fault: {0s false}
}

Tests against the original prototype

review_internal_test.go, in package fit next to the original prototype's files in the proposal above. Each test passes, which demonstrates the behaviour it describes.

package fit

import (
	"context"
	"testing"
	"time"

	"github.com/stretchr/testify/require"
	"go.opentelemetry.io/otel/baggage"
)

var rnow = time.Date(2026, 10, 15, 12, 0, 0, 0, time.UTC)

func rin(t *testing.T, maxDelay time.Duration) Injector {
	k, err := NewKey("k1", []byte("0123456789abcdef0123456789abcdef"))
	require.NoError(t, err)
	return Injector{Keys: []Key{k}, MaxDelay: maxDelay, Now: func() time.Time { return rnow }}
}

func ctxWithRaw(t *testing.T, v string) context.Context {
	m, err := baggage.NewMemberRaw(BaggageKey, v)
	require.NoError(t, err)
	b, err := baggage.New(m)
	require.NoError(t, err)
	return baggage.ContextWithBaggage(context.Background(), b)
}

// A signer can write a delay whose decimal value is ~9 million years, and the
// Go decoder silently wraps it to 1s. Another-language decoder would read the
// literal value and reject it (or sleep forever): same signed bytes, two
// meanings.
func TestReviewDelayOverflowWraps(t *testing.T) {
	in := rin(t, 5*time.Second)
	payload := "1~k1~1792065660~a=delay:288230376151712744"
	v := payload + "~" + mac(in.Keys[0], payload)
	f, err := in.Eval(ctxWithRaw(t, v), "a")
	require.NoError(t, err)
	require.Equal(t, time.Second, f.Delay)
}

// Non-canonical numbers are accepted, so the grammar is looser than documented.
func TestReviewNonCanonicalDelay(t *testing.T) {
	in := rin(t, 5*time.Second)
	payload := "1~k1~+1792065660~a=delay:+0250"
	v := payload + "~" + mac(in.Keys[0], payload)
	f, err := in.Eval(ctxWithRaw(t, v), "a")
	require.NoError(t, err)
	require.Equal(t, 250*time.Millisecond, f.Delay)
}

// The verifier has no cap on lifetime: a signer (or a leaked signed value)
// with Expires in year 9999 is honoured until then.
func TestReviewNoVerifierTTLCap(t *testing.T) {
	in := rin(t, 5*time.Second)
	ctx, err := in.ContextWithScenario(context.Background(), Scenario{
		Rules:   []Rule{{Point: "*", Fault: Fault{Abort: true}}},
		Expires: time.Date(9999, 1, 1, 0, 0, 0, 0, time.UTC),
	})
	require.NoError(t, err)
	f, err := in.Eval(ctx, "anything")
	require.NoError(t, err)
	require.True(t, f.Abort)
}

// The edge may mint a Scenario that is already expired; the Expires check is
// one-sided.
func TestReviewMintAlreadyExpired(t *testing.T) {
	in := rin(t, 5*time.Second)
	_, err := in.ContextWithScenario(context.Background(), Scenario{Expires: rnow.Add(-time.Hour)})
	require.NoError(t, err)
}

// Async hop: a message carrying a scenario is consumed after Expires. Every
// injection point now returns an error, and the documented usage
// ("if err != nil { return nil, err }") fails the message forever: a poison
// pill, not an inert scenario.
func TestReviewExpiredScenarioIsFatalNotInert(t *testing.T) {
	edge := rin(t, 5*time.Second)
	ctx, err := edge.ContextWithScenario(context.Background(), Scenario{Expires: rnow.Add(time.Minute)})
	require.NoError(t, err)
	consumer := edge
	consumer.Now = func() time.Time { return rnow.Add(time.Hour) } // consumer lag
	_, err = consumer.Eval(ctx, "kafka:orders")
	require.ErrorIs(t, err, ErrExpired)
}

// Services with different MaxDelay disagree on the whole scenario: a 2s rule
// aimed at point "a" makes Eval fail at unrelated point "b" in a service
// whose MaxDelay is 1s.
func TestReviewHeterogeneousMaxDelayRejectsWholeScenario(t *testing.T) {
	edge := rin(t, 5*time.Second)
	ctx, err := edge.ContextWithScenario(context.Background(), Scenario{
		Rules:   []Rule{{Point: "a", Fault: Fault{Delay: 2 * time.Second}}},
		Expires: rnow.Add(time.Minute),
	})
	require.NoError(t, err)
	_, err = rin(t, time.Second).Eval(ctx, "b")
	require.ErrorIs(t, err, ErrInvalid)
}

// Any other producer of a baggage member named "fit" (a different tool, a
// different team) makes every injection point return ErrInvalid.
func TestReviewForeignFitMemberBreaksEveryPoint(t *testing.T) {
	_, err := rin(t, time.Second).Eval(ctxWithRaw(t, "enabled"), "http:anything")
	require.ErrorIs(t, err, ErrInvalid)
}

// Injection-point names outside [A-Za-z0-9._:/-] are unrepresentable, e.g.
// an OTel HTTP span name or route template, or an IPv6 server.address.
func TestReviewPointCharset(t *testing.T) {
	in := rin(t, time.Second)
	for _, p := range []string{"GET /users/{id}", "http:[::1]:8080", "nats:orders.>", "mqtt:a/+/b"} {
		_, err := in.ContextWithScenario(context.Background(), Scenario{Rules: []Rule{{Point: p}}, Expires: rnow.Add(time.Minute)})
		require.ErrorIs(t, err, ErrInvalid, p)
	}
}

// Sub-millisecond delays are rejected on the Go API even though the field is
// a time.Duration: the wire unit leaks into the model.
func TestReviewMillisecondUnitLeaks(t *testing.T) {
	in := rin(t, time.Second)
	_, err := in.ContextWithScenario(context.Background(), Scenario{
		Rules:   []Rule{{Point: "a", Fault: Fault{Delay: 1500 * time.Microsecond}}},
		Expires: rnow.Add(time.Minute),
	})
	require.ErrorIs(t, err, ErrInvalid)
}

// Expires is truncated to whole seconds on the wire, so a decoded Scenario
// expires earlier than the one written.
func TestReviewExpiresTruncated(t *testing.T) {
	in := rin(t, time.Second)
	exp := rnow.Add(1500 * time.Millisecond)
	ctx, err := in.ContextWithScenario(context.Background(), Scenario{Expires: exp})
	require.NoError(t, err)
	s, _, err := in.ScenarioFromContext(ctx)
	require.NoError(t, err)
	require.False(t, s.Expires.Equal(exp))
}

[Zaba505]

Follow-up: which calls happen at startup and which per request?

This applies to the revised API in the review comment above. The per-call costs below come from reading the code. They have not been measured yet.

In bedrock's model, "startup" means inside App.Run, before the app starts serving. Configuration comes from the Environment on the value passed to Run, and everything built there is passed explicitly to the handlers and clients that need it.

Call	When	Who
Building the Codec: baggage key names, signing keys, TTL, clock, loading a registry	Startup	Every service and the edge
fit.Evaluator{Codec, MaxDelay, Observe}	Startup	Every service
Injection point names that are fixed per client, such as a client bound to one host, or gRPC method names known at registration	Startup, when the client is built	Injection points
Scenario values the edge reuses, such as "critical services"	Startup, since they are plain data	Edge
ContextWithoutScenario on requests from untrusted callers	Per request	Edge or ingress middleware
Deciding whether a request is under test (by account, device or percentage)	Per request	Edge; this is entirely the user's own logic
ContextWithScenario	Per request. A codec that puts an expiry in its format, like signed stamping Now()+TTL, has to encode per request. A codec without time in its format can cache the encoded value internally.	Edge
Propagator Extract and Inject	Per hop	The user's own propagators, not this package
Eval, then Inject, or Wait plus a protocol-specific abort	Per outbound call, so possibly several times per request	Injection points
Observe	Per evaluation, but only on requests that carry a scenario	Called by Eval
ContextWithoutScenario before calling a third party	Per outbound call	Egress clients

Gaps this exposes in the revised API

1. Configuration is checked on every call instead of at startup. Eval re-checks its required fields and panics on every call. signed.Codec does the same in every Encode, Decode and Strip. A misconfigured service therefore starts fine and panics on its first request. The fix is a constructor that validates once:

   • fit.NewEvaluator(codec, maxDelay, observe) panics at startup on a nil codec or observer, because those are programming errors.
   • Values that come from configuration (MaxDelay, keys, TTL) are checked by constructors that return errors, so App.Run can exit with a non-zero status at startup instead of failing request by request.
   • Eval can then trust its own invariants.

2. The scenario is decoded again at every injection point. A request without a scenario costs only one baggage lookup per point, which is cheap. But a request under test that reaches N injection points is decoded N times; with signed, that is N parses and N HMAC checks. Decoding on every call also means a signed scenario can expire partway through a request, or partway through processing a Kafka message, so some injection points fault and others don't. There are two ways to fix this:

   • Memoize inside the codec, keyed by the raw baggage value. This needs no API change, and the cost only falls on requests under test. This is the recommended starting point.
   • Decode once in ingress middleware and stash the decoded scenario under an unexported, in-process context key. No wire convention is involved, but this creates a second source of truth. ContextWithoutScenario and ContextWithScenario would have to keep the stash in step with the baggage.

3. There is no concurrency contract. One Evaluator is built at startup and shared by every request goroutine, so Codec and Observe must be safe for concurrent use. A registry codec that refreshes in the background must synchronize internally. The revised docs should say this.

4. Some things are fixed at startup.

   • The clock: Now is set once at startup, for example to time.Now passed explicitly, and called on every request. This matches bedrock's rule against global state.
   • Keys: they are also fixed at startup, so rotating them takes a redeploy unless the codec reloads them itself. That is the codec's concern, and making Codec an interface is what lets a codec handle it.

[Zaba505]

Implemented in #583, #584 and #585 and released as fit/v0.22.0. The API follows the revised core in the review comment, with two changes from the follow-up: the codec is split into Decoder (Decode and Strip) and Encoder, and NewEvaluator checks its configuration once at startup. bedrock ships no codec and no observer. Error kinds (open question 3) are left out of v1. A signed codec, the fit.evaluation observer, per-protocol injection points and Kafka header carriage are left to z5labs/humus.