Observability and Traceability

Observability and Traceability

The project uses structured, non-secret events to connect a source revision, an OCI build, CI evidence, a Kubernetes admission decision, and runtime requests.

Logging contract

Field	Meaning
time	UTC RFC3339 timestamp
level	DEBUG, INFO, WARN, or ERROR
component	stable producer such as cmd/oci-builder, internal/oci, microvm-init, or ci
operation	stable high-level activity (build, serve, supervise, evidence-publication)
phase	position inside the operation
trace_id	correlation identifier, never an authentication token
duration_ms	elapsed phase/build duration when meaningful
message	human-readable summary

Levels are used consistently:

• DEBUG: lifecycle detail useful for diagnosis (input sizes, phase transitions, descriptor generation);
• INFO: expected state transitions and successful outcomes;
• WARN: recoverable abnormal state, such as a forwarded signal or running microVM init outside PID 1;
• ERROR: failed validation, build, child process, or poweroff.

OCI builder

cmd/oci-builder writes JSON events to stderr while preserving its existing human-readable success line on stdout. Set SECURE_OCI_TRACE_ID to propagate a pipeline identifier; otherwise the CLI creates a random 128-bit identifier.

The library exposes Options.TraceID, Options.Observer, and structured oci.Event values. It reports paths, counts, sizes, phases, durations, and digests, but never file contents, labels' secret values, environment variables, or credentials. Observer data is not written into the image, so it cannot change reproducible bytes.

Runtime request correlation

cmd/example-service accepts an X-Request-ID up to 128 characters or creates a random identifier. It returns the identifier in the response header and includes it as trace_id in the structured request log alongside status, duration, and response bytes.

At an ingress or service mesh, generate X-Request-ID once and preserve it between hops. Do not place user data, credentials, session tokens, or raw trace-context baggage in this header.

MicroVM

Kernel preparation and guest boot use timestamped phase tags. CI separates the potentially long kernel build from QEMU boot, streams the guest console, emits network-wait heartbeats, and bounds download, compile, boot, and HTTP smoke-test operations with explicit timeouts.

Artifacts include kernel-build.txt, microvm-boot.txt, the resolved kernel configuration, and traceability.json.

CI evidence

scripts/ci/write-traceability.py creates a schema-versioned JSON manifest:

• repository, commit, ref, workflow, run ID/attempt, actor, runner OS/arch;
• SHA-256 and byte size for every available evidence file;
• trace_id equal to the GitHub run ID.

Benchmark, OCI validation, compatibility, and MicroVM artifacts include this manifest. Release workflows additionally retain publication digest, signature, SBOM/provenance, and layout checksum evidence.

Operational queries

Examples for a JSON log backend:

trace_id = "<run-or-request-id>"
component = "internal/oci" AND level = "ERROR"
operation = "build" AND phase = "complete"
component = "example-service" AND duration_ms > 500
component = "microvm-init" AND level IN ("WARN", "ERROR")

Alert on build/admission errors, repeated MicroVM network-wait exhaustion, signature verification failure, and elevated service error/latency rates. Debug events may have shorter retention; security and release evidence should follow the organization's audit-retention policy.