Ingesting External Captures

Overview

Kronikol's report generators read one thing: RequestResponseLog entries in the in-process RequestResponseLogger store. Until 3.0.44 the only way to fill that store was a .NET capturer running in the same process. NDJSON ingestion is the language-neutral on-ramp: any capturer — an out-of-process proxy tap, a Java or Node service, a shell script tailing a log — writes one JSON object per tracked request or response, and kronikol ingest (or IngestPipeline) replays the file into a full TestRunReport.html, the data files and the component diagram.

---

The interaction format

One line per request or response. The shape is exactly the httpInteraction object Kronikol already publishes in TestRunReport.json (same camelCase names), plus the attribution fields a capturer outside the test process must supply.

{"type":"Request","method":"POST","uri":"http://localhost:8081/sidekick","serviceName":"graphql","callerName":"web","content":"{\"query\":\"query Overview { overview }\"}","headers":[{"key":"Content-Type","value":"application/json"}],"traceId":"0af7651916cd43dd8448eb211c80319c","requestResponseId":"6f1c0d8e-9b7a-4c2e-8e1a-2d3f4b5c6d7e","timestamp":"2026-08-21T10:00:01.000Z","testId":"0af7651916cd43dd8448eb211c80319c","testName":"overview › renders"}
{"type":"Response","method":"POST","uri":"http://localhost:8081/sidekick","serviceName":"graphql","callerName":"web","content":"{\"data\":{}}","headers":[{"key":"Content-Type","value":"application/json"}],"statusCode":"200","traceId":"0af7651916cd43dd8448eb211c80319c","requestResponseId":"6f1c0d8e-9b7a-4c2e-8e1a-2d3f4b5c6d7e","timestamp":"2026-08-21T10:00:01.120Z","testId":"0af7651916cd43dd8448eb211c80319c"}

Property	Required	Meaning
type	✔	Request or Response.
uri	✔	Full URI of the call (the arrow label uses path + query). Relative paths are tolerated.
serviceName / callerName	✔	The two participants (receiver / caller).
testId	✔	The correlation key — must equal the report's Scenario.Id byte-for-byte. For browser-driven runs use the test's W3C trace id.
testName		Display name (cosmetic; a tests file wins when both exist).
method		HTTP verb (GET, POST, …) or any custom label (Query, generate [gemma]).
content		Body text, already decoded and capped by the capturer.
headers		[{ "key", "value" }]. Redact secrets — or rely on --redact at ingest.
statusCode		Responses only: "200" or a custom label.
requestResponseId		Pairs a request with its response (one arrow). Any string; non-UUIDs are hashed to a stable GUID.
traceId		Groups the hops of one call chain. Defaults to requestResponseId.
timestamp		ISO-8601. Used for ordering (the diagram follows timestamp order at ingest) and loop durations.
dependencyCategory / callerDependencyCategory		A DependencyCategories value (BigQuery, AI, Redis, …) for shape/colour.
phase		Setup / Action / Unknown.
metaType		Default or Event (fire-and-forget styling).
activityTraceId / activitySpanId		W3C ids for cross-linking to Tempo/Jaeger.
trackingIgnore		true to store but not draw.

Unknown properties are ignored, so capturers can add their own diagnostics. Anything you want rendered goes in content (the diagram note) — e.g. fold tokens/sec or a coalescing verdict into the JSON body.

In .NET the record is Kronikol.Ingestion.InteractionRecord (FromLog, ToLog, Pair(...), ToJson, FromJson); NdjsonInteractionWriter is an IRequestResponseSink that appends lines (thread-safe, flushed per line, tail-able), and NdjsonInteractionReader reads them back.

---

The tests format (optional)

A companion NDJSON of start / step / end events supplies each scenario's outcome, duration and steps:

{"event":"start","testId":"0af7651916cd43dd8448eb211c80319c","testName":"overview › renders","feature":"overview.spec.ts","timestamp":"2026-08-21T10:00:00Z"}
{"event":"step","testId":"0af7651916cd43dd8448eb211c80319c","text":"open the overview","timestamp":"2026-08-21T10:00:00.5Z"}
{"event":"end","testId":"0af7651916cd43dd8448eb211c80319c","status":"failed","durationMs":9000,"error":"expected 1 got 2","timestamp":"2026-08-21T10:00:09Z"}

• status accepts the Playwright / Jest / JUnit vocabularies: passed, failed, timedOut, interrupted, skipped, pending, bypassed (anything unknown is Failed).
• feature groups scenarios under one heading (a spec file, a class); without it they land in the default feature (Ingested, or --feature).
• Tests seen only in the interaction file still become scenarios (verdict Passed by default — there is no outcome information without an end record).
• Make the tests file per-run (truncate it at run start). Kronikol renders exactly what it is handed; a file accumulated across runs will render every old test too.

.NET types: TestRunRecord, NdjsonTestRunReader, FeatureSynthesizer.

---

kronikol ingest

dotnet tool install --global Kronikol.Tool
kronikol ingest ./captures --tests ./captures/tests.ndjson -o ./Reports -t "E2E run"

Option	Default
<inputs…>	—	Files, directories (searched recursively for *.ndjson / *.jsonl) or globs.
--tests <file>	—	The tests NDJSON.
-o, --output <dir>	./Reports	Output directory (absolute or relative).
--render <mode>	browserjs	browserjs (client-side, needs internet at view time), nodejs (offline SVG, needs node), local, server.
-t, --title <text>		Report title.
--feature <name>	Ingested	Feature for tests without one.
--collapse / --no-collapse, --collapse-threshold <n>	on / 2	Collapse consecutive identical calls into a loop ×N fragment.
--max-arrows <n>	unlimited	Cap pairs per diagram (… +N more calls omitted …).
--no-component-diagram		Skip ComponentDiagram.html.
--no-redact, --redact-header <name>	redact on	Capture-time redaction of credential headers during replay (Capture-Time-Redaction).
--allow-empty		Generate even when nothing was ingested.
--chronological	off	Strict timestamp order. By default each response is placed directly after its request (pairs ordered by request time), so concurrent calls stay readable — and collapsible, since collapsing works on adjacent pairs.
--fold-unknown <name>	off	Collect interactions whose testId is not in --tests (or all interactions when there is no tests file) into one scenario with this name — warm-ups, health probes, manual browsing, background jobs — instead of one hex-named scenario per trace.

Exit codes: 0 ok, 1 runtime failure (nothing found, malformed line), 2 usage.

Programmatic

var options = IngestPipeline.DefaultOptions();   // BrowserJs, no internal-flow, component diagram, collapse on
options.ReportsFolderPath = ".logs/kronikol";
var result = IngestPipeline.Run(new IngestRequest
{
    InteractionFiles = Directory.GetFiles(".logs/taps", "*.ndjson"),
    TestsFile = ".logs/taps/tests.ndjson",
    Options = options,
    // Optional: one scenario for everything no test caused (default: a scenario per unknown test id).
    FoldUnknownTestsInto = new UnknownTestFold("Traffic outside any test", "session"),
    // Optional: strict timeline instead of response-after-request pairing (default true).
    PairResponsesWithRequests = true,
});
Console.WriteLine(result.TestRunReportHtml);

The pipeline reads everything, clears the store (ClearExistingLogs), sorts by timestamp, folds unknown tests if asked, places each response directly after its request (PairResponsesWithRequests), normalises each log's TestName from the tests file, resets the diagram cache, synthesises Feature[] and calls ReportGenerator.CreateStandardReportsWithDiagrams. Because it goes through RequestResponseLogger.Log, RequestResponseLogger.Redaction applies during replay. The readers open files with FileShare.ReadWrite, so a capture can be ingested while its writer (a proxy tap, a test fixture) still holds it open — live/incremental reporting works.

---

Design notes

• Correlation is Scenario.Id == testId, byte-for-byte. testName is cosmetic.
• Order is enqueue order. The sequence diagram never sorts by timestamp itself; the pipeline sorts for you.
• Multiple hops of one call chain (web→graphql, graphql→data-insights) are separate pairs with the same testId — they all render in the same scenario, in time order.
• Parity. The same schema is the contract for the Java (Kronikol4J) and Node (@kronikol/*) ports — see JAVA_PORT_PLAN.md / NODE_PORT_PLAN.md in the repo.