Ingesting External Captures

Overview

Kronikol's report generators read one thing: RequestResponseLog entries in the in-process RequestResponseLogger store. Until 3.0.44 the only way to fill that store was a .NET capturer running in the same process. NDJSON ingestion is the language-neutral on-ramp: any capturer — an out-of-process proxy tap, a Java or Node service, a shell script tailing a log — writes one JSON object per tracked request or response, and kronikol ingest (or IngestPipeline) replays the file into a full TestRunReport.html, the data files and the component diagram.

---

The interaction format

One line per request or response. The shape is exactly the httpInteraction object Kronikol already publishes in TestRunReport.json (same camelCase names), plus the attribution fields a capturer outside the test process must supply.

{"type":"Request","method":"POST","uri":"http://localhost:8081/sidekick","serviceName":"graphql","callerName":"web","content":"{\"query\":\"query Overview { overview }\"}","headers":[{"key":"Content-Type","value":"application/json"}],"traceId":"0af7651916cd43dd8448eb211c80319c","requestResponseId":"6f1c0d8e-9b7a-4c2e-8e1a-2d3f4b5c6d7e","timestamp":"2026-08-21T10:00:01.000Z","testId":"0af7651916cd43dd8448eb211c80319c","testName":"overview › renders"}
{"type":"Response","method":"POST","uri":"http://localhost:8081/sidekick","serviceName":"graphql","callerName":"web","content":"{\"data\":{}}","headers":[{"key":"Content-Type","value":"application/json"}],"statusCode":"200","traceId":"0af7651916cd43dd8448eb211c80319c","requestResponseId":"6f1c0d8e-9b7a-4c2e-8e1a-2d3f4b5c6d7e","timestamp":"2026-08-21T10:00:01.120Z","testId":"0af7651916cd43dd8448eb211c80319c"}

Property	Required	Meaning
type	✔	Request or Response.
uri	✔	Full URI of the call (the arrow label uses path + query). Relative paths are tolerated.
serviceName / callerName	✔	The two participants (receiver / caller).
testId	✔	The correlation key — must equal the report's Scenario.Id byte-for-byte. For browser-driven runs use the test's W3C trace id.
testName		Display name (cosmetic; a tests file wins when both exist).
method		HTTP verb (GET, POST, …) or any custom label (Query, generate [gemma]).
content		Body text, already decoded and capped by the capturer.
headers		[{ "key", "value" }]. Redact secrets — or rely on --redact at ingest.
statusCode		Responses only: "200" or a custom label.
requestResponseId		Pairs a request with its response (one arrow). Any string; non-UUIDs are hashed to a stable GUID.
traceId		Groups the hops of one call chain. Defaults to requestResponseId.
timestamp		ISO-8601. Used for ordering (the diagram follows timestamp order at ingest) and loop durations.
dependencyCategory / callerDependencyCategory		A DependencyCategories value (BigQuery, AI, Redis, …) for shape/colour.
phase		Setup / Action / Unknown.
metaType		Default or Event (fire-and-forget styling).
activityTraceId / activitySpanId		W3C ids for cross-linking to Tempo/Jaeger.
trackingIgnore		true to store but not draw.
kind		ui for a user action (see below); step / assertion for diagram markers emitted by an interaction capturer (the tests NDJSON is the usual source of those). Absent = an ordinary request/response.
durationMs		ui records: the interval the action owns — calls that start inside it nest under it in the diagram.

User actions (kind: "ui")

What the person (or the browser test acting as one) did — Navigate to, Click, Fill, … — rendered as a single one-way arrow from an actor to the service, with the calls the app made in response nested beneath it. Playwright's step tree is the natural source (the sidekick reporter is the reference implementation of the planned @kronikol/playwright reporter).

{"kind":"ui","type":"Request","method":"Click \"Accept trial\"","uri":"http://localhost:4000/intelligence-pro/overview","serviceName":"web","callerName":"User","callerDependencyCategory":"User","content":"Click getByRole('button', { name: 'Accept trial' })","timestamp":"2026-08-21T10:00:06Z","durationMs":4000,"testId":"0af7651916cd43dd8448eb211c80319c","requestResponseId":"7f7c…"}

• method is the arrow label, content the note (full title / locator), callerName the actor (User by default; callerDependencyCategory: "User" gives it the actor shape and colour wherever it appears), serviceName the thing acted on.
• No response record: a user action has no reply arrow.
• durationMs should run to the next action (or the test end) — that is what makes "everything the app did after the click" nest under the click.

.NET: InteractionRecord.UserAction(testId, label, pageUrl, timestamp, durationMs, detail); RequestResponseLog.IsUserAction; DependencyCategories.User.

Unknown properties are ignored, so capturers can add their own diagnostics. Anything you want rendered goes in content (the diagram note) — e.g. fold tokens/sec or a coalescing verdict into the JSON body.

In .NET the record is Kronikol.Ingestion.InteractionRecord (FromLog, ToLog, Pair(...), ToJson, FromJson); NdjsonInteractionWriter is an IRequestResponseSink that appends lines (thread-safe, flushed per line, tail-able), and NdjsonInteractionReader reads them back.

---

The tests format (optional)

A companion NDJSON of start / step / end events supplies each scenario's outcome, duration and steps:

{"event":"start","testId":"0af7651916cd43dd8448eb211c80319c","testName":"overview › renders","feature":"overview.spec.ts","timestamp":"2026-08-21T10:00:00Z"}
{"event":"step","testId":"0af7651916cd43dd8448eb211c80319c","text":"open the overview","timestamp":"2026-08-21T10:00:00.5Z"}
{"event":"end","testId":"0af7651916cd43dd8448eb211c80319c","status":"failed","durationMs":9000,"error":"expected 1 got 2","timestamp":"2026-08-21T10:00:09Z"}

{"event":"step","testId":"…","text":"the user accepts the trial","keyword":"When","timestamp":"2026-08-21T10:00:05.5Z","durationMs":4500,"status":"passed"}
{"event":"step","testId":"…","text":"the button is clicked","level":1,"timestamp":"2026-08-21T10:00:05.9Z","status":"passed"}
{"event":"assertion","testId":"…","text":"\"trial banner\" to be visible","status":"passed","timestamp":"2026-08-21T10:00:08Z"}
{"event":"assertion","testId":"…","text":"\"customers\" to have text 42","status":"failed","error":"Expected 42, received 41","timestamp":"2026-08-21T10:00:09Z"}

• Steps draw in the diagram too. A top-level step (level 0 or absent) injects the same black hnote across <<stepDelimiter>> bar at its timestamp that Step Tracking emits (keyword text, or just text); nested steps (level > 0) are sub-steps in the step list only. status / durationMs / error fill the step row.
• assertion events draw the same green ✓ / red ✗ hnote across <<assertionNote>> that Assertion Tracking emits (a failed one shows error under the text) and appear as sub-steps of the enclosing step (StepTrackingOptions.IncludeTrackedAssertionsInStepList). The report's Show/Hide Steps and Show/Hide Assertions toggles apply exactly as for in-process tracking.
• Markers are placed by timestamp in call-tree order: under the user action in flight at that moment (a kind: "ui" record whose interval contains the timestamp), otherwise at top level — never inside a backend call.
• status accepts the Playwright / Jest / JUnit vocabularies: passed, failed, timedOut, interrupted, skipped, pending, bypassed (anything unknown is Failed).
• feature groups scenarios under one heading (a spec file, a class); without it they land in the default feature (Ingested, or --feature).
• Tests seen only in the interaction file still become scenarios (verdict Passed by default — there is no outcome information without an end record).
• Make the tests file per-run (truncate it at run start). Kronikol renders exactly what it is handed; a file accumulated across runs will render every old test too.

.NET types: TestRunRecord, NdjsonTestRunReader, FeatureSynthesizer.

---

kronikol ingest

dotnet tool install --global Kronikol.Tool
kronikol ingest ./captures --tests ./captures/tests.ndjson -o ./Reports -t "E2E run"

Option	Default
<inputs…>	—	Files, directories (searched recursively for *.ndjson / *.jsonl) or globs.
--tests <file>	—	The tests NDJSON.
-o, --output <dir>	./Reports	Output directory (absolute or relative).
--render <mode>	browserjs	browserjs (client-side, needs internet at view time), nodejs (offline SVG, needs node), local, server.
-t, --title <text>		Report title.
--feature <name>	Ingested	Feature for tests without one.
--collapse / --no-collapse, --collapse-threshold <n>	on / 2	Collapse consecutive identical calls into a loop ×N fragment.
--max-arrows <n>	unlimited	Cap pairs per diagram (… +N more calls omitted …).
--no-component-diagram		Skip ComponentDiagram.html.
--no-redact, --redact-header <name>	redact on	Capture-time redaction of credential headers during replay (Capture-Time-Redaction).
--allow-empty		Generate even when nothing was ingested.
--chronological	off	Strict timestamp order. By default each test is ordered as a call tree — each response directly after its request, calls a service made while handling a request nested inside it, siblings by request time — so concurrent calls stay readable — and collapsible, since collapsing works on adjacent pairs.
--fold-unknown <name>	off	Collect interactions whose testId is not in --tests (or all interactions when there is no tests file) into one scenario with this name — warm-ups, health probes, manual browsing, background jobs — instead of one hex-named scenario per trace.

Exit codes: 0 ok, 1 runtime failure (nothing found, malformed line), 2 usage.

Programmatic

var options = IngestPipeline.DefaultOptions();   // BrowserJs, no internal-flow, component diagram, collapse on
options.ReportsFolderPath = ".logs/kronikol";
var result = IngestPipeline.Run(new IngestRequest
{
    InteractionFiles = Directory.GetFiles(".logs/taps", "*.ndjson"),
    TestsFile = ".logs/taps/tests.ndjson",
    Options = options,
    // Optional: one scenario for everything no test caused (default: a scenario per unknown test id).
    FoldUnknownTestsInto = new UnknownTestFold("Traffic outside any test", "session"),
    // Optional: strict timeline instead of call-tree order (default true).
    CallTreeOrdering = true,
});
Console.WriteLine(result.TestRunReportHtml);

The pipeline reads everything, clears the store (ClearExistingLogs), sorts by timestamp, folds unknown tests if asked, orders each test as a call tree (CallTreeOrdering: response after request, nested calls inside, siblings by request time), normalises each log's TestName from the tests file, resets the diagram cache, synthesises Feature[] and calls ReportGenerator.CreateStandardReportsWithDiagrams. Because it goes through RequestResponseLogger.Log, RequestResponseLogger.Redaction applies during replay. The readers open files with FileShare.ReadWrite, so a capture can be ingested while its writer (a proxy tap, a test fixture) still holds it open — live/incremental reporting works.

---

Design notes

• Correlation is Scenario.Id == testId, byte-for-byte. testName is cosmetic.
• Order is enqueue order. The sequence diagram never sorts by timestamp itself; the pipeline sorts for you.
• Multiple hops of one call chain (web→graphql, graphql→data-insights) are separate pairs with the same testId — they all render in the same scenario, in time order.
• Parity. The same schema is the contract for the Java (Kronikol4J) and Node (@kronikol/*) ports — see JAVA_PORT_PLAN.md / NODE_PORT_PLAN.md in the repo.