Integration TcpTap Extension

The Kronikol.Extensions.TcpTap package is Kronikol's out-of-process database capture. A TcpTap is a transparent, byte-for-byte TCP tee: it listens on a port, forwards every byte to the real server unchanged, and decodes a copy of the wire protocol off the hot path into Kronikol request/response pairs. Point a service's Redis or MongoDB client at the tap instead of the database and its calls appear in the sequence diagrams — Get (Hit): redis://db0/user:123, Find ← Trial: mongodb:///app/Trial — with nothing changed inside the service.

It is the database analogue of Integration ProxyTap Extension (which tees HTTP), and it renders identically to the in-process extensions Integration Redis Extension and Integration MongoDB Extension: both taps compile the same classifier source files, so the arrow label and URI for a given command are byte-for-byte the same whichever way you captured it.

Which capture do I want? If you can change the service, use the in-process extension — it has the test's identity in hand and sees the client's own types. Use the wire tap when you cannot change the service: a polyglot stack, a vendor binary, a container you only configure through environment variables, a language with no Kronikol port yet.

---

Install

dotnet add package Kronikol.Extensions.TcpTap

The package takes no client-library dependency — no StackExchange.Redis, no MongoDB.Driver. It speaks the protocols itself and only references MongoDB.Bson to read BSON documents.

---

How it works

myDotnetService ──► redis tap :6499 ──► redis :6379
              └─► mongo tap :27199 ─► mongod :27017
                        │ decodes a copy
                        ▼
              RequestResponseLogger  /  NDJSON file  (IRequestResponseSink)

Per connection the tap:

1. Accepts the connection and opens one upstream connection to the real server.
2. Copies bytes both ways, unmodified and first. Each pump writes downstream before it queues the copy for decoding, and the queue write never blocks: the queue is bounded and a full queue drops the copy and increments SegmentsDropped. Capture can degrade; forwarding cannot stall. Half-close is propagated, so a client that shuts down its send side still receives the reply.
3. Decodes on a separate task, in wire order, into command/reply pairs — RESP2/RESP3 for Redis, OP_MSG for MongoDB.
4. Filters the handshake at the decoder, not at render (see Security below): authentication and topology chatter never reach a sink.
5. Attributes each pair (see Attribution) and logs a Request and a Response entry with the real command and reply timestamps, so the existing call-tree ordering nests the DB calls under the HTTP request that was in flight, and CollapseConsecutiveIdenticalCalls folds cache bursts into loop ×N.

A decoder that throws is caught and counted. When the error is recoverable — the decoder lost its place (the MaxBufferedBytes cap, a pending-queue overflow, a protocol error on a connection that had been decoding fine) — the tap resets it and decoding resumes at the next command boundary (DecoderResets, see Diagnostics); when the bytes are not the protocol at all, decoding is switched off for that connection only (DecodingDisabledConnections). The connection keeps forwarding either way, and neither outcome is silent: each is a Log line, an OnCaptureDegraded event and a Diagnostics() entry. A tap is never the reason a request fails.

---

Quick start

Redis

using Kronikol.Extensions.TcpTap;

await using var tap = new RedisTap(new RedisTapOptions
{
    ListenPort = 6499,
    ForwardHost = "localhost",
    ForwardPort = 6379,
    CallerName = "myDotnetService",   // the participant that dials the tap
    // ServiceName defaults to "redis", DependencyCategory to DependencyCategories.Redis
});
await tap.StartAsync();

// Point the service at :6499 instead of :6379 — it sees identical traffic.

MongoDB

await using var tap = new MongoTap(new MongoTapOptions
{
    ListenPort = 27199,
    ForwardHost = "localhost",
    ForwardPort = 27017,
    CallerName = "myDotnetService",
    // ServiceName defaults to "mongo", DependencyCategory to DependencyCategories.MongoDB
});
await tap.StartAsync();

// MongoDb__ConnectionString=mongodb://user:pw@localhost:27199/?authSource=admin

With dependency injection

services.AddRedisTapTestTracking(o =>
{
    o.ListenPort = 6499;
    o.ForwardPort = 6379;
    o.CallerName = "myDotnetService";
    o.Sink = new CompositeRequestResponseSink(
        RequestResponseLoggerSink.Instance,
        new NdjsonInteractionWriter(".logs/taps/redis.ndjson"));
});

services.AddMongoTapTestTracking(o =>
{
    o.ListenPort = 27199;
    o.ForwardPort = 27017;
    o.CallerName = "myDotnetService";
});

One IHostedService per tap starts it with the host and stops it on shutdown (TcpTapHostedService). The taps resolve as RedisTap / MongoTap, as TcpTap, or as IEnumerable<TcpTap>. Set ListenPort = 0 to bind a free ephemeral port and read tap.BoundPort — useful in tests.

---

The diagram it produces

user -[#7D3C98]> web: Open /intelligence/overview
  web -[#438DD5]> graphql: POST: /sidekick (query GetInsightsData)
    graphql -[#438DD5]> dataInsights: POST: /insights/…
      loop ×6 · 1–3 ms
      dataInsights -[#F39C12]> redis: Get (Hit): /myDotnetService-api:period:216149122232148:OneWeek:…
      end
      dataInsights -[#E74C3C]> mongo: Find ← Trial: /myDotnetService-Development/Trial

redis renders as a collections participant in cache orange and mongo as a database participant in red — the palette entries DependencyCategories.Redis / .MongoDB already carry (see Tracking Dependencies). Notes carry the key/value and the filter/documents, capped at BodyCapBytes.

---

Attribution

A wire tap has no per-request identity: the bytes on a Redis socket carry no headers, and the service that opened the socket is not the test. There are two ways to get the calls into the right scenario.

1. By test window at ingest (the default, and exact for a serial suite). Every record is stamped with FallbackTestName / FallbackTestId and a timestamp. kronikol ingest --attribute-by-window (IngestRequest.AttributeByTestWindow) then assigns each un-attributed record to the test whose [start, end] window contains its timestamp; anything outside every window lands in the fold bucket (--fold-unknown "Traffic outside any test"). See Ingesting External Captures.
2. By an in-flight registry (for concurrent suites). Set IdentityResolver to a callback that returns the identity of the request currently in flight on the calling service — a host that also runs HTTP proxy taps knows this. Returning null falls back to (1). The resolver never breaks the tap: an exception is caught and treated as null.

---

TcpTapOptions (the topology schema)

Option	Default	Meaning
ListenPort / ListenHost	— / localhost	Where the tap listens. 0 = a free ephemeral port (read BoundPort).
ForwardHost / ForwardPort	localhost / —	The real server.
CallerName / ServiceName	— / redis|mongo	The two participants this hop joins.
DependencyCategory / CallerDependencyCategory	Redis|MongoDB / null	A DependencyCategories value for shape and colour.
Verbosity	Detailed	Summarised | Detailed | Raw — see the table below.
BodyCapBytes	65536	Recorded text beyond this is truncated (…truncated (N chars total)). The forwarded bytes are never touched.
CaptureReplies	true	When false the response arrow still renders (status and timing) but carries no note.
Sink	RequestResponseLoggerSink.Instance	Where entries go; combine with NdjsonInteractionWriter via CompositeRequestResponseSink.
Phase	Unknown	TestPhase stamped on entries.
FallbackTestName / FallbackTestId	Unknown / unknown	The identity every record gets when IdentityResolver yields nothing.
IdentityResolver	null	Func<(string Name, string Id)?> — plug an in-flight registry.
EmitActivities	true	Client spans on TcpTap.ActivitySource (Kronikol.TcpTap), stamped with the real command and reply times. Free when nothing is listening.
KeyRedaction / ValueRedaction	null	Func<string,string> applied to keys and to recorded values before anything reaches a sink.
ChannelCapacity	1024	Segments buffered per connection between the pumps and the decoder. Full = drop and count.
MaxBufferedBytes	8 MiB	Undecoded bytes a decoder may hold per direction for one unfinished message. A Redis bulk payload never counts against it (payloads over MaxBulkBytes are streamed past) and a MongoDB message longer than it is skipped, so crossing it means the stream is desynchronised: the decoder is reset (ResyncAfterOverflow) or, if that is off, stops decoding the connection.
ResyncAfterOverflow	true	After a recoverable decode error, reset the decoder and resume at the next command boundary instead of disabling decoding for the rest of the connection. The first interaction after a reset is stamped [resynchronised — pairing uncertain] (+ x-kronikol-capture: resynced). Off = disable, still counted and reported.
OnCaptureDegraded	null	Action<CaptureDegradation> called as capture-loss events happen — see Diagnostics.
DecodingStallBytes	1 MiB	Heuristic stall detector for Diagnostics(): this many bytes forwarded since the last recorded interaction adds a "decoding may have stalled" entry. null = off.
ReadBufferBytes	32 KiB	Socket read size.
AcceptBacklog	128	Listener backlog.
ConnectTimeout / DrainTimeout	5 s / 2 s	Upstream connect; how long DisposeAsync waits for live connections so their captures land.
Log / Name	null	Diagnostics callback and display name (defaults to caller→service).
DecoderFactory	set by RedisTap/MongoTap	Func<TcpTapConnectionContext, IProtocolDecoder> — implement IProtocolDecoder and AddTcpTapTestTracking to tee any other protocol.

RedisTapOptions

Option	Default	Meaning
ExcludedCommands	PING, CLIENT, CONFIG, INFO, ECHO, HELLO, SELECT, AUTH, COMMAND, CLUSTER, SENTINEL, SUBSCRIBE, UNSUBSCRIBE, PSUBSCRIBE, PUNSUBSCRIBE, SSUBSCRIBE, SUNSUBSCRIBE, QUIT	Verbs never recorded. SELECT is still followed, so the database index in the URI stays right. PUBLISH is deliberately not excluded — publishing is something the application did.
ExcludedKeyPrefixes	__Booksleeve_, __redis__	Keys never recorded. StackExchange.Redis probes __Booksleeve_TieBreak on every connection it opens: a real GET, but not something your code asked for.
DefaultDatabase	0	The index a fresh connection starts on, before any SELECT.
CapturePubSub	true	Whether PUBLISH is recorded. Delivered messages are unsolicited and never recorded either way.
MaxBulkBytes	null = BodyCapBytes (else MaxBufferedBytes)	A bulk payload longer than this is never buffered: it is streamed past, keeping a preview and the length on the wire — see Values larger than the capture cap. Never above MaxBufferedBytes; EffectiveMaxBulkBytes says which cap is in force.

MongoTapOptions

Option	Default	Meaning
ExcludedCommands	hello, isMaster, ismaster, saslStart, saslContinue, saslSupportedMechs, ping, buildInfo, getParameter, getLastError, killCursors, endSessions, logout, authenticate, getnonce, whatsmyuri, connectionStatus	Commands never recorded (matches the in-process extension's IgnoredCommands, plus the auth family).
TrackGetMore	false	Whether cursor continuations are recorded.
LogFilterText	true	Include the command's filter as the request note at Detailed.
LogResponseContent	true	Include cursor.firstBatch documents in the response note.
MaxResponseDocuments	10	How many of them (… (N more documents not shown) after that).
DocumentRedaction	null	Func<string,string> applied to every recorded command and reply text.

Verbosity

	Redis	MongoDB
Summarised	Get, URI redis://db0/, no notes; unclassified verbs dropped entirely	Find, URI mongodb:///{db}, no notes
Detailed (default)	Get (Hit), URI redis://db0/{key}, value in the note	Find ← Trial, URI mongodb:///{db}/{coll}, filter in the request note, n=… / documents in the reply note
Raw	the verb (GET), URI redis://{host}:{port}/{db}/{key}	Find app.Trial filter={…}, the whole command and reply

These map one-for-one onto RedisTrackingVerbosity / MongoDbTrackingVerbosity, and the golden tests assert the labels match the in-process extension at every level.

---

What the decoders understand

Redis (RESP2 and RESP3)

• Every RESP type: simple string, error, integer, bulk string, array, null bulk / null array, and the RESP3 additions — null (_), double, boolean, big number, verbatim string, blob error, map, set and push. A connection that upgrades with HELLO 3 just starts producing the RESP3 types; the same parser reads both.
• Commands are arrays of bulk strings (a legacy inline command line is accepted too). Replies are matched FIFO per connection — Redis answers a pipelined connection strictly in order. Excluded commands still take a slot in the queue, because their replies still occupy one on the wire.
• Hit/miss is decided on the reply, with the in-process extension's rule: a null bulk is a miss, an aggregate is a hit when any element is non-null, an error is a miss (and a 500).
• SELECT n is followed per connection, so redis://db{n}/… is right after the client switches databases. A failed SELECT does not move the index.
• Multi-key commands (MGET, DEL, UNLINK, EXISTS, TOUCH, WATCH, MSET) put every key in the URI joined with commas — redis://db0/a,b,c. That is exactly what the in-process extension does for a RedisKey[] argument, so both captures produce the same URI for the same operation.
• Notes: the value for a string write, field=value for a hash write, the message for a publish, the elements for a list/set push; the reply value (aggregates rendered as [a, b], maps as {k=v}).
• Unsolicited traffic — RESP3 pushes, and RESP2 arrays headed message/pmessage/smessage/invalidate — is skipped without consuming a pending command, so a subscription connection cannot desynchronise the queue.
• An error reply becomes StatusCode 500 with the error text as the note.

Values larger than the capture cap

The decoder is a streaming parser (RespStreamParser): it never needs a whole value in memory. A bulk payload — a SET value, a GET reply, any element of an MGET array — longer than RedisTapOptions.MaxBulkBytes (default = BodyCapBytes, 64 KB) is consumed segment by segment as it passes: the first bytes are kept as a preview, the rest are counted and let go. The interaction is still recorded exactly where it belongs in the FIFO, a GET of a 10 MB value is still a Get (Hit), and the note is the preview followed by

 …[bulk string truncated: 4,400,751 bytes on the wire, 65,408 kept]

(the preview leaves room for the marker under BodyCapBytes, so the record-time cap never cuts it off). RespValue.Truncated / DeclaredLength carry the same facts for code that reads values directly. Each such payload increments OversizePayloadsSkipped / BytesSkipped and raises an OversizePayloadSkipped event — it is a normal, expected thing for a cache of big JSON blobs, reported so you can see it happened. Memory per connection is bounded by the caps, not by the data; MaxBufferedBytes only bounds the bytes held for an unfinished value (a header line, sub-cap payloads, the elements of an open aggregate), which is why crossing it now means "desynchronised stream" rather than "big value".

MongoDB has the equivalent: a message longer than MaxBufferedBytes (a big find batch, a bulk insert) is skipped by its header's messageLength; a skipped reply still closes its arrow with the note [reply of N bytes skipped — larger than the capture cap], a skipped command records nothing (its $db/collection are inside the skipped BSON) but is counted and reported.

MongoDB (OP_MSG, with the legacy handshake passed through)

• The 16-byte header (messageLength, requestID, responseTo, opCode), OP_MSG flag bits (checksumPresent, moreToCome, exhaustAllowed), section kind 0 (body) and section kind 1 (document sequences). Kind-1 sequences are folded into the body under their identifier, so an insert's documents sequence classifies exactly like an inline documents array. A trailing checksum is skipped, never validated.
• Replies are matched by responseTo, so out-of-order answers are fine. A moreToCome reply that answers nothing — the streaming hello on a monitoring connection — is skipped. A moreToCome command (an unacknowledged w:0 write) is recorded immediately, since no reply is coming.
• {db} in the URI is the command's $db on the wire — whatever the application configured, environment suffix and all.
• ok: 0 or an errmsg becomes StatusCode 500 with the errmsg as the note.
• OP_QUERY / OP_REPLY (the legacy connection handshake, which every driver still opens with) are recognised, stepped over and recorded never; the first one is reported through Log.
• OP_COMPRESSED is passed straight through and not decoded — the first one is reported through Log with the compressor id and the advice to remove compressors= from the client's connection string if you want those commands captured.

---

Security

The tap is designed so a credential cannot reach a report even by accident.

• The tap never learns a connection string. It sees bytes on an already-open socket. The Mongo URI it records is mongodb:///{db}/{coll} — there is no host, no user and no password in it, because the tap does not know them.
• Authentication is hard-excluded in the decoder, not filtered at render: Redis AUTH, HELLO and RESET; MongoDB saslStart, saslContinue, saslSupportedMechs, authenticate, getnonce, copydbsaslstart, copydbgetnonce, createUser, updateUser. Clearing ExcludedCommands does not re-enable them. Nothing from those commands is stored, so it can never appear in the in-memory store, TestRunReport.json, or an NDJSON file.
• Redaction runs at capture. KeyRedaction, ValueRedaction and DocumentRedaction are applied before the entry reaches the sink — the same boundary Capture-Time-Redaction describes for headers. A hook that throws yields [REDACTION FAILED] rather than the raw value.
• Values are capped at BodyCapBytes (64 KB by default), so a large cached payload cannot inflate the report.

Verified end to end against real servers: with mongo:7 requiring SCRAM authentication, the password appears in no recorded URI or note, and no hello/sasl* command is recorded at all.

---

What the real clients actually put on the wire

Measured through the tap against redis:7 and mongo:7 (these are the sequences the default exclusions are built from):

StackExchange.Redis 2.6.48 opens two connections (interactive + subscription) and sends:

CLIENT, CLIENT, CONFIG, CONFIG, SENTINEL, INFO, INFO, ECHO, CLUSTER, SUBSCRIBE,
GET __Booksleeve_TieBreak, ECHO, INFO, <your commands>, QUIT, QUIT

No HELLO — this client version speaks RESP2 by default, so the RESP3 support in the parser is there for newer clients and for a server that pushes RESP3 types. Everything the client sends of its own accord is covered by ExcludedCommands / ExcludedKeyPrefixes, so a tapped service contributes no handshake noise to a diagram.

MongoDB.Driver 2.30.0 against a standalone mongo:7:

OP_QUERY admin.$cmd (isMaster) ×3   ← one per connection: the legacy handshake, still OP_QUERY
OP_MSG   hello ($db=admin)          ← the monitoring connection's streaming hello
OP_QUERY admin.$cmd (saslContinue)  ← SCRAM, also legacy framing
OP_MSG   find ($db=<your database>) ← from here on, everything is OP_MSG

Two things this settles: the driver still opens with the legacy OP_QUERY hello (so a decoder that only understands OP_MSG must pass OP_QUERY through rather than choke on it), and against a standalone the driver keeps dialling the seed address it was given — every command arrives at the tap. No compression is negotiated unless the connection string asks for it.

---

Same process or different process?

RequestResponseLogger is a process-wide static store, so a tap running in the same process as report generation (a test host, an orchestrator) can log directly — the default sink. A tap running in another process cannot reach that store: give it an NdjsonInteractionWriter sink and replay the file with kronikol ingest (see Ingesting External Captures). Use both at once for a live report and a replayable artifact:

using var file = new NdjsonInteractionWriter(".logs/taps/redis.ndjson");
o.Sink = new CompositeRequestResponseSink(RequestResponseLoggerSink.Instance, file);

---

Legible diagrams for chatty caches

A warm cache produces long runs of identical arrows. Enable collapsing on the report: ReportConfigurationOptions.CollapseConsecutiveIdenticalCalls = true (+ CollapseThreshold, MaxArrowsPerDiagram) — see Report Configuration. Because the tap records the real command and reply timestamps, the collapsed loop label carries the true min–max ms range and the calls nest under the HTTP request that made them.

---

Diagnostics

Capture loss is never silent. A tap can lose capture (never forwarding) in a handful of ways, and each one is a counter, a Log line, an OnCaptureDegraded event as it happens, and an entry in Diagnostics() afterwards — so the fact reaches the report, not just a log file.

Counters

Counter	Meaning	Healthy value
ConnectionsAccepted / LiveConnections	Connections through the tap, ever / now.	—
InteractionsCaptured	Command/reply pairs recorded.	growing while traffic flows
BytesClientToServer / BytesServerToClient	Bytes forwarded each way.	—
OversizePayloadsSkipped / BytesSkipped / LargestOversizePayload	Payloads longer than the cap that were streamed past (Redis) or skipped (MongoDB); their interactions were still recorded, as previews.	any — expected for big cache values
DecoderResets	Times a decoder lost its place and was reset to resume at the next command boundary (ResyncAfterOverflow). Interactions after a reset may be mis-paired until the connection goes idle; the first is stamped.	0
DecodingDisabledConnections	Connections whose decoding was abandoned for good (not the protocol, resync off, or eight resets without recording anything). Everything on them after that moment is missing from the diagrams.	0
SegmentsDropped (+ per direction)	Byte segments the decode queue could not take (ChannelCapacity); the decoder could not keep up. Interactions on that connection may be missing or mis-paired.	0
ConnectionsClosedMidMessage	Connections that closed with a command unanswered or a message only partly received; the last interaction(s) were not recorded.	0 (a few at teardown are normal)
DecodeErrors	Every decode problem of any kind — the specific counters above say which.	0
LastInteractionAt / BytesSinceLastInteraction	When something was last recorded, and how much has flowed since — the stall detector's input.	—

OnCaptureDegraded

TcpTapOptions.OnCaptureDegraded : Action<CaptureDegradation> is called on the decoder task as each event happens, with CaptureDegradation(Tap, ConnectionId, Kind, Detail) and Kind one of OversizePayloadSkipped, DecoderReset, DecodingDisabled, SegmentsDropped (at most once per connection per minute — it is the hot path), ConnectionClosedMidMessage. Use it to flag the tap on a dashboard the moment it happens — if (d.Kind is DecodingDisabled) mark the tap degraded — rather than at report time. An exception thrown by the callback is caught and logged.

Diagnostics()

TcpTap.Diagnostics() returns IReadOnlyList<DiagnosticEntry> (kind DiagnosticKind.CaptureDegraded): one entry per non-zero counter, worded for the report —

tap-di-redis: decoding disabled on 1 connection(s) — redis arrows on them after 14:03:35Z are missing (TapProtocolException: …)
tap-di-redis: 3 oversize payload(s) streamed past (largest 9,123,456 B, 27,174,912 B not kept) — values recorded as previews
tap-di-redis: decoder reset 1 time(s) after a desynchronised stream — interactions after a reset may be mis-paired …
tap-di-redis: 12 segment(s) dropped because the decode queue was full (…) — interactions may be missing or mis-paired; forwarding was unaffected
tap-di-redis: 2 connection(s) closed mid-message — their last command(s) were not recorded

— plus a heuristic stall entry when at least DecodingStallBytes (default 1 MiB) have been forwarded since the last recorded interaction (… B have flowed since the last recorded interaction at 15:45:56Z — decoding may have stalled, or the traffic is all excluded chatter). Empty when capture was complete. Collect the lists from every tap before the ingest and hand them in as host diagnostics so they land in IngestResult.Diagnostics and the report — see Diagnostics and Debugging.

---

Limitations

• OP_COMPRESSED is not decoded. A MongoDB connection string with compressors= gives you forwarding without capture on that connection (reported once through Log).
• TLS is not terminated. The tap tees bytes; an encrypted connection is forwarded intact and decodes to nothing. Tap the plaintext hop, or terminate TLS elsewhere.
• Replica sets and sharded clusters. A standalone advertises no other address, so the driver keeps talking to the tap. A replica set's hello returns the members' real addresses (hosts, me) and the driver will connect to those, routing around the tap. The supported shape is a standalone (or a directConnection=true client whose seed is the tap).
• Exhaust cursors (exhaustAllowed with a stream of moreToCome replies) are forwarded and not paired.
• One decoder per connection, and resync is a best effort. A decoder that loses its place (a dropped segment, a pending-queue overflow, the MaxBufferedBytes cap) is reset and re-arms at the next command boundary — a segment starting with *<n>\r\n$ for Redis, a plausible message header for MongoDB. Replies have no boundary marker, so a reply still in flight at the reset may pair with the wrong command until the connection goes idle; the first interaction after a reset is stamped [resynchronised — pairing uncertain] to keep that honest, and DecoderResets counts it. Eight resets without a single interaction recorded in between give up on the connection (DecodingDisabledConnections). A tap cannot join a connection that was already open before it started.
• The tap is a participant in the path: latencies include one extra loopback hop (small, but not zero).

---

See also

• Integration ProxyTap Extension — the same idea for HTTP hops.
• Integration Redis Extension / Integration MongoDB Extension — the in-process capture the tap is label-compatible with.
• Ingesting External Captures — replaying the NDJSON, and window attribution.
• Capture-Time-Redaction — where the security boundary sits.