Summary
Add one native RecordSink between the completed D2H RecordEnvelope stream and the existing destination sinks. It matches each record's producer output_id to configured routes, shares one immutable input across the matching routes, runs each route as a forward, stateless, or stateful pipeline, and submits each result to that route's destination sink.
Architecture and module I/O are defined in #124.
What changes
RecordDescriptor (native) carries output_id; bind_hook reports output name to id so routes can be configured by name.- New routing sink: route table, shared immutable input, forward executor inline on the record worker, stateless worker pool, stateful owner workers over a fixed
K x N state region with same-key ordering.
- Processors registered by name at build time (
DMI_REGISTER_*_PROCESSOR); stateful processors also register key, key range, and slot spec functions.
- Output records validated against their registered layout before submission.
- Destinations are existing
RecordSink objects, leased by the routing sink; one RecordSchema covers input and output layouts and is validated against the ClickHouse stage at construction.
- Unmatched records forward unchanged to a default destination. Full route queues block the record worker, matching the ClickHouse insert queue today.
- Checked flush drains route queues then flushes each destination; first failure latched and surfaced.
- Counters: per-route queue depth, backlog, processed, emitted, failed; per-destination records and bytes; state slot validity.
- Python: route config dataclasses, sink wrapper,
create_record_runtime(record_sink=...) unchanged.
Not in this issue
Remote or storage-side workflows, cross-rank aggregation, runtime plugin ABI, route chaining, GPU preprocessing changes, D2H scheduling, and the concrete signal processors (Q/K spectral, expert health, pathway, TwoNN), which plug in as registered processors.
Verification
Raw-only, processed-only, and raw-plus-processed fan-out; shared input lifetime; stateless concurrency and same-key serialization; overload blocking; flush, shutdown, and failure propagation; parity of a raw-only route with the direct ClickHouse sink; end-to-end through ClickHouse and the capture sink.
Summary
Add one native
RecordSinkbetween the completed D2HRecordEnvelopestream and the existing destination sinks. It matches each record's produceroutput_idto configured routes, shares one immutable input across the matching routes, runs each route as a forward, stateless, or stateful pipeline, and submits each result to that route's destination sink.Architecture and module I/O are defined in #124.
What changes
RecordDescriptor(native) carriesoutput_id;bind_hookreports output name to id so routes can be configured by name.K x Nstate region with same-key ordering.DMI_REGISTER_*_PROCESSOR); stateful processors also register key, key range, and slot spec functions.RecordSinkobjects, leased by the routing sink; oneRecordSchemacovers input and output layouts and is validated against the ClickHouse stage at construction.create_record_runtime(record_sink=...)unchanged.Not in this issue
Remote or storage-side workflows, cross-rank aggregation, runtime plugin ABI, route chaining, GPU preprocessing changes, D2H scheduling, and the concrete signal processors (Q/K spectral, expert health, pathway, TwoNN), which plug in as registered processors.
Verification
Raw-only, processed-only, and raw-plus-processed fan-out; shared input lifetime; stateless concurrency and same-key serialization; overload blocking; flush, shutdown, and failure propagation; parity of a raw-only route with the direct ClickHouse sink; end-to-end through ClickHouse and the capture sink.