Data Representation and Processing Boundary

Data Representation, Processor, and Runtime Boundary

Campaign C made the ingress representation and parse-once boundary explicit. Campaign D built on that foundation with a synchronous processor contract, one production Python reference processor, explicit runtime stages, ordered batch completion, and process-local task supervision. Campaign E completed the current Python/native-ready boundary with compiled numeric target-only routing, exact immutable output bytes, the public OutputBatch result, and one numeric production egress path. Campaign F completed Worker Readiness with bounded stage queues, capacity-safe snapshot binding, explicit processor lifecycle/reset ownership, and pull-based runtime observability.

The current aismixer data plane remains entirely Python and process-local.

Cumulative scope

Campaign C: representation

Campaign C established a stable internal representation boundary between ingress producers and the existing processing lifecycle. It made byte ownership, text provenance, scanner results, and parsed metadata explicit. It delivered:

• immutable frame representation;
• explicit payload text mode;
• bytes-native scanning;
• immutable half-open spans;
• parse-once fragment and TAG metadata; and
• the parsed assembler production entry point.

Campaign D: processor and runtime

Campaign D established:

• the synchronous DataPlaneProcessor contract;
• PythonDataPlaneProcessor as the sole production and behavioural Python reference processor;
• explicit ingress fan-in, processor, and egress stages;
• whole-frame synchronous processing before asynchronous egress;
• a private processor-to-egress completion barrier; and
• process-local fail-fast supervision of essential runtime tasks.

Campaign E: numeric routing and immutable output

Campaign E established:

• an immutable dense, zero-based numeric ID for every configured egress destination, including unnamed legacy destinations;
• complete compilation of named routes to numeric target-only matching before a routing candidate is installed;
• ProcessingSnapshot as the immutable target-only processor view;
• exact immutable bytes in every ProcessorOutput;
• OutputBatch as the public ordered processor result;
• one UTF-8 encoding for each emitted NMEA sentence; and
• unified production dispatch through Forwarder.send_to_ids().

Campaign E preserved whole-frame processing, sequential output and destination dispatch, and the non-empty-batch completion barrier.

Campaign F: worker readiness

Campaign F established:

• private bounded ingress queues and shared bounded processing admission;
• a bounded one-batch egress handoff and explicit process-local backpressure;
• admission-time binding of one IngressFrame and one immutable ProcessingSnapshot into ProcessingWorkItem;
• one explicitly owned, serially consumed PythonDataPlaneProcessor instance with a synchronous reset boundary;
• fresh immutable queue, processor, egress-operation, input-traffic, and output-traffic statistics; and
• read-only statistics through the local control protocol and aismixerctl.

Campaign F established bounded and observable in-process boundaries that can later become worker boundaries. It did not introduce worker processes.

Campaigns C through F introduced no native code, native API or ABI, bindings, multiprocessing, coordinator or worker processes, IPC, or performance claims.

The normative observable semantics remain in the behavioural contract. This page explains how the cumulative representation, processor, and runtime boundaries implement those semantics.

Production pipeline

plain UDP producers        UDPSEC producers
         \                    /
          \                  /
        immutable IngressFrame objects
                    |
       private bounded ingress queues
        1024 items for each input
                    |
              ingress fan-in
                    |
       bounded processing admission
              1024 items
                    |
 permit -> snapshot -> target match -> ProcessingWorkItem
                    |
             processor stage
                    |
  PythonDataPlaneProcessor.process(frame, snapshot)
 bytes-native scanning, parse-once metadata, assembly,
 processor-owned TAG context, and deduplication
                    |
         complete ordered OutputBatch
 ProcessorOutput(message: bytes, target_ids: tuple[EgressTargetId, ...])
             /                 \
         empty                 non-empty
           |                       |
 local completion      bounded egress handoff
                              1 batch
                                 |
                    private completion barrier
                                 |
                           egress stage
                                 |
              sequential Forwarder.send_to_ids() dispatch

The UDP and UDPSEC producers own normalization and frame construction. Each fan-in reader performs compatibility coercion, waits for processing capacity, and then synchronously binds the exact frame and current target-only snapshot into one immutable work item. It does not scan, parse, assemble, deduplicate, or send. The serial processor stage invokes the processor once for each work item. The processor returns the complete ordered OutputBatch before egress begins. An empty batch completes locally; only a non-empty batch enters the bounded egress handoff and waits on its private completion barrier.

IngressFrame

IngressFrame is the immutable ingress representation. Conceptually it contains:

• the byte payload;
• ingress kind;
• internal routing source_id;
• a candidate alias for emitted TAG s;
• remote IP when the producer has one;
• assembler source key; and
• explicit payload text mode.

The text mode is one of:

UTF8_IGNORE
UTF8_SURROGATEPASS

Direct bytes-native frames use UTF8_IGNORE by default. Legacy text and secure text frames use UTF8_SURROGATEPASS, so lone surrogate code points survive the text-to-bytes-to-text compatibility path. Text provenance is carried by the frame rather than guessed later from its payload.

See core/ingress_frame.py for the implementation.

Producer boundaries

UDP

Plain UDP deliberately preserves its historical full-datagram normalization order:

data.decode("utf-8", errors="ignore").strip()

The normalized text is then encoded back to UTF-8 and stored in an IngressFrame. Consequences include:

• invalid UTF-8 input bytes are ignored;
• Python Unicode whitespace stripping remains compatibility behaviour;
• a datagram that normalizes to empty still produces an accepted empty frame; and
• decode, strip, and re-encode mean this producer path is not zero-copy.

An accepted empty frame still reaches the frame-level routing boundary and then produces no sentence output.

UDPSEC

After authenticated decryption and secure message validation, an NMEA payload must be a string. Secure text is not stripped. It is converted with built-in str.encode(..., errors="surrogatepass") semantics and stored in an IngressFrame.

If the NMEA payload is not a string, no frame is enqueued. The already established secure ordering remains unchanged: nonce acceptance and session activity processing occur before frame construction rejects that value. The packet does not terminate the listener, and later valid packets continue normally.

Cryptographic packet, replay, session, nonce, and trust details belong on Secure UDP and nmea_sproxy and UDPSEC Security and State.

Compatibility adapter

The legacy boundary is:

IngressEvent -> frame_from_ingress_event() -> IngressFrame

IngressEvent remains a supported compatibility surface. A valid string raw_line, including a string subclass, is adapted once with explicit legacy text provenance. A non-string payload produces no frame. Unsupported queue items, including bare strings and bytes, are also ignored.

A direct IngressFrame passes through coercion by object identity. After a direct frame is accepted or a compatibility event is adapted, both use the same routing, scanning, parsing, assembly, metadata, deduplication, and processor-and-egress pipeline.

Bytes-native scanning

Scanning operates against frame.payload as bytes. Each immutable result contains half-open byte spans:

[start, end)

The spans point into the original payload. The scanner itself does not decode or copy sentence or associated TAG text. Sentence-family acceptance and checksum-field syntax remain governed by the behavioural contract.

This is bytes-native scanning, not a claim that the complete data plane is zero-copy or bytes-native. See core/nmea_scanner.py.

Parse-once metadata

Each immutable ParsedSentence retains:

• its original IngressFrame;
• the scanner match and its spans;
• parsed NMEA fragment metadata; and
• parsed associated TAG metadata.

Only required slices are materialized as text using the frame's explicit text mode. Parsed fragment fields include declared total, ordinal, sequential ID, and channel. Relevant TAG fields include:

• TAG s, where absence and an explicit empty s: remain distinct;
• TAG c as both exact c_text and an independently parsed integer candidate; and
• TAG g as structural part, total, and group-ID text, with a separate preservable group-ID candidate.

The structural recognition of TAG g is therefore distinct from deciding whether its group ID is eligible for preservation.

A digit-like malformed timestamp such as c:² remains visible as exact text, but has no integer timestamp candidate because Python cannot convert it with int(). It cannot terminate the processor stage. Exact TAG edge cases remain in the TAG Handling page and the behavioural contract.

See core/parsed_sentence.py.

Routing scope

Configuration, route definitions, status, the control protocol, and the operator CLI continue to identify targets by external strings such as udp:aishub. Numeric IDs are an internal production boundary, not configuration values.

Forwarder owns an immutable tuple of destinations. Each position is that destination's dense, zero-based EgressTargetId; unnamed legacy destinations receive IDs as well. Named destinations additionally appear in an immutable name-to-ID mapping. These numeric values are process-local and may change after a restart if destination declaration order changes.

Before a routing candidate can be installed, every external target name is resolved and the complete immutable numeric target-only plan is compiled. Failed compilation leaves the active routing snapshot unchanged. Production matching then calls match_target_ids(frame.source_id) without target-name lookup. Route declaration order and target declaration order are preserved, and a target matched repeatedly appears once at its first occurrence.

For each direct or successfully adapted frame, fan-in first waits for a shared processing-capacity permit. Only after capacity is available does admission acquire one immutable routing-state snapshot and derive exactly one:

ProcessingSnapshot(
    routing_generation: int,
    deduplication_mode: DeduplicationMode,
    target_ids: tuple[EgressTargetId, ...],
)

The exact frame and snapshot are then bound into one immutable ProcessingWorkItem and immediately inserted into the processing queue. This processor view contains no RoutingTable, compiled route program, mapping, transport, queue, Future, or asyncio object. With no active routing table it uses GLOBAL deduplication and all numeric forwarder IDs. With an active table it uses PER_TARGET deduplication and the result of the one numeric match. Every accepted sentence scanned from that frame reuses the same resolved tuple.

A processing-admitted frame with no accepted sentences still crosses the snapshot and, when routing is enabled, numeric matching boundaries, then returns an empty OutputBatch. An invalid compatibility event or unsupported queue item stops before processing admission, snapshot acquisition, processor invocation, or matching.

A frame waiting for processing capacity has no captured routing state, so a concurrent replacement can affect it. An admitted ProcessingWorkItem keeps its bound snapshot. For multipart input spanning several frames, the processing-admitted frame carrying the completing fragment supplies the routing result used for completed output. Routing generation is observational and does not reset processor or deduplication state.

Empty-target distinction

The two empty-target cases have intentionally different processor semantics:

Snapshot	Current behavior
GLOBAL with an empty destination registry	Global deduplication and normal processor effects still run. A unique message is formatted as a ProcessorOutput with target_ids=(), but no datagram is dispatched.
PER_TARGET with no matched target	No global deduplication admission occurs, the output builder is not called, and the processor returns an empty OutputBatch; assembly and multipart metadata cleanup still follow their normal lifecycle.

The runtime barrier follows batch contents, not destination count. A globally unique output with an empty target tuple is still a non-empty batch and crosses the barrier; a no-match PER_TARGET result has no egress work. For exact edge cases, see the behavioural contract.

Parsed assembler boundary

Production assembly enters through feed_parsed_outcome(). The assembler uses the fragment total, ordinal, sequential ID, and channel already retained by ParsedSentence; it does not split or reparse those fields.

The parsed API and legacy string APIs feed() and feed_outcome() converge on one lifecycle implementation. Campaign A ordering, duplicate, conflict, completion, and timeout semantics remain unchanged. Campaign B limits, expiry, reset, cleanup, discarded-key, and statistics semantics also remain unchanged.

The current Python compatibility assembler still materializes the exact matched sentence span as text and stores pending and completed sentences as strings. It is not a bytes-native assembler. See assembler.py.

Synchronous processor contract

core/data_plane.py defines the public Python-side processing boundary:

ProcessingSnapshot(
    routing_generation,
    deduplication_mode,
    target_ids,
)
ProcessorOutput(
    message: bytes,
    target_ids: tuple[EgressTargetId, ...],
)
OutputBatch(
    outputs: tuple[ProcessorOutput, ...],
)
DataPlaneProcessor.process(frame, snapshot) -> OutputBatch

DataPlaneProcessor.process() is synchronous. It receives one accepted IngressFrame and one immutable target-only processing snapshot, then returns one complete ordered OutputBatch. Each ProcessorOutput represents one fully formatted emitted sentence. Its message is exact immutable bytes, and its target_ids are explicit ordered numeric egress IDs; an empty target tuple is valid. An empty OutputBatch is also a valid processor result.

These public values contain no completion Future, routing table, compiled route program, queue, transport, forwarder, or other runtime state. The synchronous contract owns no asyncio tasks and depends on no multiprocessing, IPC, native implementation, or binding.

The acknowledgement future used by current orchestration is deliberately not part of this contract. It is a private process-local runtime mechanism for ordering processor and egress work.

Python reference processor and ownership

core/python_data_plane.py contains PythonDataPlaneProcessor. It is currently the sole production data-plane processor and the behavioural Python reference implementation. One main() invocation constructs exactly one long-lived instance for exactly one serial processor-stage consumer. There is no import-time global production processor.

The processor owns:

• its AIVDMAssembler;
• its Deduplicator;
• its SourceState;
• multipart output TAG s, c, and g contexts associated with the assembler lifecycle; and
• parsing and processing configuration, helper references, and processor metrics.

The assembler owns multipart fragment groups. The processor's TAG contexts are separate state keyed by the exact AssemblyKey; assigning one role does not assign the other.

The processor does not own ingress queues, asyncio tasks, network sockets, forwarders, mutable RoutingState replacement, coordinator or worker processes, multiprocessing, IPC, or native bindings. Fan-in admission owns snapshot acquisition and work-item binding; the processor stage owns serialized processor invocation. The egress stage owns asynchronous dispatch, and Forwarder owns UDP destinations and transports.

The instance's synchronous reset() clears assembler, deduplication, source, and multipart-context state in a defined order and returns an immutable discard report. It is fail-fast and non-transactional and retains cumulative processor metrics. It is not currently exposed by the control protocol or aismixerctl; routing replacement and normal shutdown do not invoke it. See State, Lifecycle, and Limits.

Runtime stages and completion barrier

The current runtime has one production path:

ingress producers
    -> private bounded ingress queues
    -> ingress fan-in and bounded processing admission
    -> processor stage
    -> bounded egress handoff
    -> egress stage
    -> Forwarder

Each ingress queue defaults to 1024 items. Fan-in owns one private reader per queue; that reader performs compatibility coercion and may hold one supported frame while waiting for the shared 1024-item processing admission. Private queued backlog is isolated between inputs, but scheduling fairness and total cross-input arrival order are not guaranteed. Once capacity is reserved, admission binds the frame and snapshot into a work item. The processor stage calls the processor exactly once for each dequeued work item.

These capacities count items, not bytes. Full aismixer stage queues wait and apply backpressure rather than dropping queued work. That policy is separate from the nmea_sproxy serial queue and supplies no UDP reliability, durability, replay, or recovery guarantee.

The processor finishes every synchronous effect for the frame and constructs the complete ordered OutputBatch before asynchronous egress starts. An empty batch completes locally. A non-empty batch crosses the private egress handoff, whose queue capacity is one batch, and the processor stage awaits its completion Future. It does not begin the next processor call merely because egress removed the batch from its queue; it waits until the final awaited local dispatch completes. Later work may already be admitted and carry a bound snapshot.

Egress processes OutputBatch.outputs sequentially in stored order. Every production output follows the same numeric call:

await forwarder.send_to_ids(output.target_ids, output.message)

Forwarder dispatches selected destinations sequentially and passes the same immutable bytes object to each selected UDP transport. Forwarder.send() and the named Forwarder.send_to() remain compatibility APIs, but production orchestration calls neither.

A failure stops later dispatch in that batch and prevents any later frame from being processed. Processor state and effects already completed before egress are not rolled back. The private completion signal is an in-process ordering and failure barrier, not an ingress acknowledgement or confirmation that a UDP datagram reached its destination. It provides no transactional delivery, durable acknowledgement, retry, rollback, replay, or recovery.

Process-local runtime supervision

One supervisor owns every essential top-level task: plain UDP producers, UDPSEC producers, ingress fan-in, the processor stage, and the egress stage. Tasks are created from lazy, role-named specifications. The first real failure is propagated. Unexpected normal return and unexpected internal cancellation become role-named runtime failures; external cancellation is re-raised as CancelledError.

On exit, siblings are cancelled and awaited and all task outcomes are retrieved. Fan-in reader tasks cannot outlive fan-in, and empty fan-in remains idle until cancellation. If task creation fails partway through, already created tasks are cleaned and the rejected coroutine is closed. UDPSEC closes its owned socket after bind failure, runtime failure, or cancellation.

This is single-process asyncio task supervision. It is not task restart, cross-process supervision, worker restart policy, automatic service restart logic, delivery retry, replay, or durable recovery. A systemd unit may independently restart the whole service according to its policy; that operational behavior is outside the data-plane processor contract.

Deduplication and output

Preserved observable behaviour

Campaigns C through F preserved:

• logical deduplication identity;
• global versus per-target deduplication scope;
• group-atomic multipart decisions;
• outbound TAG format;
• CRLF output;
• metadata cleanup semantics; and
• sequential, non-transactional dispatch behaviour.

Campaign D replaced processing/send interleaving with whole-frame synchronous processing before the first asynchronous send. Campaign E changed the internal target and payload representation while preserving that ordering.

Changed architectural ownership

The current ownership boundaries are:

• PythonDataPlaneProcessor owns its assembler, deduplicator, SourceState, processing configuration, multipart TAG s, c, and g contexts, and processor metrics;
• the assembler remains authoritative for multipart fragment groups;
• fan-in processing admission owns routing snapshot acquisition, numeric target resolution, and immutable work-item binding;
• the processor stage owns serialized processor invocation;
• the egress stage owns ordered asynchronous dispatch of each ProcessorOutput; and
• Forwarder owns the immutable numeric destination registry, UDP destinations, and transports.

The logical key remains:

• the exact extracted sentence string for a single sentence; or
• the ordinally ordered tuple of exact extracted sentence strings for a multipart group.

Ingress TAG metadata is not part of either key. Global mode makes one decision for the logical key. Routing mode makes independent decisions scoped by each numeric EgressTargetId, then emits one formatted sentence with the eligible numeric targets attached.

Immutable output construction

core.output_builder.build_output_bytes() is the sole production output builder. It delegates canonical TAG formatting and checksum calculation to the existing string-facing meta writer, appends exactly one CRLF terminator to the complete TAG-plus-NMEA sentence, and performs one explicit UTF-8 encoding.

Encoding therefore occurs once per emitted NMEA sentence. Each multipart fragment is built as its own ProcessorOutput; a multipart group is never concatenated into one network payload. At the processor boundary the final message is exact immutable bytes, not string output awaiting egress encoding.

Forwarder.send_to_ids() validates the bytes payload but performs no encoding, decoding, normalization, or per-destination payload construction. It reuses the same bytes object for every selected destination. Debug output may remove one trailing CRLF and decode a local display view with replacement, but that observational view cannot modify the network payload.

Implemented and future boundary

Implemented now

• Immutable IngressFrame values and the legacy IngressEvent compatibility adapter.
• Bytes-native scanning, immutable half-open spans, and parse-once metadata.
• The parsed assembler production path.
• The synchronous DataPlaneProcessor contract.
• PythonDataPlaneProcessor as the sole production and behavioural Python reference processor.
• Explicit ingress fan-in, processor, and egress stages.
• Private ingress queues (1024 items each), shared processing admission (1024 items), and a one-batch egress queue with process-local backpressure.
• Admission-time binding of each frame and immutable target-only ProcessingSnapshot into ProcessingWorkItem.
• Explicit per-instance processor ownership and synchronous reset semantics.
• Compiled numeric target-only production matching.
• Exact immutable bytes in each ProcessorOutput.
• A complete ordered OutputBatch and process-local completion barrier.
• Unified numeric production egress through send_to_ids().
• Process-local fail-fast task supervision.
• Pull-based queue, processor, egress-operation, input-traffic, and output-traffic statistics, exposed read-only through the optional local control plane and aismixerctl.

Campaign F closed Worker Readiness on 2026-08-09. The implemented boundaries remain inside one Python OS process. There is no coordinator, ingress, processor, or egress worker process, multiprocessing, IPC, cross-process supervision, distributed metrics, native code, binding, or ABI.

Later process architecture

Only a later process-architecture campaign may introduce:

• a coordinator process and dedicated ingress, processor, and egress workers;
• cross-process lifecycle supervision and failure handling;
• IPC and routing-snapshot distribution;
• worker restart and recovery policy; and
• cross-process metrics aggregation.

Later native implementation

A later native implementation may place a native processor and bindings behind the established contracts and execute differential-conformance checks against the Python reference. No native processor, API, ABI, binding strategy, or benchmark claim exists today.

These future items are design work, not implied functionality or a schedule. See the Roadmap and Runtime Observability.

Authoritative and implementation sources:

• README
• Behavioural contract
• Roadmap
• core/data_plane.py
• core/python_data_plane.py
• core/metrics.py
• core/runtime_statistics.py
• core/output_builder.py
• core/routing.py
• core/ingress_frame.py
• core/nmea_scanner.py
• core/parsed_sentence.py
• assembler.py
• forwarder.py
• aismixer.py
• aismixer_secure.py