Data Representation and Processing Boundary

Data Representation, Processor, and Runtime Boundary

Campaign C made the ingress representation and parse-once boundary explicit. Campaign D built on that foundation with a synchronous processor contract, one production Python reference processor, explicit runtime stages, ordered batch completion, and process-local task supervision. Campaign E completed the current Python/native-ready boundary with compiled numeric target-only routing, exact immutable output bytes, the public OutputBatch result, and one numeric production egress path.

The current AISMixer data plane remains entirely Python and process-local.

Cumulative scope

Campaign C: representation

Campaign C established a stable internal representation boundary between ingress producers and the existing processing lifecycle. It made byte ownership, text provenance, scanner results, and parsed metadata explicit. It delivered:

• immutable frame representation;
• explicit payload text mode;
• bytes-native scanning;
• immutable half-open spans;
• parse-once fragment and TAG metadata; and
• the parsed assembler production entry point.

Campaign D: processor and runtime

Campaign D established:

• the synchronous DataPlaneProcessor contract;
• PythonDataPlaneProcessor as the sole production and behavioural Python reference processor;
• explicit ingress fan-in, processor, and egress stages;
• whole-frame synchronous processing before asynchronous egress;
• a private processor-to-egress completion barrier; and
• process-local fail-fast supervision of essential runtime tasks.

Campaign E: numeric routing and immutable output

Campaign E established:

• an immutable dense, zero-based numeric ID for every configured egress destination, including unnamed legacy destinations;
• complete compilation of named routes to numeric target-only matching before a routing candidate is installed;
• ProcessingSnapshot as the immutable target-only processor view;
• exact immutable bytes in every ProcessorOutput;
• OutputBatch as the public ordered processor result;
• one UTF-8 encoding for each emitted NMEA sentence; and
• unified production dispatch through Forwarder.send_to_ids().

Campaign E preserved whole-frame processing, sequential output and destination dispatch, and the non-empty-batch completion barrier.

Campaigns C through E introduced no native code, native API or ABI, bindings, multiprocessing, coordinator or worker processes, IPC, or performance claims.

The normative observable semantics remain in the behavioural contract. This page explains how the cumulative representation, processor, and runtime boundaries implement those semantics.

Production pipeline

plain UDP producers        UDPSEC producers
         \                    /
          \                  /
        immutable IngressFrame objects
                    |
           ingress input queues
                    |
              ingress fan-in
                    |
           shared processor queue
                    |
             processor stage
 compatibility coercion and one routing snapshot
                    |
 compiled match_target_ids(), when routing is enabled
                    |
           immutable ProcessingSnapshot
 generation + deduplication mode + numeric target IDs
                    |
  PythonDataPlaneProcessor.process(frame, snapshot)
 bytes-native scanning, parse-once metadata, assembly,
 processor-owned TAG context, and deduplication
                    |
         complete ordered OutputBatch
 ProcessorOutput(message: bytes, target_ids: tuple[EgressTargetId, ...])
                    |
 private completion barrier for a non-empty batch
                    |
               egress stage
                    |
 sequential Forwarder.send_to_ids() dispatch

The UDP and UDPSEC producers own normalization and frame construction. Ingress fan-in moves queue items unchanged into one shared processor queue; it does not convert, validate, route, parse, assemble, deduplicate, or send them. The processor stage establishes the accepted-frame boundary, captures one routing snapshot, resolves one target-only ProcessingSnapshot, and invokes the processor once. The processor returns the complete ordered OutputBatch before egress begins.

IngressFrame

IngressFrame is the immutable ingress representation. Conceptually it contains:

• the byte payload;
• ingress kind;
• internal routing source_id;
• a candidate alias for emitted TAG s;
• remote IP when the producer has one;
• assembler source key; and
• explicit payload text mode.

The text mode is one of:

UTF8_IGNORE
UTF8_SURROGATEPASS

Direct bytes-native frames use UTF8_IGNORE by default. Legacy text and secure text frames use UTF8_SURROGATEPASS, so lone surrogate code points survive the text-to-bytes-to-text compatibility path. Text provenance is carried by the frame rather than guessed later from its payload.

See core/ingress_frame.py for the implementation.

Producer boundaries

UDP

Plain UDP deliberately preserves its historical full-datagram normalization order:

data.decode("utf-8", errors="ignore").strip()

The normalized text is then encoded back to UTF-8 and stored in an IngressFrame. Consequences include:

• invalid UTF-8 input bytes are ignored;
• Python Unicode whitespace stripping remains compatibility behaviour;
• a datagram that normalizes to empty still produces an accepted empty frame; and
• decode, strip, and re-encode mean this producer path is not zero-copy.

An accepted empty frame still reaches the frame-level routing boundary and then produces no sentence output.

UDPSEC

After authenticated decryption and secure message validation, an NMEA payload must be a string. Secure text is not stripped. It is converted with built-in str.encode(..., errors="surrogatepass") semantics and stored in an IngressFrame.

If the NMEA payload is not a string, no frame is enqueued. The already established secure ordering remains unchanged: nonce acceptance and session activity processing occur before frame construction rejects that value. The packet does not terminate the listener, and later valid packets continue normally.

Cryptographic packet, replay, session, nonce, and trust details belong on Secure UDP and nmea_sproxy and UDPSEC Security and State.

Compatibility adapter

The legacy boundary is:

IngressEvent -> frame_from_ingress_event() -> IngressFrame

IngressEvent remains a supported compatibility surface. A valid string raw_line, including a string subclass, is adapted once with explicit legacy text provenance. A non-string payload produces no frame. Unsupported queue items, including bare strings and bytes, are also ignored.

A direct IngressFrame passes through coercion by object identity. After a direct frame is accepted or a compatibility event is adapted, both use the same routing, scanning, parsing, assembly, metadata, deduplication, and processor-and-egress pipeline.

Bytes-native scanning

Scanning operates against frame.payload as bytes. Each immutable result contains half-open byte spans:

[start, end)

The spans point into the original payload. The scanner itself does not decode or copy sentence or associated TAG text. Sentence-family acceptance and checksum-field syntax remain governed by the behavioural contract.

This is bytes-native scanning, not a claim that the complete data plane is zero-copy or bytes-native. See core/nmea_scanner.py.

Parse-once metadata

Each immutable ParsedSentence retains:

• its original IngressFrame;
• the scanner match and its spans;
• parsed NMEA fragment metadata; and
• parsed associated TAG metadata.

Only required slices are materialized as text using the frame's explicit text mode. Parsed fragment fields include declared total, ordinal, sequential ID, and channel. Relevant TAG fields include:

• TAG s, where absence and an explicit empty s: remain distinct;
• TAG c as both exact c_text and an independently parsed integer candidate; and
• TAG g as structural part, total, and group-ID text, with a separate preservable group-ID candidate.

The structural recognition of TAG g is therefore distinct from deciding whether its group ID is eligible for preservation.

A digit-like malformed timestamp such as c:² remains visible as exact text, but has no integer timestamp candidate because Python cannot convert it with int(). It cannot terminate the processor stage. Exact TAG edge cases remain in the TAG Handling page and the behavioural contract.

See core/parsed_sentence.py.

Routing scope

Configuration, route definitions, status, the control protocol, and the operator CLI continue to identify targets by external strings such as udp:aishub. Numeric IDs are an internal production boundary, not configuration values.

Forwarder owns an immutable tuple of destinations. Each position is that destination's dense, zero-based EgressTargetId; unnamed legacy destinations receive IDs as well. Named destinations additionally appear in an immutable name-to-ID mapping. These numeric values are process-local and may change after a restart if destination declaration order changes.

Before a routing candidate can be installed, every external target name is resolved and the complete immutable numeric target-only plan is compiled. Failed compilation leaves the active routing snapshot unchanged. Production matching then calls match_target_ids(frame.source_id) without target-name lookup. Route declaration order and target declaration order are preserved, and a target matched repeatedly appears once at its first occurrence.

For each accepted direct or successfully adapted frame, the processor stage acquires one immutable routing-state snapshot and derives exactly one:

ProcessingSnapshot(
    routing_generation: int,
    deduplication_mode: DeduplicationMode,
    target_ids: tuple[EgressTargetId, ...],
)

This processor view contains no RoutingTable, compiled route program, mapping, transport, queue, Future, or asyncio object. With no active routing table it uses GLOBAL deduplication and all numeric forwarder IDs. With an active table it uses PER_TARGET deduplication and the result of the one numeric match. Every accepted sentence scanned from that frame reuses the same resolved tuple.

A frame with no accepted sentences still crosses the snapshot boundary and, when routing is enabled, the numeric matching boundary, then returns an empty OutputBatch. An invalid compatibility event or unsupported queue item stops before snapshot acquisition, processor invocation, or matching.

A concurrent routing replacement affects a later accepted frame, not the frame already being processed. For multipart input spanning several frames, the frame that carries the completing fragment supplies the routing result used for completed output. Routing generation is observational and does not reset processor or deduplication state.

Empty-target distinction

The two empty-target cases have intentionally different processor semantics:

Snapshot	Current behavior
GLOBAL with an empty destination registry	Global deduplication and normal processor effects still run. A unique message is formatted as a ProcessorOutput with target_ids=(), but no datagram is dispatched.
PER_TARGET with no matched target	No global deduplication admission occurs, the output builder is not called, and the processor returns an empty OutputBatch; assembly and multipart metadata cleanup still follow their normal lifecycle.

The runtime barrier follows batch contents, not destination count. A globally unique output with an empty target tuple is still a non-empty batch and crosses the barrier; a no-match PER_TARGET result has no egress work. For exact edge cases, see the behavioural contract.

Parsed assembler boundary

Production assembly enters through feed_parsed_outcome(). The assembler uses the fragment total, ordinal, sequential ID, and channel already retained by ParsedSentence; it does not split or reparse those fields.

The parsed API and legacy string APIs feed() and feed_outcome() converge on one lifecycle implementation. Campaign A ordering, duplicate, conflict, completion, and timeout semantics remain unchanged. Campaign B limits, expiry, reset, cleanup, discarded-key, and statistics semantics also remain unchanged.

The current Python compatibility assembler still materializes the exact matched sentence span as text and stores pending and completed sentences as strings. It is not a bytes-native assembler. See assembler.py.

Synchronous processor contract

core/data_plane.py defines the public Python-side processing boundary:

ProcessingSnapshot(
    routing_generation,
    deduplication_mode,
    target_ids,
)
ProcessorOutput(
    message: bytes,
    target_ids: tuple[EgressTargetId, ...],
)
OutputBatch(
    outputs: tuple[ProcessorOutput, ...],
)
DataPlaneProcessor.process(frame, snapshot) -> OutputBatch

DataPlaneProcessor.process() is synchronous. It receives one accepted IngressFrame and one immutable target-only processing snapshot, then returns one complete ordered OutputBatch. Each ProcessorOutput represents one fully formatted emitted sentence. Its message is exact immutable bytes, and its target_ids are explicit ordered numeric egress IDs; an empty target tuple is valid. An empty OutputBatch is also a valid processor result.

These public values contain no completion Future, routing table, compiled route program, queue, transport, forwarder, or other runtime state. The synchronous contract owns no asyncio tasks and depends on no multiprocessing, IPC, native implementation, or binding.

The acknowledgement future used by current orchestration is deliberately not part of this contract. It is a private process-local runtime mechanism for ordering processor and egress work.

Python reference processor and ownership

core/python_data_plane.py contains PythonDataPlaneProcessor. It is currently the sole production data-plane processor and the behavioural Python reference implementation. It is synchronous, process-local, and long-lived within one runtime process.

The processor owns:

• its AIVDMAssembler;
• its Deduplicator;
• multipart output TAG s, c, and g contexts associated with the assembler lifecycle; and
• parsing and processing configuration.

The assembler owns multipart fragment groups. The processor's TAG contexts are separate state keyed by the exact AssemblyKey; assigning one role does not assign the other.

The processor does not own ingress queues, asyncio tasks, network sockets, forwarders, mutable RoutingState replacement, coordinator or worker processes, multiprocessing, IPC, or native bindings. The processor stage owns snapshot acquisition and serialized processor invocation. The egress stage owns asynchronous dispatch, and Forwarder owns UDP destinations and transports.

Runtime stages and completion barrier

The current runtime has one production path:

ingress producers
    -> ingress fan-in
    -> processor stage
    -> egress stage
    -> Forwarder

Ingress fan-in owns one private reader per producer queue and transports items without processing them. The processor stage rejects unsupported items before snapshot acquisition, gives each accepted frame exactly one snapshot, and calls the processor exactly once.

The processor finishes every synchronous effect for the frame and constructs the complete ordered OutputBatch before asynchronous egress starts. An empty batch completes locally. A non-empty batch crosses the private egress handoff, and the processor stage awaits its completion Future. It does not consume the next ingress item merely because egress removed the batch from its queue; it waits until the final output has been dispatched successfully.

Egress processes OutputBatch.outputs sequentially in stored order. Every production output follows the same numeric call:

await forwarder.send_to_ids(output.target_ids, output.message)

Forwarder dispatches selected destinations sequentially and passes the same immutable bytes object to each selected UDP transport. Forwarder.send() and the named Forwarder.send_to() remain compatibility APIs, but production orchestration calls neither.

A failure stops later dispatch in that batch and prevents any later frame from being processed. Processor state and effects already completed before egress are not rolled back. The private completion signal is an in-process ordering and failure barrier, not an ingress acknowledgement or confirmation that a UDP datagram reached its destination. It provides no transactional delivery, durable acknowledgement, retry, rollback, replay, or recovery.

Process-local runtime supervision

One supervisor owns every essential top-level task: plain UDP producers, UDPSEC producers, ingress fan-in, the processor stage, and the egress stage. Tasks are created from lazy, role-named specifications. The first real failure is propagated. Unexpected normal return and unexpected internal cancellation become role-named runtime failures; external cancellation is re-raised as CancelledError.

On exit, siblings are cancelled and awaited and all task outcomes are retrieved. Fan-in reader tasks cannot outlive fan-in, and empty fan-in remains idle until cancellation. If task creation fails partway through, already created tasks are cleaned and the rejected coroutine is closed. UDPSEC closes its owned socket after bind failure, runtime failure, or cancellation.

This is single-process asyncio task supervision. It is not cross-process supervision, worker restart policy, automatic service restart logic, delivery retry, or durable recovery. A systemd unit may independently restart the whole service according to its policy; that operational behavior is outside the data-plane processor contract.

Deduplication and output

Preserved observable behaviour

Campaigns C through E preserved:

• logical deduplication identity;
• global versus per-target deduplication scope;
• group-atomic multipart decisions;
• outbound TAG format;
• CRLF output;
• metadata cleanup semantics; and
• sequential, non-transactional dispatch behaviour.

Campaign D replaced processing/send interleaving with whole-frame synchronous processing before the first asynchronous send. Campaign E changed the internal target and payload representation while preserving that ordering.

Changed architectural ownership

The current ownership boundaries are:

• PythonDataPlaneProcessor owns its assembler and deduplicator;
• it owns processing configuration and multipart TAG s, c, and g contexts;
• the assembler remains authoritative for multipart fragment groups;
• the processor stage owns routing snapshot acquisition and numeric target resolution;
• the egress stage owns ordered asynchronous dispatch of each ProcessorOutput; and
• Forwarder owns the immutable numeric destination registry, UDP destinations, and transports.

The logical key remains:

• the exact extracted sentence string for a single sentence; or
• the ordinally ordered tuple of exact extracted sentence strings for a multipart group.

Ingress TAG metadata is not part of either key. Global mode makes one decision for the logical key. Routing mode makes independent decisions scoped by each numeric EgressTargetId, then emits one formatted sentence with the eligible numeric targets attached.

Immutable output construction

core.output_builder.build_output_bytes() is the sole production output builder. It delegates canonical TAG formatting and checksum calculation to the existing string-facing meta writer, appends exactly one CRLF terminator to the complete TAG-plus-NMEA sentence, and performs one explicit UTF-8 encoding.

Encoding therefore occurs once per emitted NMEA sentence. Each multipart fragment is built as its own ProcessorOutput; a multipart group is never concatenated into one network payload. At the processor boundary the final message is exact immutable bytes, not string output awaiting egress encoding.

Forwarder.send_to_ids() validates the bytes payload but performs no encoding, decoding, normalization, or per-destination payload construction. It reuses the same bytes object for every selected destination. Debug output may remove one trailing CRLF and decode a local display view with replacement, but that observational view cannot modify the network payload.

Implemented and future boundary

Implemented now

• Immutable IngressFrame values and the legacy IngressEvent compatibility adapter.
• Bytes-native scanning, immutable half-open spans, and parse-once metadata.
• The parsed assembler production path.
• The synchronous DataPlaneProcessor contract.
• PythonDataPlaneProcessor as the sole production and behavioural Python reference processor.
• Explicit ingress fan-in, processor, and egress stages.
• One immutable target-only ProcessingSnapshot per accepted frame.
• Compiled numeric target-only production matching.
• Exact immutable bytes in each ProcessorOutput.
• A complete ordered OutputBatch and process-local completion barrier.
• Unified numeric production egress through send_to_ids().
• Process-local fail-fast task supervision.

Campaign F: worker readiness

Campaign F prepares the contracts for later worker separation. Its future work includes:

• defining an end-to-end bounded-queue and backpressure policy;
• formalizing processor-instance ownership, lifecycle, and reset semantics at a future worker boundary;
• defining routing-snapshot handoff semantics;
• defining a metrics interface or boundary between stages; and
• using explicit egress-worker terminology for future forwarding workers.

The current processor already owns its state per instance, and the current runtime already resolves one snapshot per frame. Campaign F defines how those properties would remain safe across later boundaries. It does not introduce a coordinator, ingress or egress worker processes, multiprocessing, IPC, cross-process supervision, native code, bindings, or an ABI.

Later process architecture

Only a later process-architecture campaign may introduce:

• a coordinator process and dedicated ingress and egress workers;
• cross-process lifecycle supervision and failure handling;
• IPC and routing-snapshot distribution; and
• worker restart and recovery policy.

Later native implementation

A later native implementation may place a native processor and bindings behind the established contracts and execute differential-conformance checks against the Python reference. No native processor, API, ABI, binding strategy, or benchmark claim exists today.

These future items are design work, not implied functionality or a schedule. See the Roadmap.

Authoritative and implementation sources:

• README
• Behavioural contract
• Roadmap
• core/data_plane.py
• core/python_data_plane.py
• core/output_builder.py
• core/routing.py
• core/ingress_frame.py
• core/nmea_scanner.py
• core/parsed_sentence.py
• assembler.py
• forwarder.py
• aismixer.py
• aismixer_secure.py