Skip to content

Data Representation and Processing Boundary

Jump to bottom
iliyan85 edited this page Jul 26, 2026 · 5 revisions

Data Representation, Processor, and Runtime Boundary

Campaign C made the ingress representation and parse-once boundary explicit. Campaign D built on that foundation with a synchronous processor contract, one production Python reference processor, explicit runtime stages, ordered batch completion, and process-local task supervision.

The current AISMixer data plane remains entirely Python and process-local.

Cumulative scope

Campaign C: representation

Campaign C established a stable internal representation boundary between ingress producers and the existing processing lifecycle. It made byte ownership, text provenance, scanner results, and parsed metadata explicit. It delivered:

  • immutable frame representation;
  • explicit payload text mode;
  • bytes-native scanning;
  • immutable half-open spans;
  • parse-once fragment and TAG metadata; and
  • the parsed assembler production entry point.

Campaign D: processor and runtime

Campaign D established:

  • the synchronous DataPlaneProcessor contract;
  • PythonDataPlaneProcessor as the sole production and normative Python reference processor;
  • explicit ingress fan-in, processor, and egress stages;
  • a complete ordered tuple[ProcessorOutput, ...] per accepted frame;
  • a private processor-to-egress completion barrier; and
  • process-local fail-fast supervision of essential runtime tasks.

Neither campaign introduced native code, a native API or ABI, bindings, multiprocessing, coordinator or worker processes, IPC, or performance claims.

The normative observable semantics remain in the behavioural contract. This page explains how the cumulative representation, processor, and runtime boundaries implement those semantics.

Production pipeline

plain UDP producers        UDPSEC producers
         \                    /
          \                  /
        immutable IngressFrame objects
                    |
           ingress input queues
                    |
              ingress fan-in
                    |
           shared processor queue
                    |
             processor stage
 compatibility coercion and snapshot acquisition
                    |
       PythonDataPlaneProcessor
 bytes-native scanning, parse-once metadata, assembly,
 processor-owned TAG context, routing, and deduplication
                    |
 complete ordered ProcessorOutput tuple
                    |
        private completion barrier
                    |
               egress stage
                    |
      broadcast or named UDP target sends

The UDP and UDPSEC producers own normalization and frame construction. Ingress fan-in moves queue items unchanged into one shared processor queue; it does not convert, validate, route, parse, assemble, deduplicate, or send them. The processor stage establishes the accepted-frame boundary, captures one processing snapshot, and invokes the processor once. The processor returns the complete ordered output tuple before egress begins.

IngressFrame

IngressFrame is the immutable ingress representation. Conceptually it contains:

  • the byte payload;
  • ingress kind;
  • internal routing source_id;
  • a candidate alias for emitted TAG s;
  • remote IP when the producer has one;
  • assembler source key; and
  • explicit payload text mode.

The text mode is one of:

UTF8_IGNORE
UTF8_SURROGATEPASS

Direct bytes-native frames use UTF8_IGNORE by default. Legacy text and secure text frames use UTF8_SURROGATEPASS, so lone surrogate code points survive the text-to-bytes-to-text compatibility path. Text provenance is carried by the frame rather than guessed later from its payload.

See core/ingress_frame.py for the implementation.

Producer boundaries

UDP

Plain UDP deliberately preserves its historical full-datagram normalization order:

data.decode("utf-8", errors="ignore").strip()

The normalized text is then encoded back to UTF-8 and stored in an IngressFrame. Consequences include:

  • invalid UTF-8 input bytes are ignored;
  • Python Unicode whitespace stripping remains compatibility behaviour;
  • a datagram that normalizes to empty still produces an accepted empty frame; and
  • decode, strip, and re-encode mean this producer path is not zero-copy.

An accepted empty frame still reaches the frame-level routing boundary and then produces no sentence output.

UDPSEC

After authenticated decryption and secure message validation, an NMEA payload must be a string. Secure text is not stripped. It is converted with built-in str.encode(..., errors="surrogatepass") semantics and stored in an IngressFrame.

If the NMEA payload is not a string, no frame is enqueued. The already established secure ordering remains unchanged: nonce acceptance and session activity processing occur before frame construction rejects that value. The packet does not terminate the listener, and later valid packets continue normally.

Cryptographic packet, replay, session, nonce, and trust details belong on Secure UDP and nmea_sproxy and UDPSEC Security and State.

Compatibility adapter

The legacy boundary is:

IngressEvent -> frame_from_ingress_event() -> IngressFrame

IngressEvent remains a supported compatibility surface. A valid string raw_line, including a string subclass, is adapted once with explicit legacy text provenance. A non-string payload produces no frame. Unsupported queue items, including bare strings and bytes, are also ignored.

A direct IngressFrame passes through coercion by object identity. After a direct frame is accepted or a compatibility event is adapted, both use the same routing, scanning, parsing, assembly, metadata, deduplication, and processor-and-egress pipeline.

Bytes-native scanning

Scanning operates against frame.payload as bytes. Each immutable result contains half-open byte spans:

[start, end)

The spans point into the original payload. The scanner itself does not decode or copy sentence or associated TAG text. Sentence-family acceptance and checksum-field syntax remain governed by the behavioural contract.

This is bytes-native scanning, not a claim that the complete data plane is zero-copy or bytes-native. See core/nmea_scanner.py.

Parse-once metadata

Each immutable ParsedSentence retains:

  • its original IngressFrame;
  • the scanner match and its spans;
  • parsed NMEA fragment metadata; and
  • parsed associated TAG metadata.

Only required slices are materialized as text using the frame's explicit text mode. Parsed fragment fields include declared total, ordinal, sequential ID, and channel. Relevant TAG fields include:

  • TAG s, where absence and an explicit empty s: remain distinct;
  • TAG c as both exact c_text and an independently parsed integer candidate; and
  • TAG g as structural part, total, and group-ID text, with a separate preservable group-ID candidate.

The structural recognition of TAG g is therefore distinct from deciding whether its group ID is eligible for preservation.

A digit-like malformed timestamp such as c:² remains visible as exact text, but has no integer timestamp candidate because Python cannot convert it with int(). It cannot terminate the processor stage. Exact TAG edge cases remain in the TAG Handling page and the behavioural contract.

See core/parsed_sentence.py.

Routing scope

The processor stage gives each accepted direct or successfully adapted frame:

  • one immutable routing snapshot; and
  • one invocation of the configured DataPlaneProcessor.

When that snapshot has an active table, PythonDataPlaneProcessor performs one route match based on frame.source_id. Every accepted sentence scanned from that frame reuses the same table and match. A frame with no accepted sentences still crosses this boundary and returns an empty output tuple. An invalid compatibility event or unsupported queue item stops before it and therefore receives no snapshot, processor call, or match.

A concurrent routing replacement affects a later accepted frame, not the frame already being processed. For multipart input spanning several frames, the frame that carries the completing fragment supplies the routing result used for completed output.

Parsed assembler boundary

Production assembly enters through feed_parsed_outcome(). The assembler uses the fragment total, ordinal, sequential ID, and channel already retained by ParsedSentence; it does not split or reparse those fields.

The parsed API and legacy string APIs feed() and feed_outcome() converge on one lifecycle implementation. Campaign A ordering, duplicate, conflict, completion, and timeout semantics remain unchanged. Campaign B limits, expiry, reset, cleanup, discarded-key, and statistics semantics also remain unchanged.

The current Python compatibility assembler still materializes the exact matched sentence span as text and stores pending and completed sentences as strings. It is not a bytes-native assembler. See assembler.py.

Synchronous processor contract

core/data_plane.py defines the public Python-side processing boundary:

ProcessingSnapshot
RoutingDisposition
ProcessorOutput
DataPlaneProcessor.process(frame, snapshot)

DataPlaneProcessor.process() is synchronous. It receives one accepted IngressFrame and one immutable processing/routing snapshot, then returns one complete ordered tuple[ProcessorOutput, ...]. The contract owns no asyncio tasks and depends on no socket, queue, transport, forwarder, multiprocessing, IPC, native implementation, or binding.

The acknowledgement future used by current orchestration is deliberately not part of this contract. It is a private process-local runtime mechanism for ordering processor and egress work.

Python reference processor and ownership

core/python_data_plane.py contains PythonDataPlaneProcessor. It is currently the sole production data-plane processor and the normative Python reference implementation. It is synchronous, process-local, and long-lived within one runtime process.

The processor owns:

  • its AIVDMAssembler;
  • its Deduplicator;
  • multipart output TAG s, c, and g contexts associated with the assembler lifecycle; and
  • parsing and processing configuration.

The assembler owns multipart fragment groups. The processor's TAG contexts are separate state keyed by the exact AssemblyKey; assigning one role does not assign the other.

The processor does not own ingress queues, asyncio tasks, network sockets, forwarders, mutable RoutingState replacement, coordinator or worker processes, multiprocessing, IPC, or native bindings. The processor stage owns snapshot acquisition and serialized processor invocation. The egress stage owns asynchronous dispatch, and Forwarder owns UDP destinations and transports.

Runtime stages and completion barrier

The current runtime has one production path:

ingress producers
    -> ingress fan-in
    -> processor stage
    -> egress stage
    -> Forwarder

Ingress fan-in owns one private reader per producer queue and transports items without processing them. The processor stage rejects unsupported items before snapshot acquisition, gives each accepted frame exactly one snapshot, and calls the processor exactly once.

The processor finishes every synchronous effect for the frame and constructs the complete ordered output tuple before asynchronous egress starts. An empty tuple completes locally. A non-empty tuple crosses the private egress handoff, and the processor stage awaits the batch's completion future. It does not consume the next ingress item merely because egress removed the batch from its queue; it waits until the final output has been dispatched successfully.

Egress sends ProcessorOutput values sequentially in tuple order. Broadcast output uses the existing broadcast send path, and targeted output uses named target sends. A failure stops later sends in that batch and prevents any later frame from being processed. Processor state and effects already completed before egress are not rolled back.

This barrier is not transactional delivery, durable acknowledgement, retry, rollback, replay, or recovery.

Process-local runtime supervision

One supervisor owns every essential top-level task: plain UDP producers, UDPSEC producers, ingress fan-in, the processor stage, and the egress stage. Tasks are created from lazy, role-named specifications. The first real failure is propagated. Unexpected normal return and unexpected internal cancellation become role-named runtime failures; external cancellation is re-raised as CancelledError.

On exit, siblings are cancelled and awaited and all task outcomes are retrieved. Fan-in reader tasks cannot outlive fan-in, and empty fan-in remains idle until cancellation. If task creation fails partway through, already created tasks are cleaned and the rejected coroutine is closed. UDPSEC closes its owned socket after bind failure, runtime failure, or cancellation.

This is single-process asyncio task supervision. It is not cross-process supervision, worker restart policy, automatic service restart logic, delivery retry, or durable recovery. A systemd unit may independently restart the whole service according to its policy; that operational behavior is outside the data-plane processor contract.

Deduplication and output

Preserved observable behaviour

Campaign C left the established observable behaviour unchanged, and Campaign D preserved:

  • logical deduplication identity;
  • global versus per-target deduplication scope;
  • group-atomic multipart decisions;
  • outbound TAG format;
  • CRLF output;
  • metadata cleanup semantics; and
  • sequential, non-transactional delivery behaviour.

Campaign D replaced processing/send interleaving with whole-frame synchronous processing before the first asynchronous send. It did not change the exact scanning, parsing, assembly, TAG, routing, or deduplication semantics described above.

Changed architectural ownership

Campaign D changed the ownership boundaries:

  • PythonDataPlaneProcessor owns its assembler and deduplicator;
  • it owns processing configuration and multipart TAG s, c, and g contexts;
  • the assembler remains authoritative for multipart fragment groups;
  • the egress stage owns asynchronous dispatch and interprets each RoutingDisposition; and
  • Forwarder owns UDP destinations and transports, broadcast fan-out, and targeted sends.

The logical key remains:

  • the exact extracted sentence string for a single sentence; or
  • the ordinally ordered tuple of exact extracted sentence strings for a multipart group.

Ingress TAG metadata is not part of either key. Output remains string-based NMEA/TAG data in complete ProcessorOutput tuples passed to the egress stage.

Implemented and future boundary

Implemented now

  • Immutable IngressFrame values and the legacy IngressEvent compatibility adapter.
  • Bytes-native scanning, immutable half-open spans, and parse-once metadata.
  • The parsed assembler production path.
  • The synchronous DataPlaneProcessor contract.
  • PythonDataPlaneProcessor as the sole production and normative Python reference processor.
  • Explicit ingress fan-in, processor, and egress stages.
  • A complete ordered output tuple and process-local completion barrier.
  • Process-local fail-fast task supervision.

Still future

  • A native processor implementation.
  • A C or C++ API or ABI and any binding strategy.
  • Differential execution against a second native implementation.
  • Bytes-native assembler and output state.
  • Coordinator and dedicated ingress and egress worker processes.
  • Multiprocessing, IPC, and routing-snapshot distribution.
  • Cross-process supervision, worker restart, and recovery policy.
  • Benchmarks, profiling results, and native-performance claims.

These future items are design work, not implied functionality or a schedule. See the Roadmap.

Campaign C closure

The Campaign C closure snapshot is dated 2026-07-25. This Wiki audit used source branch main at commit 120ce1d81ef4eb291f73dfe646e26a2452d5fa32 (Document Campaign C closure). The normative behavioural contract records Campaign C implementation snapshot 8f3e608611bfc9e6c4f0dc92e5087618917a354d and its final full-suite baseline: 1,164 passed, 18 skipped, and 0 failed (1,182 collected). That historical count is not a promise that the suite will remain fixed.

Campaign D closure

The Campaign D closure snapshot is dated 2026-07-26.

These counts are historical Campaign D closure evidence, not a permanent promise about future suite size or results.

Authoritative and implementation sources:

Repository · Website · Community and contact · Behavioural contract · Security policy

Getting Started

Architecture and Processing

Security and Operations

Project and Reference

Clone this wiki locally