4.2 Batch Data Types

Batch Data Types

Relevant source files

The following files were used as context for generating this wiki page:

• src/models/config.rs
• src/models/embeddings.rs
• src/models/gradients.rs
• src/models/metadata.rs
• src/models/spikes.rs
• src/models/traces.rs

This page provides a technical reference for the data structures used to encapsulate neural activity, training gradients, and configuration updates within the corpus-ipc framework. These types serve as the payload for the IpcMessage protocol, facilitating communication between the Rust compute environment and external backends.

Overview of Batching Strategy

The corpus-ipc library utilizes a batch-centric approach to data transmission to minimize IPC overhead. Every data structure designed for neural signal transport includes mechanisms for session isolation and temporal correlation.

Core Identification Fields

Most batch types share two critical fields for managing asynchronous data flow:

1. session_id: Used for concurrent experiment isolation. In input batches (Rust → External), this is an Option<String> src/models/spikes.rs:31, where None signifies a global context. In output batches (External → Rust), it is a mandatory String src/models/traces.rs:28 or Option<String> to ensure data is routed back to the correct experiment or session.
2. batch_id: A u64 unique identifier used to correlate outputs (like gradients or traces) with the original input spikes or embeddings that triggered them src/models/spikes.rs:33, src/models/traces.rs:30, src/models/embeddings.rs:19.

Data Flow and Code Entity Mapping

The following diagram illustrates how high-level neural concepts map to specific Rust structs defined in the codebase.

Neural Space to Code Entity Mapping

graph TD
    subgraph "Natural Language Space"
        A["Neural Spiking Activity"]
        B["Latent Representations"]
        C["Learning Gradients"]
        D["Credit Assignment"]
        E["System Tuning"]
    end

    subgraph "Code Entity Space (src/models/)"
        A --> S["struct SpikeBatch"]
        S --> SE["struct SpikeEvent"]
        
        B --> EB["struct EmbeddingBatch"]
        
        C --> GB["struct GradientBatch"]
        GB --> GU["struct GradientUpdate"]
        
        D --> TB["struct TraceBatch"]
        TB --> TD["struct TraceData"]
        
        E --> CP["struct ConfigPayload"]
        CP --> CV["enum ConfigValue"]
    end

Loading

Sources: src/models/spikes.rs:14-100, src/models/traces.rs:13-93, src/models/embeddings.rs:12-24, src/models/gradients.rs:15-89, src/models/config.rs:14-114

---

Spike and Embedding Input Types

These types represent the primary feed-forward signals sent from the Rust compute layer to the external runtime via the IpcMessage enum.

SpikeBatch, IpcSpikeBatch, and SpikeEvent

SpikeBatch aggregates discrete neural events. Each SpikeEvent represents a single activation on a specific channel src/models/spikes.rs:14-39.

To avoid name collisions with in-memory training batches of spike trains (spikes / times / targets) that other libraries also call SpikeBatch (the rustdoc stopped naming SynapticDistill.jl in #68), corpus-ipc defines an explicit type alias IpcSpikeBatch = SpikeBatch src/models/spikes.rs:20-26, src/models/spikes.rs:82-86.

Field	Type	Description
session_id	Option<String>	Isolation ID for multi-tenant environments src/models/spikes.rs:31.
batch_id	u64	Correlation ID for tracking this specific set of spikes src/models/spikes.rs:33.
timestamp	u64	Nanosecond timestamp (UTC or relative) src/models/spikes.rs:35.
spikes	Vec<SpikeEvent>	A collection of individual spike occurrences src/models/spikes.rs:37.
metadata	Option<BatchMetadata>	Diagnostic info like processing latency src/models/spikes.rs:39.

SpikeEvent Details:

• channel (u16): The identifier for the neuron or input channel src/models/spikes.rs:95.
• time (u32): The offset or absolute time of the spike src/models/spikes.rs:97.
• strength (f32): The amplitude/magnitude of the spike src/models/spikes.rs:99.

EmbeddingBatch

Used for high-dimensional vectors, typically for projector and transformer components src/models/embeddings.rs:12-24.

• embedding: A Vec<f32> containing the raw vector data src/models/embeddings.rs:21.
• sequence_length: A usize field ensuring compatibility with sequence-aware models src/models/embeddings.rs:23.

Sources: src/models/spikes.rs:14-100, src/models/embeddings.rs:12-24

---

Training and Feedback Types

These types are generated by the external backend and sent back to Rust for weight updates and credit assignment.

GradientBatch and GradientUpdate

GradientBatch carries weight updates derived from training algorithms src/models/gradients.rs:15-25. It contains a vector of GradientUpdate structs, each targeting a specific layer src/models/gradients.rs:79-89.

• layer_id: A String identifying the target layer src/models/gradients.rs:84.
• gradients: Flattened or sparse Vec<f32> of gradient values src/models/gradients.rs:86.
• eligibility_trace: An optional Vec<f32> specifically for E-prop or STDP-based learning algorithms src/models/gradients.rs:88.

TraceBatch, IpcTraceBatch, and TraceData

Used for credit assignment in spiking networks, providing decay-modulated spike history src/models/traces.rs:13-33. Like spike batches, an explicit alias IpcTraceBatch = TraceBatch prevents collisions with training-side batch types src/models/traces.rs:17-39.

• channel_id: u16 identifier (supports neuron_id as a serde alias for backward wire compatibility) src/models/traces.rs:85-88.
• trace_value: f32 representing the accumulated eligibility trace src/models/traces.rs:90.
• last_spike_time: u32 timestamp of the last spike that contributed to this trace src/models/traces.rs:92.

Sources: src/models/gradients.rs:15-89, src/models/traces.rs:13-93

---

Configuration and Metadata

ConfigPayload and ConfigValue

The ConfigPayload allows for runtime parameter updates (e.g., adjusting learning rates or thresholds) src/models/config.rs:14-22. It uses a HashMap<String, ConfigValue> src/models/config.rs:21.

The ConfigValue enum is untagged (#[serde(untagged)]), meaning the JSON representation is the raw value (e.g., 42.5 instead of {"Float": 42.5}) src/models/config.rs:89-91.

Variant	Internal Type	Notes
Float	f32	First variant; catches all JSON numbers by default. Round-tripping integers through JSON often results in Float src/models/config.rs:92-96.
Integer	u64	Only reached for values that were originally Integer and serialized/deserialized in specific contexts src/models/config.rs:98-104.
String	String	Raw JSON string src/models/config.rs:106-109.
Boolean	bool	Raw JSON boolean src/models/config.rs:112.
FloatArray	Vec<f32>	JSON array of numbers src/models/config.rs:113.

BatchMetadata

Provides a side-channel for telemetry and debugging without altering the core neural data src/models/metadata.rs:11-21.

• processing_latency_ns: Measures the time taken to generate the batch src/models/metadata.rs:16.
• source: A string like "encoder" or "snn_layer_2" to trace data provenance src/models/metadata.rs:18.
• custom: A HashMap<String, String> for arbitrary user-defined metadata src/models/metadata.rs:20.

Sources: src/models/config.rs:14-114, src/models/metadata.rs:11-21

---

Data Correlation Flow

The following diagram illustrates how batch_id and session_id are used to maintain state across the IPC boundary using the IpcMessage envelope.

Batch Correlation Sequence

sequenceDiagram
    participant R as "IpcBackend (Rust)"
    participant E as "External Backend"
    
    Note over R: "Generates batch_id: 101"
    R->>E: "IpcMessage::Spikes(SpikeBatch { batch_id: 101, .. })"
    
    Note over E: "Processes spikes for batch 101"
    E->>R: "IpcMessage::GradientUpdate(GradientBatch { batch_id: 101, .. })"
    
    Note over R: "Applies gradients to batch 101 parameters"

Loading

Sources: src/models/spikes.rs:27-40, src/models/gradients.rs:16-25