Release v0.4.0

Neat Library v0.4.0

Neat Library v0.4.0 adds packaged PCIe inference APIs, native H.265 workflows, application-owned MIPI capture buffers, per-frame metadata through hardware decoding, stricter vision postprocessing, faster model loading, and expanded LLiMa 0.4 integration.

This release targets Platform 2.1.3, uses Model Zoo 2.1.3, and introduces C++ ABI version 4.

Important: Native applications and plugins must be rebuilt against the v0.4.0 headers and libsima_neat.so.4. Python applications should reinstall the matching PyNeat package.

Highlights

PCIe inference APIs

Applications can load models and run inference on a Modalix PCIe card through new C++ and Python APIs.

• Use pcie::Model for synchronous or asynchronous single-model inference.
• Use pcie::Runtime for request-correlated, multi-model execution.
• Run tensor, image-preprocessing, and decoded-detection workflows.
• Use multiple independent queues with bounded inflight work and explicit backpressure.
• Reuse models and queues after timeouts and recoverable failures.
• Build native card-side graphs with PCIeSrc and PCIeSink.
• Install runtime and development packages for Ubuntu 22.04 and 24.04 on x86_64 and AArch64.
• Use the corresponding pyneatpcie Python package.

The host and card PCIe packages must come from the same release set.

H.265 and encoded-video workflows

Neat now supports H.265/HEVC alongside H.264, JPEG, and MJPEG.

• Decode H.265 Main, 8-bit, 4:2:0 streams through SimaDecode.
• Receive parsed H.265 RTSP access units without decoding.
• Run decoded H.265 frames through inference graphs.
• Forward the original H.264 or H.265 stream over RTP/UDP without decoding or re-encoding.
• Fuse matching encoded-video and decoded-inference branches before decode.
• Use H265 and HEVC, or H264 and AVC, interchangeably in C++ and Python.
• Select the RTSP codec once and use the codec-neutral payload_type option.
• Use VideoSenderOptions::Passthrough(codec) for encoded output.

Supported H.265 input is limited to Main-profile, 8-bit, 4:2:0. Main 10, 10-bit, 4:2:2, monochrome 4:0:0, raw H.265 encoding, and transcoding are not included.

Per-frame metadata through decoding

HttpMjpegDecodedInput can capture selected multipart JPEG headers as string attributes on each Sample.

Header capture is allowlist-based and disabled by default. When enabled, attributes remain associated with the correct decoded frame through decoder reordering, dropped pictures, queues, branches, buffer reuse, reconnects, and public C++ and Python Sample boundaries.

The underlying correlation contract covers JPEG, H.264, and H.265. This allows identifiers, timestamps, sequence numbers, and other frame-specific values to travel with the image they describe.

MIPI camera buffer ownership

CameraInput can provide libcamera with a standard per-plane DMA-BUF pool backed by Neat-owned SiMaAI memory.

• Compatible frames enter the graph without an additional payload copy.
• Retained frames keep the complete libcamera buffer alive.
• Holding the capture pool applies backpressure instead of allowing early reuse.
• Restart and repeated start/stop cycles preserve buffer ownership.
• Pool depth can grow to the depth selected by the camera pipeline.
• Per-plane stride, alignment, offset, and padding requirements are preserved.
• Strict mode requires direct capture and never copies.
• Preferred mode can fall back to the existing explicit CPU-copy path.
• C++ and Python applications can request a minimum capture-buffer count.

The workflow was exercised with an IMX568 camera. Direct capture requires the matching Internals camera-memory implementation.

Video and synchronization improvements

• Raw VideoSender avoids format conversion when its input is already compatible NV12 memory.
• Codec-valid padded raw-video layouts are handled correctly.
• Internal graph boundaries no longer invent timestamps for untimed frames.
• Video, inference, and application metadata remain on the same timeline across split and fan-out graphs.
• Encoded H.264 and H.265 branches can remain holder-backed before decode.

SuperPoint, SSD, and YOLO26

SuperPoint is now a first-class BoxDecode result type in C++ and Python.

• Typed keypoints, scores, and descriptors are exposed through the default FeaturePointsV1 representation.
• INT8, BF16, and FP32 detector and descriptor outputs are supported across dense and packed layouts.
• Feature coordinates are mapped back to the source image.
• Points that fall only inside preprocessing padding are removed.
• Incomplete or ambiguous contracts fail during graph preparation.

SSD decoding now uses exact prepared contracts for:

• SSD300 at 300×300.
• SSD-Mobile at 300×300.
• SSD-Mobile at 320×320.
• TorchVision SSDlite-Mobile at 320×320.

Each recipe validates ordered tensor geometry, anchor counts, class depth, activation, layout, quantization, and stretch-resize behavior. Unknown, reordered, incomplete, or contradictory layouts are rejected instead of being decoded heuristically.

YOLO26 improvements include:

• Correct half-cell anchors for pose keypoints.
• Fail-fast class-count validation.
• A dedicated EV74 preprocessing path for MLA-tessellated inputs.
• Correct preservation of candidates through non-maximum suppression.

BenchmarkOptions can now include BoxDecode when measuring complete model and postprocessing execution.

Faster and safer model loading

Model archives are inflated once per load instead of being reopened and decompressed for each operation.

The loader also:

• Uses in-process zlib instead of an external gzip process.
• Creates private extraction roots for concurrent loads.
• Prefers eligible local NVMe storage.
• Performs bounded disk-space checks before and during extraction.
• Rejects corrupt, incomplete, unsafe, or unsupported archives.
• Validates required MPK manifests and JSON before execution.
• Preserves the existing no-extraction path for organized model directories.
• Accepts pathlib.Path and other os.PathLike values in pyneat.Model.

Across two Modalix benchmark campaigns, the measured load time for a 150 MB reference archive fell from 278.65 seconds to approximately 4.96 seconds, about 56× faster. These measurements cover archive loading and extraction, not inference latency.

Expanded GenAI integration

Core is aligned with the LLiMa 0.4 package and runtime contracts.

• Reasoning content can be returned separately and selected per request.
• Streaming tool-call handling is expanded across supported Gemma, Llama, Qwen, Qwen-VL, and LFM paths.
• EAGLE3 target and draft packages can be discovered and routed through the direct and server APIs.
• Direct C++, Python, and server APIs expose set_lora and unset_lora.
• Multi-ELF vision-model configurations are recognized.
• Cached vision outputs preserve quantized embeddings, per-token scales, and optional deep-stack features without assuming one embedding dtype.
• Whisper results include detected language, translation output, average log probability, and no-speech probability.

EAGLE3 output validation and real LoRA weight switching require matching model artifacts and should be qualified separately from API availability.

Runtime reliability and diagnostics

• Single-decoder graphs are admitted correctly.
• Decoder capacity remains reserved until workers stop.
• Partial decoder startup failures release their resources.
• Composite graph runs close without retaining child runs or Python objects.
• Rank-2 detessellated outputs preserve their intended geometry.
• Conflicting batch aliases, duplicate graph nodes, and conflicting runtime element names fail early.
• End-of-stream handling and repeated application execution are more deterministic.
• M4 recovery keeps AppComplex alive while firmware restarts.

GStreamer and model-loading failures now use more specific Neat error categories. Diagnostic storage removes passwords, tokens, cookies, signed URLs, and other credentials while preserving the underlying pipeline failure.

Breaking changes and migration

Rebuild native applications

The public C++ ABI is version 4 and the shared-library SONAME is:

libsima_neat.so.4

Rebuild all native C++ applications and plugins. Binaries built against an earlier ABI-4 development snapshot must also be rebuilt because public metadata and feature-result layouts changed before release.

Reinstall Python packages

Reinstall the matching PyNeat package. PCIe applications must also install the matching pyneatpcie package when using Python.

Update error-code handling

Applications that branch on exact error strings must adopt the new error taxonomy. In particular:

• Replace misconfig.caps handling with misconfig.media_caps and misconfig.media_format.
• Add the more specific build errors build.plugin_missing, build.property_invalid, and build.pipeline_syntax.
• Accept propagated root-cause errors from pull operations instead of relying only on runtime.pull.
• Retain a default branch for future error categories.

Update encoded-video configuration

New RTSP code should use payload_type.

New encoded output code should use:

VideoSenderOptions::Passthrough(codec)

The per-codec RTSP payload settings and H.264-specific encoded sender helper remain available for compatibility but are deprecated.

Upgrade matching runtime components

• Install Core, Internals, and LLiMa from the same Neat 0.4.0 release set.
• Upgrade the decoder plugin and decoder daemon together to preserve per-frame attributes.
• Install matching PCIe host and card packages.
• Use preferred camera mode only when an explicit CPU-copy fallback is acceptable.
• Local documentation builds require Node.js 22.12.0 or newer.

Install

Install or update Neat Library with:

sima-cli neat install core@v0.4.0

To download only the PyNeat wheel:

sima-cli neat install core@v0.4.0 -t pyneat

Compatibility

• Core package version: 0.4.0.
• Compatible Platform release: 2.1.3.
• Model Zoo release: 2.1.3.
• Public C++ ABI: 4.
• H.265 input: Main, 8-bit, 4:2:0.
• Matching Core, Internals, LLiMa, and PCIe artifacts are required for their corresponding workflows.

Feature-level validation

The included feature work reports validation covering:

• Live H.265 RTSP decode, inference, encoded boundaries, RTP delivery, and decoder configurations.
• IMX568 direct DMA-BUF capture, buffer retention, backpressure, pool growth, and repeated restart.
• Model-loader security, malformed archives, concurrent loads, storage selection, and Modalix timing.
• Exact prepared SSD models and SuperPoint contract matrices on AArch64/Modalix.
• C++ and Python API-surface checks.
• PCIe installed-package smoke tests and local hardware lifecycle, queue, timeout, and backpressure testing.

Final release-candidate validation should be recorded after the v0.4.0 tag is cut from the approved PR head.

Key included pull requests

• #624 - Strict prepared SSD decode contracts
• #626 - Native H.265 decode and RTP output
• #635 - Platform 2.1.3 compatibility
• #643 - Codec-neutral RTSP and encoded output
• #653 - Single-inflation model loading
• #665 - Structured GStreamer diagnostics
• #669 - In-process model decompression and storage selection
• #696 - SuperPoint BoxDecode support
• #712 - EAGLE3 and LoRA API integration
• #720 - PCIe accelerator APIs and graph support
• #724 - Per-frame metadata through decoding
• #763 - Application-owned DMA-BUF camera capture

Known issues

• When browser-based VS Code is mapped to a host port other than 10000, neat still prints URLs using port 10000. Use the VS Code URL printed during installation, or replace the port with the codeUIHttps host port listed under Exposed Ports.

Full Changelog: v0.3.0...v0.4.0

Contributors

Thank you to everyone represented in this release boundary:

• @manuel-roldan
• @Onat-Inak
• @florianvoss-commit
• @shanguangzai
• @dotimothy
• @masanoriotobe-wq
• @abhishekkumar-k