Skip to content

Release v0.3.0

Latest
Latest

Choose a tag to compare

View all tags
@github-actions github-actions released this 17 Jul 18:11
v0.3.0
17a7c77
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Neat Library v0.3.0

Neat Library v0.3.0 is a major application-facing release for live video, camera ingestion, high-density multi-stream inference, and on-device vision processing.

This release adds first-class MIPI/libcamera input, native H.264/JPEG/MJPEG decoding, reusable RTSP and HTTP input APIs, automatic realtime graph optimization, stronger zero-copy lifetime guarantees, SSD BoxDecode support, and full PyNeat integration with LLiMa.

Important: v0.3.0 introduces C++ ABI version 3 and removes the preview realtime graph APIs. Native applications and plugins must be rebuilt.

Highlights

First-Class MIPI And Multi-Camera Support

Applications can now use CameraInput from C++ or PyNeat to build source-owned MIPI/libcamera pipelines.

  • Supports MIPI/libcamera camera ingestion without application-managed GStreamer pipelines.

  • Passes compatible camera buffers directly into CVU and MLA routes.

  • Uses a bounded copy only when an upstream DMA buffer cannot be consumed safely.

  • Supports mixed-resolution MIPI, USB, and decoded video streams.

  • Refreshes frame geometry per sample so preprocessing and detection metadata use the correct resolution.

  • Preserves pooled camera buffers until every zero-copy branch has completed.

  • Adds shallow Branch fan-out and CombinePolicy::RoundRobin for multi-source applications.

Validation included a 120-second YOLO26 application using two IMX477 MIPI cameras and one USB MJPEG camera.

Native H.264, JPEG, And MJPEG Support

v0.3.0 introduces reusable public codec APIs for common camera and streaming workflows.

  • SimaDecode provides native H.264, JPEG, and MJPEG decoding.

  • RtspEncodedInput provides reusable encoded H.264 and MJPEG RTSP input.

  • RtspDecodedInput is now codec-aware while keeping H.264 as the default.

  • HttpMjpegDecodedInput provides complete HTTP multipart-MJPEG ingestion and decode.

  • RTSP MJPEG can derive its source frame rate from SDP information.

  • Explicit source_fps configuration can repair missing or invalid source caps.

  • Local HTTPS MJPEG sources can disable strict certificate validation when required.

  • The existing CPU JpegDecode path remains available.

Existing RTSP H.264 applications remain source-compatible unless they use the removed preview realtime APIs. H.265 is not included in this release.

High-Density Realtime Graphs

Applications continue to use the standard Graph::connect() and Graph::build() workflow. Core now automatically selects the optimized realtime execution path when the graph topology is eligible.

  • Keeps only the newest pending frame from each stream under load.

  • Preserves stream identity and schedules active streams fairly.

  • Adds per-stream and application-wide raw-frame admission limits.

  • Prevents stale frames from building up and increasing live-video latency.

  • Fuses encoded H.264 output before decode when the graph sends the same stream to Insight.

  • Avoids a second RTSP session, decoded-frame CPU copies, and board-side re-encoding.

  • Keeps slow visualization output from blocking decode and inference.

  • Adds bounded RTSP-aware shutdown and clearer diagnostics for incomplete teardown.

These changes underpin the companion Apps high-density object-detection profiles:

  • 16 × 720p at 25 FPS

  • 24 × 720p at 20 FPS

  • 48 × 720p at 10 FPS

A 24-stream candidate completed a 30-minute soak at approximately 483 aggregate FPS with all streams active and no metadata failures, stalls, decoder restarts, or lingering processes.

Stronger Memory And Dynamic-Input Handling

v0.3.0 strengthens buffer ownership across graph boundaries and long-running video pipelines.

  • Preserves encoded sample holders and negotiated caps across graph builds.

  • Honors the public input memory_policy contract.

  • Retains the pooled parent buffer for zero-copy views until all consumers finish.

  • Prevents small camera pools from stalling after only a few frames.

  • Keeps dynamic preprocessing capacity large enough for later higher-resolution frames.

  • Correctly handles device-backed tensors with alignment padding.

  • Adds clearer diagnostics when an input exceeds the configured preprocessing capacity.

Expanded On-Device Vision Processing

  • Adds BoxDecodeType::Ssd for model-managed on-device SSD detection.

  • Supports SSD300/dboxes300_coco and SSD-MobileNetV2-COCO decoding.

  • Keeps existing YOLO BoxDecode workflows available.

  • Fixes multi-input pre-MLA routing when public model-input order differs from internal MLA input order.

  • Rejects ambiguous input mappings instead of silently sending incorrect tensor data to MLA.

PyNeat GenAI And LLiMa Integration

GenAI-enabled PyNeat builds now require and use the full LLiMa backend instead of exposing stub-backed APIs.

Validation covers:

  • LLM execution

  • Vision-language model execution

  • ASR execution

  • Graph-based GenAI workflows

  • GenAI server construction and HTTP requests

Reliable Insight Metadata Delivery

MetadataSender now keeps UDP datagrams at or below 1200 bytes.

  • Existing payloads up to 1200 bytes remain unchanged.

  • Larger JSON messages are divided into versioned, ordered chunks.

  • Logical payloads up to 65,507 bytes are supported.

  • Application-facing C++ and Python APIs remain unchanged.

  • Sending remains nonblocking so visualization cannot stall inference.

  • Use Insight v0.0.6 or later to reconstruct chunked metadata.

Installation And Documentation Improvements

  • Core, Internals, and LLiMa packages are aligned on the 0.3.0 package line.

  • Runtime and development packages are installed as a coherent set.

  • Installer validation protects unrelated Modalix packages, including Palette and OTA components.

  • DevKit package inventory checks detect incomplete or incompatible installations.

  • New documentation covers offline SDK and Model Compiler installation.

  • SDK-only users now have a complete model-compilation workflow.

  • Documentation explains SDK architecture detection and artifact connectivity requirements.

Breaking Changes And Migration

Rebuild Native Applications

The C++ ABI is now version 3 and the shared-library SONAME is:

libsima_neat.so.3

Rebuild all native C++ applications and plugins. Do not mix binaries linked against libsima_neat.so.2 and libsima_neat.so.3.

Update Realtime Graph Calls

Replace the removed preview APIs:

  • RealtimeGraphLinkOptions

  • connect_realtime()

  • build_fused_realtime_source()

  • build_fused_realtime_sources()

  • RealtimeEveryFrameByStream

Use:

  • GraphLinkOptions

  • Standard Graph::connect()

  • Standard Graph::build()

  • RealtimeLatestByStream where replacing stale pending frames is acceptable

Saved graphs containing realtime_every_frame_by_stream must be recreated.

Install

Install or update Neat Library with sima-cli:

sima-cli neat install core@v0.3.0

To download only the PyNeat wheel:

sima-cli neat install core@v0.3.0 -t pyneat

Use Core, Internals, LLiMa, Apps, and Insight artifacts from the matching Neat 0.3.0 release set.

Compatibility Notes

  • Compatible platform release: 2.1.2.

  • Core package version: 0.3.0.

  • Core C++ ABI version: 3.

  • Existing RTSP H.264 applications remain compatible unless they use the removed preview realtime APIs.

  • H.265 decoding is not included in this release.

  • Metadata larger than 1200 bytes requires Insight v0.0.6 or later.

  • MIPI camera workflows require the appropriate device-tree overlays on the target system.

  • Native applications and plugins must be rebuilt against the v0.3.0 libraries.

Validation

Release validation covered:

  • Vulcan SDK and ARM64 builds

  • Code hygiene and release-policy checks

  • Modalix DevKit runtime tests

  • Native H.264, JPEG, and MJPEG codec paths

  • RTSP H.264, RTSP MJPEG, and HTTP MJPEG applications

  • Mixed-resolution MIPI and USB camera execution

  • High-density 24-stream runtime and soak testing

  • Zero-copy buffer lifetime and dynamic preprocessing

  • SSD on-device BoxDecode

  • Multi-input pre-MLA tensor routing

  • Large metadata boundary and reconstruction tests

  • PyNeat LLM, VLM, ASR, and GenAI server workflows

  • DevKit package installation and inventory validation

Key Included Pull Requests

  • #509 - Native H.264/JPEG/MJPEG codec APIs

  • #512 - SSD-family on-device BoxDecode

  • #535 - Codec source FPS contract

  • #539 - Multi-input pre-MLA routing correction

  • #543 - Offline SDK installation documentation

  • #548 - Public realtime inflight limits

  • #554 - PyNeat GenAI and LLiMa integration

  • #564 - Zero-copy parent-buffer retention

  • #572 - MIPI and high-throughput multi-camera execution

  • #575 - Pooled-buffer lifetime hardening

  • #580 - Network-safe metadata chunking

  • #581 - Bounded per-stream live fan-in

  • #591 - Topology-driven realtime execution and encoded fan-out

  • #594 - Mixed-resolution MIPI and runtime-install stabilization

Full Changelog: v0.2.0...v0.3.0

Contributors

Thank you to everyone who contributed to this release:

Assets 2
Loading