Beyond the Vendor

Beyond the vendor stack

Audience: developers of applications that use the IMP API. This page lists what OpenIMP and open-tx-isp do differently from, or in addition to, the vendor stack, and how an application uses it. Only items that are device-tested are listed as supported; opt-in or pending items are marked. State: RC1 (v2026.10.11-rc1). The full table with SoC, API, default, detection and status per item is docs/OPENIMP_BEYOND_VENDOR.md pinned to the tag; switches are listed in Environment Variables.

Application integration notes: everything is reachable through the standard IMP API with the vendor signatures. Where behaviour differs, it is in what the call returns or what the hardware now does. Applications that must also run on the vendor stack should keep anything extra behind a compile-time switch.

Application integration notes

There is deliberately no runtime capability query; a query call was prototyped and withdrawn. Applications that also build against the vendor SDK should put every call that the vendor headers lack behind a compile-time switch and declare the OpenIMP prototype themselves. OpenIMP's own imp_isp.h declares all ISP setters on every platform; vendor headers lack some (for example SetDPC_Strength, SetDefog_Strength, SetDRC_Strength, SetBcshHue, SetBacklightComp, SetFrontCrop on T10/T20/T30).

New in RC1 (since the 2026-10-06 state)

• Encoder ROI (region QP). T31 has an ROI the vendor library does not have: IMP_Encoder_SetChnRoiAttr / GetChnRoiAttr (the T40/T41 call and struct), up to 10 windows, relative or absolute QP, H.264 only, effective on the next picture (no IDR). SetChnROI is effective on T21 and T23 (the vendor T21 never programs ROIs; the vendor T23 accepts regions without visible effect) and on T10/T20. Device-tested with the stream checked in VA-API and strict ffmpeg. Rules: window deltas are limited to -25..+25 and the picture QP plus delta to 0..51 (a warning ROI: requested QP delta ... clamped is logged); do not offer -26 as the lowest relative QP (it produces out-of-range mb_qp_delta that software decoders hide and Intel VA-API, VLC, Firefox and Edge show as broken blocks). Under FixQP an extreme window (absolute 5 or relative -25) lowers the P frames almost frame-wide; do not offer the extremes on a fixed-QP stream. Switches: OPENIMP_T31_ROI, OPENIMP_T21_ROI, OPENIMP_T23_ROI. T41 ROI is code only and off (OPENIMP_T41_ROI=1, experimental).
• Capability marker for T23 ROI. libimp exports the const symbol OpenIMP_Cap_T23HelixRoi; older libimps drop T23 ROI regions silently. Probe it with a weak extern (extern const int OpenIMP_Cap_T23HelixRoi __attribute__((weak));, test the address) or dlsym; absent means an old OpenIMP or the vendor libimp.
• Front crop (IMP_ISP_Tuning_SetFrontCrop). The window is applied in sensor space, before the flip: an application that mirrors the picture must mirror the window too (T23, T21, T31 both axes; T20, T10 vertical only). Convert picture to sensor coordinates, then verify with a labelled picture. Guards beyond stock: T31 refuses (-EINVAL) a window smaller than a running channel's scaler output (the MSCA cannot upscale), and an unscaled main stream makes the call fail; T23 crop off restores the full window; T20/T10 keep a window until a downscaled channel opens. The window is released with the ISP session. Runtime detection of the safe path: the file /sys/module/tx_isp_t23/parameters/fcrop_upscale_pct (T23) or /sys/module/tx_isp_t31/parameters/fcrop_upscale_pct (T31) exists only in drivers with the guards; test for it before offering front crop.
• T20/T10 isp-m0 gain cap. "MAX SENSOR analog gain" in /proc/jz/isp/isp-m0 now reports the value the AE really reaches (smaller of sensor maximum and the SetMaxAgain ceiling, rounded to the sensor's gain step). A day/night rule (decided by the application) that waits for the cap works; with the old value it never saw the cap and stayed in Day mode in the dark.
• AE compensation timing. IMP_ISP_Tuning_SetAeComp works on T20 (monotonic) and T31 (OEM formula); the T31 AE has about 6 s dead time and needs 25 to 30 s to settle, so do not judge or re-adjust earlier.
• T21 image controls beyond the vendor kernel: SetBrightness acts as an exposure compensation (value/128 scales the AE luma target; 128 is the vendor picture; allow 2 to 7 s for the AE) and SetColorfxMode(SEPIA) works.
• T23 autofocus getters (GetAfHist, GetAFMetrices, Get/SetAfWeight, GetAfZone) return live values by default (source_af=1, as the stock module).
• Frame source and encoder API: a JPEG channel feeds itself from the frame source when its video channel is not polled (snapshot-only use needs no H.264 reader); IMP_Encoder_GetFd returns a pollable pipe (use only for select/poll); GetFrame/SnapFrame wait up to 2 s like the vendor library.
• Audio: IMP_AO enable/SendFrame/SetVol, IMP_AI enable/GetFrame/SetVol/SetGain and EnableAec (OpenIMP AECM, mono 16 kHz) are device-tested on T10, T20, T21, T23 and T31 (ERLE on a speech segment 19 to 25 dB; AECM does not cancel pure tones). On T21 a microphone gain above 0 raises the noise floor.
• Robustness: T31 WDR statistics ring persistent across STREAMOFF (100 stop/start cycles and 20 reloads without anomaly); T23 MSCA scratch buffer and frame-source parking (no write into freed buffers after a channel stop).
• Experimental SoCs: T41 gamma, CCM, CSC presets, manual exposure readback, frame drop, sensor register, WDR enable and AF weight (device-tested, see the matrix); T40 first device run (H.264 on two channels, OSD, IVS; CBR about 20 % of target, no H.265).

Encoder and rate control

• Capped modes are real: CappedVBR/CappedQuality run a closed-loop regulator with the vendor PSNR cap (42 dB) on T31; plain VBR is closed loop. Vendor behaviour was silent CBR.
• T31 uses the ported Allegro rate-control core for VBR, CappedVBR, CappedQuality and CBR (OPENIMP_T31_RC_CORE=legacy restores the old controller).
• Deviations (less than vendor): T31 CBR writes no filler NAL, so a static scene stays below the target bitrate; CappedQuality behaves like CappedVBR. Do not rely on CBR output being exactly at the target.
• T10/T20/T21 accept mode, bitrate, QP range, GOP and fps. Some knobs (QP steps, staticTime, changePos, qualityLvl, I/P delta, bias) are accepted but ignored on paths that do not run the vendor controller; do not expose them as live controls without checking the effective rate-control log line.
• A log line per channel states the effective rate control. Out-of-range QP or fps is clamped with a warning naming the field.
• CreateChn(PT_H265) returns -1 on SoCs without HEVC hardware (T10/T20/T21/T23); the vendor returns success and creates a channel that never produces frames. Check the return value and fall back to H.264.
• Real HEVC on T31.
• After 3 failed pictures the T20/T21 encoder is re-created; after 2 fruitless re-creates the channel stops instead of stalling for 20 s per picture. Expect GetStream to fail and restart the channel from the application.
• Encoder channel statistics use the vendor layout with a real average bitrate and can be polled for telemetry.

ISP tuning

• Scene mode and colour effects act on hardware (T21, T20 colour effects); getters return what was set.
• Sinter/Temper denoise strength takes effect on T10, T20, T21 where the vendor firmware ignores it. Value 128 equals the IQ table, so the default picture is identical to the vendor.
• AWB state is kept across stream restarts (T23); on-demand snapshots no longer show a colour cast.
• Maximum analog gain, maximum digital gain (an extra AE stage on T23), IT max and SetSensorFPS reach the AE. T21 ae_it_max_us acts (the vendor ignores it).
• Unknown tuning control IDs are rejected (T23). Check return codes: an error means the control does not exist.
• T23 front crop rejects windows outside the sensor, smaller than 64x64 or odd-sized with -EINVAL.
• T31 privacy mask: 4 rectangles per channel.

JPEG / snapshots

• Configured JPEG quality is applied; Helix SoCs adapt quality (-5 on truncation, +5 after 100 clean frames), so the configured value is only the starting point.
• The last JPEG is re-delivered when the encoder is busy or video memory is short. A snapshot can repeat the previous frame under load.
• Hardware JPEG on T31 (CPU about 70 % down to 17 % at one snapshot per second).
• JPEG shares the H.264 bitstream area on T23 and uses a 1 MiB bitstream buffer on T20/T10, saving video memory.

IVS / motion

• An IVS channel already in use returns EBUSY. Handle it, do not retry blindly.
• Real frame-difference motion detection on T20/T21/T30 (the vendor reported "no motion" always).
• Motion v2 (on by default since 2026-10-04): per-cell background model, hold-off after day/night or gain jumps, object grouping, bounding boxes and strength. OPENIMP_MOTION_V2=0 (or features = 0 through the API) restores the vendor algorithm; IMP_IVS_MoveOutput is then bit-identical to the vendor. Configure with OPENIMP_MOTION_V2* (see Environment Variables) or per channel through the extension API in include/imp/openimp_ivs_move_ex.h; the box/strength result is read with OpenIMP_IVS_MoveGetResultEx() after IMP_IVS_GetResult():

#include "openimp_ivs_move_ex.h"          /* copy the header, no other dependencies */
#pragma weak OpenIMP_IVS_MoveGetResultEx  /* weak: also links against the vendor libimp */
#pragma weak OpenIMP_IVS_MoveSetConfigEx

if (OpenIMP_IVS_MoveSetConfigEx) {
    OpenIMP_IVS_MoveConfigEx c = { .size = sizeof c,
                                   .version = OPENIMP_IVS_MOVE_EX_VERSION,
                                   .features = 0xf };
    OpenIMP_IVS_MoveSetConfigEx(chn, &c);
}
/* after IMP_IVS_GetResult(): */
OpenIMP_IVS_MoveOutputEx out = { .size = sizeof out, .version = OPENIMP_IVS_MOVE_EX_VERSION };
if (OpenIMP_IVS_MoveGetResultEx && !OpenIMP_IVS_MoveGetResultEx(chn, &out)) { /* objects, suppress reasons */ }

ABI rules: every struct starts with size and version; fields are only appended. Errors: -1 with EINVAL (bad channel/size) or ENOENT (no move interface on the channel). Status: device-tested on two cameras (T20, T23) over one night in shadow and override mode; fewer false alarms than the vendor algorithm (blinking lights, room light, IR switch); car headlights can still form an object.

Audio, OSD, rotation

• Real echo cancellation (WebRTC AECM) on T31: EnableAec returns an error when it cannot run instead of a fake success. Check the return value.
• T31 OSD drawn by the IPU (default on), also in hardware-JPEG snapshots; T20/T21 OSD through an IPU hook (the vendor path never drew).
• Software rotation 90/270 on T31 (the vendor-equivalent tiles) before OSD, IVS and encoder.

Reference buffer sharing

T21/T23: reference frames shared between streams, about 1.5 MB less video memory, on by default (OPENIMP_REF_SHARE=0 disables). Not applicable to T10/T20/T31.

Robustness

• Reload and stop robustness: 10 stop/start and 10 rmmod/insmod cycles with 0 oops on T20/T21/T23/T31 and T41; kill -9 of the application recovers.
• rmmod of a sensor module while streaming is refused (the sensor module is pinned while the ISP is open).
• QBUF buffers outside the rmem window are rejected (qbuf_guard).
• Device nodes are opened with O_CLOEXEC, so helper processes do not hold /dev/isp-m0.
• One startup warning when pools plus fixed buffers exceed video memory, see Memory.
• Smaller binaries: libimp about 0.47-0.73 MB depending on the SoC (T41 465,728 B, T23 726 KB) instead of 1.0-1.3 MB.
• Measured against the vendor stack on T20 (night scene, same application): application CPU 10.5 % instead of 25.2 %, snapshot latency 0.20 s instead of 0.45 s; free memory about 5 MB lower because the open module is larger.

Known deviations (less than vendor)

Real WDR on T23 is missing, T41 flip has no effect (the live flip setter is not exercised by the test application), T31 CBR writes no filler, CappedQuality equals CappedVBR, T40 CBR delivers only about 20 % of the target rate and T40 has no H.265 yet. The T10/T20 vendor two-pass encoder denoise is not built. The feature matrix keeps the current list.