2.6.0

2.6.0

Summary

This PR substantially refactors and hardens the flatbuffer_direct backend while preserving the existing CLI/Python API and artifact conventions. It turns the direct converter into a more explicit, testable pipeline, improves shape/layout/control-flow handling across a broad ONNX corpus, expands TensorFlow-free validation, and modularizes the PyTorch export path.

The branch contains 126 focused commits and changes 120 files. The implementation intentionally stays within the existing dependency set and keeps normal flatbuffer_direct conversion and -cotof evaluation independent of TensorFlow.

Motivation

The direct backend had accumulated a large number of interacting, order-sensitive rules. A local fix could easily invalidate unrelated models, producer/consumer relationships were repeatedly reconstructed, layout assumptions were spread across individual operator lowerers, and large characterization tests consumed significant review and maintenance effort.

This work establishes clearer internal contracts and adds regression evidence before and alongside behavioral changes. The goal is not merely to make individual sample models pass, but to make future changes easier to reason about and safer to validate.

Architecture and pipeline changes

Explicit conversion core

• Added conversion contracts and session-scoped state under onnx2tf/tflite_builder/core/.
• Added reusable graph indexing for producers, consumers, constants, and tensor usage.
• Added centralized layout state instead of relying exclusively on local NCHW/NHWC guesses.
• Added pass specifications/results, deterministic phase orchestration, diagnostics, and validation helpers.
• Added ModelIR utilities and invariant checks for tensor availability, shapes, dtypes, graph outputs, and recurrent state handling.
• Added structured lowering registry support and smaller operator-family modules for normalization, quantization, recurrent, reduction, selection, and shape operations.
• Added focused pass modules for constant folding, padding/layout behavior, precision handling, and PyTorch compatibility.

The resulting flow is substantially closer to:

request -> ONNX analysis/normalization -> conversion session and graph index -> ordered passes -> operator lowering -> ModelIR validation/cleanup -> requested exporters

Mechanical decomposition

• Split reusable PyTorch layout, ONNX bridge, optimizer, code-generation, and export-support logic into dedicated modules.
• Reduced central orchestration responsibilities and made unit-level testing possible without running complete conversions.
• Added architecture tests that protect dependency direction and the intended internal boundaries.

flatbuffer_direct correctness improvements

Shape, rank, and layout recovery

• Added demand-driven static shape reconciliation around attention, gather, layer normalization, matmul, and related dynamic tails.
• Preserved semantic reshape/flatten contracts instead of committing placeholder dimensions too early.
• Improved unknown-rank and high-rank handling for GatherElements, BatchMatMul, padding, slicing, grouped convolution, Conv3D, Resize, DepthToSpace, and transpose convolution paths.
• Added idempotent layout-recovery passes and guards against stale or duplicated boundary transposes.
• Repaired channel-axis handling for concat/split/conv pipelines, channel shuffle, residual paths, attention blocks, and rank-3 broadcasts.
• Added explicit handling for non-image recurrent inputs and regression profile support for -kat axis preservation.

Control flow

• Added deterministic branch-local tensor namespaces to prevent collisions between If branches.
• Recursively remapped lexical captures for nested control-flow graphs.
• Improved branch dtype and shape reconciliation, mixed numeric operands, scalar handling, and graph-output preservation.
• Made speculative/inactive flattened branches executable when an otherwise valid Squeeze sees a non-singleton extent from the inactive path.
• Corrected Where lowering so a rank-1 singleton condition uses SELECT_V2 when prefix-style SELECT broadcasting is invalid.
• Added ONNX graph repairs for torchvision NMS guard captures and several legacy control-flow/reference-evaluation patterns.

Recurrent operators

• Preserved recurrent input axes for GRU/RNN/LSTM-style non-image tensors.
• Safely initialized constant zero LSTM states.
• Added primitive forward-LSTM lowering for explicit runtime states where TFLite's mutable sequence-LSTM state contract cannot accept an ordinary runtime producer.
• Preserved recurrent states through reshape cleanup and validated ONNX/TFLite parity for branch-local and direct state I/O.

Quantized operators

• Improved QLinearConv, QLinearMatMul, pooling, concat, residual, softmax, split, and Q/DQ handling.
• Matched ONNX rounding, truncation, zero-point, padding, and requantization semantics more closely.
• Preserved quantization boundaries and constant bias contracts through layout cleanup.
• Added tests for uint8 and mixed-encoding pipelines, integer TopK, integer resize, and quantized layout recovery.

Additional builtin coverage and compatibility

• Added or improved builtin paths for grouped Conv3D, rank-5 Resize, integer TopK, nearest GridSample, standard linear GridSample mode, edge padding, dynamic resize scales, and half-pixel round/ceil resize behavior.
• Added guarded custom fallback for genuinely unresolved shapes or unsupported capability instead of emitting structurally invalid builtin graphs.
• Improved Microsoft-domain/operator compatibility and legacy opset preparation for ONNX Runtime reference evaluation.

Accuracy evaluation without TensorFlow

• Kept default direct TFLite conversion and -cotof evaluation TensorFlow-free.
• Added ONNX Runtime compatibility preparation for legacy opsets, GroupNorm, sequence If, integer MatMul, optional tensors, unknown-rank Conv, Microsoft contrib operators, and graph ordering.
• Added deterministic seeded control inputs, including static TopK values and audio sample-rate inputs.
• Added narrowly scoped ONNX ReferenceEvaluator compatibility for malformed nested-LSTM shape inference, including complete Y, Y_h, and Y_c outputs.
• Preserved sequential evaluation: inference validation uses one active process at a time.

Representative recovered regressions include:

• superpoint_lightglue_end2end_fused_cpu.onnx: maximum absolute error 1.946091651916504e-05.
• silero_vad.onnx with -kat input state sr: evaluation_pass=true, maximum absolute error 1.375097781419754e-06, RMSE 1.222495367934982e-07.
• Multiple recurrent OCR, RF-DETR, quantized pooling, MobileFaceNet, ResNeXt/FPN, MIRNet, GridSample/attention, D3Net, and other Tier 0-4 regressions promoted after focused validation.

Regression corpus and execution policy

• Added recorded root-model baselines for Tiers 0 through 5.
• Added an active managed profile for every root model with 1-1,999 ONNX nodes (Tiers 0-4).
• Tier 5 is retained only as historical baseline data and is excluded from the active profile.
• Models previously recorded as timeouts are retained for provenance but excluded from subsequent validation.
• The active profile currently contains 394 models: 343 expected passes and 51 known non-passes, with 26 recorded timeouts excluded.
• Added corpus manifest and bulk-runner validation for fixed model ordering, root-only discovery, tier limits, per-model shape hints, -ois, and -kat options.
• All corpus inference is designed to run sequentially with concurrency fixed to one.

PyTorch-family export refactoring

• Extracted shared layout utilities, ONNX utilities, bridge passes, layout passes, optimizer orchestration, code-generation helpers, and export error handling.
• Kept common canonicalization reusable across native PyTorch, TorchScript, Dynamo ONNX, and ExportedProgram paths.
• Added focused tests for each extracted layer to reduce reliance on a single monolithic exporter test surface.

Public compatibility and packaging

• Preserved the existing public CLI/Python entry points and the flatbuffer_direct backend selection behavior.
• Preserved optional TensorFlow boundaries for explicitly requested TensorFlow artifacts while avoiding TensorFlow imports in direct TFLite paths.
• Added no new package dependency.
• Updated the project, lockfile, Python package, and documented container image version to 2.6.0.

Tests

The final relevant local suite was executed with uv:

uv run pytest -q \
  tests/test_tflite_builder_direct.py \
  tests/test_accuracy_evaluator_seeded_input.py \
  tests/test_onnx_reference_compat.py \
  tests/test_flatbuffer_direct_bulk_runner.py \
  -k 'not test_tflite_backend_matrix_add and not test_tflite_backend_matrix_hardswish_rewrite_on_off and not test_tf_converter_resize_cubic_avoids_flex_resize_bicubic and not test_tf_converter_resize_cubic_honors_cubic_coeff_a and not test_flatbuffer_direct_group_norm_alias_builtin_conversion'

796 passed, 5 deselected, 2 warnings

Additional focused validation included:

• 61 control-flow/LSTM/Squeeze/Where tests.
• 30 ONNX reference-compatibility and deterministic-input evaluator tests.
• Direct isolated -cotof verification of the recovered Silero model.
• A package/version consistency check for 2.6.0.

The five deselected tests are environment-specific rather than code regressions:

• Four tf_converter tests require the optional TensorFlow/tf-keras environment, which is intentionally absent from the core environment.
• One GroupNorm alias test resolves an external Python 3.10 Torch binary under Python 3.12 and fails with an ABI symbol error (_PyCode_GetExtra).

The two warnings are existing NumPy float16 overflow warnings from tests that intentionally cast extreme values.

Known limitations and intentionally retained non-passes

• silero_vad (1).onnx references 14 lexical captures that do not exist in the serialized model, including all four LSTM weight/bias captures. It cannot produce a trustworthy ONNX reference result and remains documented as an input-model defect.
• inverse_11.onnx contains a 224x224 matrix inverse. The current exact direct lowering is intentionally limited to matrices up to 16x16; larger matrices fall back to ONNX_INVERSE, which stock LiteRT cannot execute. The observed reference matrices are nearly singular, so a low-order approximation would not satisfy the required absolute-error bound and has not been introduced.
• The active profile still records 51 known non-passes. This PR improves and characterizes the backend but does not claim that every Tier 0-4 model is fully recovered.
• Final whole-corpus performance/RSS measurements and the remaining non-pass improvements are follow-up work, not hidden in this PR.

Suggested review order

1. docs/flatbuffer_direct_architecture.md for scope and design.
2. onnx2tf/tflite_builder/core/ for contracts, session state, graph/layout indexing, pass management, and validation.
3. onnx2tf/tflite_builder/op_families/, op_registry.py, and representative op_builders/ changes.
4. onnx2tf/utils/onnxruntime_compat.py, onnx_graph_repair.py, and accuracy_evaluator.py for TensorFlow-free validation.
5. PyTorch utility/pass extraction modules.
6. Managed baseline/profile files and the focused test modules.

Change size

126 commits
120 files changed
34,855 insertions
11,788 deletions

What's Changed

• Refactor and harden the flatbuffer_direct conversion pipeline by @PINTO0309 in #945

Full Changelog: 2.5.2...2.6.0