v1.0.0

Browse the Repository | Everything since v0.3.12

The first stable release, and the source code tree switched from TensorFlow to LiteRT.

LiteRT

• TfLite is built from LiteRT 2.2.0 (tflite/ in the LiteRT tree) instead of tensorflow/lite/. TensorFlow is still fetched, but only as a build dependency.
• tflite_beam:tflite_version/0 answers <<"2.2.0">>. A delegate plugin loaded through tflite_beam_delegate:external/1 has to be built against the same LiteRT release.
• tflite_beam:source_tree/0 answers litert, so a stale or wrong binary can be told apart. tensorflow_version/0 reports the TensorFlow release the build pulled in.
• The seven precompiled targets and their glibc requirements are unchanged.

New

• LiteRT compiled model API. tflite_beam_litert_compiled_model: accelerators by name (cpu, gpu, npu), fully_accelerated/1 to check whether the accelerator took the whole graph, and a per-operator profiler (profile/2, summarise_profile/1). tflite_beam_litert_compiled_model_server shares one model between callers behind a bounded queue. tflite_beam_litert_compiled_model_isolated runs it on its own node, so a crash in native code comes back as {error, _} instead of taking the VM down. This is a build option: TFLITE_BEAM_ENABLE_LITERT_API=true fetches the precompiled variant that has it, or turns it on in a source build. The GPU accelerator is built whenever the LiteRT API is. armv6 and armv7l have no LiteRT variant, because they build without XNNPACK and LiteRT's CPU accelerator is XNNPACK.
• Delegates. tflite_beam_delegate:xnnpack/0,1, tflite_beam_delegate:external/1,2 for any library implementing TfLite's delegate plugin interface, tflite_beam_coral:edge_tpu_delegate/0,1, and tflite_beam_interpreter_builder:add_delegate/2,3 with #{on_decline => error | fallback}. tflite_beam_delegate:available/0 reports what the build compiled in. XNNPACK is now attached explicitly in build/2 instead of by TfLite inside allocate_tensors/1: same output, but it shows up in the execution plan right away and can be configured.
• tflite_beam_interpreter_server. One interpreter in a process, so setting inputs, invoking and reading outputs is one step. Two processes sharing a raw interpreter got each other's answers 147 times in 400 calls on a real model. For the direct API there is tflite_beam_interpreter:controlling_process/1,2, like gen_tcp.
• Signature runners. tflite_beam_interpreter:get_signature_runner/2 and tflite_beam_signature_runner: inputs and outputs by name instead of by index. Also resize_input_tensor/3 and resize_input_tensor_strict/3, enable_cancellation/1 and cancel/1, release_non_persistent_memory/1, reset_variable_tensors/1, signature_inputs/2, signature_outputs/2, subgraphs_size/1 and verify_and_build_from_buffer/1,2.
• Tokenizer. CJK ideographs are split the way BERT does it, so CJK characters no longer come back as [UNK]. The punctuation lookup no longer makes a process call per character: 5.56us to 0.21us per character, and 24,000 characters of Chinese from 923ms to 13ms.
• Verified downloads. Precompiled tarballs are checked against a sha256 manifest shipped in the package (checksum.term) before unpacking. An https download that cannot be verified is refused; TFLITE_BEAM_CACERT names a CA store and TFLITE_BEAM_UNSAFE_HTTPS skips the check.
• Model metadata reports custom_metadata. tflite_beam:xnnpack_max_tensor_dims/0. Tensor types {f, 8} and {f8_e4m3fn, 8}. Edge TPU context options are applied instead of ignored.

Memory safety

Each of these has a test that fails without the fix.

• Tensor handles keep their interpreter alive, and are retired when allocate_tensors/1, a resize or a second build/2 moves the tensor. They used to read freed memory.
• Resizing an input past XNNPACK's maximum rank wrote past a stack buffer (SIGBUS or SIGSEGV depending on the rank). Refused now.
• A failed build/2 returned ok and left an empty interpreter that the next call dereferenced.
• Truncated or corrupt models are verified instead of walked, and a tensor with a null name no longer crashes the node.
• Every NIF is behind an exception guard, every interpreter call takes the in-use guard, and three data races in the ownership state found by ThreadSanitizer are fixed.
• Resource destructors run on a reaper thread instead of the scheduler: a 49MB model took 15.8ms in the destructor, now under 5us. Tensor copies, build/2 and allocate_tensors/1 run on dirty schedulers.
• Leaks: the model buffer on parse failure, the signature runner registry, the error reporter, and references stranded when an allocation failed partway.

Upgrading from 0.3

Nothing was renamed. One module was removed and a few return values changed.

• tflite_beam_interpreter_builder:build/2 returns {error, Reason} on failure instead of ok.
• A tensor handle stops working once allocate_tensors/1, a resize or a second build/2 has moved it. Fetch it again.
• tflite_beam_tensor:set_data/2 requires a binary of exactly the tensor's size. A short one used to be written partially and reported as success.
• tflite_beam_interpreter:predict/2 returns list(binary()) | {error, binary()}. A failed invoke or an unreadable output fails the whole call.
• get_signature_defs/1 returns signature keys and input/output names as binaries, not atoms.
• tflite_beam_tensor:type/1 can also return {u, 16}, {bf, 16}, {f, 8}, {f8_e4m3fn, 8} and unknown.
• Two processes calling one interpreter at the same time are refused instead of raced.
• XNNPACK is attached in build/2. tflite_beam_ops_builtin_builtin_resolver:new(#{apply_default_delegates => true}) gives TfLite's lazy delegation back.
• Negative tensor indices are refused by set_inputs/2, set_outputs/2 and set_variables/2. TfLite reads before its tensor table with them.
• set_num_threads/2 accepts -1.
• list_associated_files/1 and get_associated_file/2 return {error, _} for a model with no archive, instead of bad_eocd from zip.
• A machine with no CA certificate store no longer downloads precompiled binaries unverified.
• tflite_beam_contrib_huggingface is removed.