v2.0.0-rc.2

Changes

• f30ba57 Update to ONNX Runtime v1.17.3
  • New: CUDA 12 binaries. ort will automatically detect CUDA 12/11 in your environment and install the correct binary.
  • New: Binaries for ROCm on Linux.
  • Note that WASM is still on v1.17.1.
• b12c43c Support for wasm32-unknown-unknown, wasm32-wasi
  • With some minor limitations; see https://ort.pyke.io/setup/webassembly.
  • Thank you to Yunho Cho, whose sponsorship made this possible! If you'd also like to support us, you may do so on Open Collective 💖
• cedeb55 Swap specialized value upcast and downcast function names to reflect their actual meaning (thanks @/messense for pointing this out!)
• de3bca4 Fix a segfault with custom operators.
• 681da43 Fix compatibility with older versions of rustc.
• 63a1818 Accept ValueRefMut as a session input.
• 8383879 Add a function to create tensors from a raw device pointer, allowing you to create tensors directly from a CUDA buffer.
• 4af33b1 Re-export ort-sys as ort::sys.

---

If you have any questions about this release, we're here to help:

• Discord: #💬｜ort-discussions
• GitHub Discussions

Love ort? Consider supporting us on Open Collective 💖

❤️💚💙💛

v2.0.0-rc.1

Value specialization

The Value struct has been refactored into multiple strongly-typed structs: Tensor<T>, Map<K, V>, and Sequence<T>, and their type-erased variants: DynTensor, DynMap, and DynSequence.

Values returned by session inference are now DynValues, which behave exactly the same as Value in previous versions.

Tensors created from Rust, like via the new Tensor::new function, can be directly and infallibly extracted into its underlying data via extract_tensor (no try_):

let allocator = Allocator::new(&session, MemoryInfo::new(AllocationDevice::CUDAPinned, 0, AllocatorType::Device, MemoryType::CPUInput)?)?;
let tensor = Tensor::<f32>::new(&allocator, [1, 128, 128, 3])?;

let array = tensor.extract_array();
// no need to specify type or handle errors - Tensor<f32> can only extract into an f32 ArrayView

You can still extract tensors, maps, or sequence values normally from a DynValue using try_extract_*:

let generated_tokens: ArrayViewD<f32> = outputs["output1"].try_extract_tensor()?;

DynValue can be upcast()ed to the more specialized types, like DynMap or Tensor<T>:

let tensor: Tensor<f32> = value.upcast()?;
let map: DynMap = value.upcast()?;

Similarly, a strongly-typed value like Tensor<T> can be downcast back into a DynValue or DynTensor.

let dyn_tensor: DynTensor = tensor.downcast();
let dyn_value: DynValue = tensor.into_dyn();

Tensor extraction directly returns an ArrayView

extract_tensor (and now try_extract_tensor) now return an ndarray::ArrayView directly, instead of putting it behind the old ort::Tensor<T> type (not to be confused with the new specialized value type). This means you don't have to .view() on the result:

-let generated_tokens: Tensor<f32> = outputs["output1"].extract_tensor()?;
-let generated_tokens = generated_tokens.view();
+let generated_tokens: ArrayViewD<f32> = outputs["output1"].try_extract_tensor()?;

Full support for sequence & map values

You can now construct and extract Sequence/Map values.

Value views

You can now obtain a view of any Value via the new view() and view_mut() functions, which operate similar to ndarray's own view system. These views can also now be passed into session inputs.

Mutable tensor extraction

You can extract a mutable ArrayViewMut or &mut [T] from a mutable reference to a tensor.

let (raw_shape, raw_data) = tensor.extract_raw_tensor_mut();

Device-allocated tensors

You can now create a tensor on device memory with Tensor::new & an allocator:

let allocator = Allocator::new(&session, MemoryInfo::new(AllocationDevice::CUDAPinned, 0, AllocatorType::Device, MemoryType::CPUInput)?)?;
let tensor = Tensor::<f32>::new(&allocator, [1, 128, 128, 3])?;

The data will be allocated by the device specified by the allocator. You can then use the new mutable tensor extraction to modify the tensor's data.

What if custom operators were 🚀 blazingly 🔥 fast 🦀?

You can now write custom operator kernels in Rust. Check out the custom-ops example.

Custom operator library feature change

Since custom operators can now be written completely in Rust, the old custom-ops feature, which enabled loading custom operators from an external dynamic library, has been renamed to operator-libraries.

Additionally, Session::with_custom_ops_lib has been renamed to Session::with_operator_library, and the confusingly named Session::with_enable_custom_ops (which does not enable custom operators in general, but rather attempts to load onnxruntime-extensions) has been updated to Session::with_extensions to reflect its actual behavior.

Asynchronous inference

Session introduces a new run_async method which returns inference results via a future. It's also cancel-safe, so you can simply cancel inference with something like tokio::select! or tokio::time::timeout.

---

If you have any questions about this release, we're here to help:

• Discord: #💬｜ort-discussions
• GitHub Discussions

Love ort? Consider supporting us on Open Collective 💖

❤️💚💙💛

v2.0.0-alpha.4

Features

• af97600 Add support for extracting sequences & maps.

Changes

• 153d7af Remove built-in ONNX Model Zoo structs (note that you can still use with_model_downloaded, just now only with URLs)

This is likely one of the last alpha releases before v2.0 becomes stable 🎉

Love ort? Consider supporting us on Open Collective 💖

❤️💚💙💛

v2.0.0-alpha.3

Fixes

• 863f1f3 Pin Model Zoo URLs to the old repo structure, new models will be coming soon.

Features

• 32e7fab Add ort::init_from on feature load-dynamic to set the path to the dylib at runtime.
• 52559e4 Cache downloaded binaries & models across all projects. Please update to save my bandwidth =)
• 534a42a Removed with_log_level. Instead, logging level will be controlled entirely by tracing.
• a9e146b Implement TryFrom<(Vec<i64>, Arc<Box<[T]>>)> for Value, making default-features = false more ergonomic
• 152f97f Add Value::dtype() to get the dtype of a tensor value.

Changes

• 32e7fab Remove the dependency on once_cell.
• acfa782 Remove the ORT_STRATEGY environment variable. No need to specify ORT_STRATEGY=system anymore, you only need to set ORT_LIB_LOCATION.

Love ort? Consider supporting us on Open Collective 💖

❤️💚💙💛

v2.0.0-alpha.2

Fixes

• 6938da1 Fix compilation on Windows in some cases when using pyke binaries by linking to DirectML dependencies.
• 2373ec5 Fix linking for Android. (#121)
• b04a964 Fix linking for iOS and add profile option (#121)

Features

• 30360eb Update binaries to ONNX Runtime v1.16.3 - compile Linux libraries on Ubuntu 20.04 to fix glibc issues.
• 9ed222f Make XNNPACK & ArmNN structs public

Love ort? Consider supporting us on Open Collective 💖

❤️💚💙💛

v2.0.0-alpha.1

This is the first alpha release for ort v2.0. This update overhauls the API, improves performance, fixes bugs, and makes ort simpler to use than ever before.

Our shiny new website will help you migrate: https://ort.pyke.io/migrating/v2

Stuck? We're here to help:

• Discord: #💬｜ort-discussions
• GitHub Discussions

A huge thank you to those who contributed to this release: Ben Harris, Ivan Krivosheev, Rui He, Andrea Corradi, & Lenny

Love ort? Consider supporting us on Open Collective 💖

❤️💚💙💛

v1.16.3

Changes

• e5f48ca Fixes a segfault that can occur when explicitly registering the CPU execution provider.

v1.16.2

Changes

• 3b23cea Fix usage of TensorRT execution provider, part 2

v1.16.1

Changes

• 99c5d43 Fix build on ARM targets.

v1.16.0

Changes

• d28e8ba Update to ONNX Runtime v1.16.0