Skip to content
ONNX Runtime: cross-platform, high performance scoring engine for ML models
C++ C# Assembly Python CMake C Other
Branch: master
Clone or download
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.github Issue template update (#1339) Jul 8, 2019
cmake update MKLML to version which contains fix for thread hang. (#1636) Aug 19, 2019
csharp Allow user disable multiple threading (#1647) Aug 20, 2019
dockerfiles Added license files in the base image (#1595) Aug 9, 2019
docs Update onnx test runner documentation (#1651) Aug 20, 2019
include/onnxruntime/core Allow user disable multiple threading (#1647) Aug 20, 2019
onnxruntime Fix memory leak in mlas unitest (#1654) Aug 20, 2019
package/rpm Upgrade version number for ORT in preparation for release (#1468) Jul 23, 2019
samples/c_cxx Add docs for the fns candy demo (#1479) Jul 24, 2019
tools Add AutoML to 3 main builds. (#1631) Aug 17, 2019
.clang-format Initial bootstrap commit. Nov 20, 2018
.clang-tidy Add remaining build options and make minor changes in documentation (#39 Nov 28, 2018
.dockerignore Allow building Docker container based on a different git repo. (#1222) Jun 20, 2019
.gitattributes Initial bootstrap commit. Nov 20, 2018
.gitignore Add an HTTP server for hosting of ONNX models (#806) May 1, 2019
.gitmodules Remove memory copy between TensorRT and CUDA (#1561) Aug 9, 2019
BUILD.md [OpenVINO-EP] Update hardware branding of VAD-R as VAD-M (#1552) Aug 5, 2019
CODEOWNERS Fix codeowners file Nov 28, 2018
CONTRIBUTING.md Miscellaneous fixes (#123) Dec 7, 2018
LICENSE Initial bootstrap commit. Nov 20, 2018
README.md Upgrade version number for ORT in preparation for release (#1468) Jul 23, 2019
ThirdPartyNotices.txt Update cgmanifest.json and ThirdPartyNotices.txt for DNNLibrary (#1431) Jul 18, 2019
VERSION_NUMBER Upgrade version number for ORT in preparation for release (#1468) Jul 23, 2019
build.amd64.1411.bat Initial bootstrap commit. Nov 20, 2018
build.bat Initial bootstrap commit. Nov 20, 2018
build.sh update Jan 9, 2019
cgmanifest.json update onnx to latest commit (#1622) Aug 16, 2019
requirements-dev.txt Initial bootstrap commit. Nov 20, 2018
requirements-doc.txt Update the documentation, run all examples during the generation of t… Dec 5, 2018
requirements.txt Initial bootstrap commit. Nov 20, 2018
setup.py Added missing libraries to Windows wheel (#1415) Jul 17, 2019

README.md

Build Status Build Status Build Status Build Status Build Status

ONNX Runtime is a performance-focused complete scoring engine for Open Neural Network Exchange (ONNX) models, with an open extensible architecture to continually address the latest developments in AI and Deep Learning. ONNX Runtime stays up to date with the ONNX standard with complete implementation of all ONNX operators, and supports all ONNX releases (1.2+) with both future and backwards compatibility. Please refer to this page for ONNX opset compatibility details.

ONNX is an interoperable format for machine learning models supported by various ML and DNN frameworks and tools. The universal format makes it easier to interoperate between frameworks and maximize the reach of hardware optimization investments.


Setup

Getting Started

More Info

Contributions and Feedback

License


Key Features

Run any ONNX model

ONNX Runtime provides comprehensive support of the ONNX spec and can be used to run all models based on ONNX v1.2.1 and higher. See version compatibility details here.

Note: Some operators not supported in the current ONNX version may be available as a Contrib Operator

Traditional ML support

In addition to DNN models, ONNX Runtime fully supports the ONNX-ML profile of the ONNX spec for traditional ML scenarios.

High Performance

ONNX Runtime supports both CPU and GPU. Using various graph optimizations and accelerators, ONNX Runtime can provide lower latency compared to other runtimes for faster end-to-end customer experiences and minimized machine utilization costs.

Currently ONNX Runtime supports the following accelerators:

Not all variations are supported in the official release builds, but can be built from source following these instructions.

We are continuously working to integrate new execution providers for further improvements in latency and efficiency. If you are interested in contributing a new execution provider, please see this page.

Cross Platform

API documentation and package installation

ONNX Runtime is available for Linux, Windows, Mac with Python, C#, and C APIs, with more to come! If you have specific scenarios that are not currently supported, please share your suggestions and scenario details via Github Issues.

Installation

Quick Start: The ONNX-Ecosystem Docker container image is available on Dockerhub and includes ONNX Runtime (CPU, Python), dependencies, tools to convert from various frameworks, and Jupyter notebooks to help get started.

Additional dockerfiles for some features can be found here.

APIs and Official Builds

API Documentation

Official Builds

CPU (MLAS+Eigen) CPU (MKL-ML) GPU (CUDA)
Python pypi: onnxruntime

Windows (x64)
Linux (x64)
Mac OS X (x64)
-- pypi: onnxruntime-gpu

Windows (x64)
Linux (x64)
C# Nuget: Microsoft.ML.OnnxRuntime

Windows (x64, x86)
Linux (x64, x86)
Mac OS X (x64)
Nuget: Microsoft.ML.OnnxRuntime.MKLML

Windows (x64)
Linux (x64)
Mac OS X (x64)
Nuget: Microsoft.ML.OnnxRuntime.Gpu

Windows (x64)
Linux (x64)
C Nuget: Microsoft.ML.OnnxRuntime

.zip, .tgz

Windows (x64, x86)
Linux (x64, x86)
Mac OS X (x64
Nuget: Microsoft.ML.OnnxRuntime.MKLML

Windows (x64)
Linux (x64)
Mac OS X (x64)
Nuget: Microsoft.ML.OnnxRuntime.Gpu

.zip, .tgz

Windows (x64)
Linux (x64)

System Requirements (pre-requisite dependencies)

  • ONNX Runtime binaries in the CPU packages use OpenMP and depend on the library being available at runtime in the system.
    • For Windows, OpenMP support comes as part of VC runtime. It is also available as redist packages: vc_redist.x64.exe and vc_redist.x86.exe
    • For Linux, the system must have libgomp.so.1 which can be installed using apt-get install libgomp1.
  • GPU builds require the CUDA 10.0 and cuDNN 7.3 runtime libraries being installed on the system. Older releases used 9.1/7.1 - please refer to release notes for more details.
  • Python binaries are compatible with Python 3.5-3.7. See Python Dev Notes
  • Certain operators makes use of system locales. Installation of the English language package and configuring en_US.UTF-8 locale is required.
    • For Ubuntu install language-pack-en package
    • Run the following commands: locale-gen en_US.UTF-8 update-locale LANG=en_US.UTF-8
    • Follow similar procedure to configure other locales on other platforms.

Building from Source

If additional build flavors are needed, please find instructions on building from source at Build ONNX Runtime. For production scenarios, it's strongly recommended to build from an official release branch.

Dockerfiles are available here to help you get started.

Getting ONNX Models

Deploying ONNX Runtime

ONNX Runtime can be deployed to the cloud for model inferencing using Azure Machine Learning Services. See detailed instructions and sample notebooks.

ONNX Runtime Server (beta) is a hosted application for serving ONNX models using ONNX Runtime, providing a REST API for prediction. Usage details can be found here, and image installation instructions are here.

Examples and Tutorials

Python

Deployment with AzureML

C#

C/C++

Technical Design Details

Extensibility Options

Contribute

We welcome contributions! Please see the contribution guidelines.

Feedback

For any feedback or to report a bug, please file a GitHub Issue.

Code of Conduct

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.

License

MIT License

You can’t perform that action at this time.