Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Proposal] ONNX Interface for Framework Integration (previously ONNX Backend API) header and docs #551

Merged
merged 21 commits into from Apr 25, 2018
Merged
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
4aa4cee
ONNX Backend API header and docs
Feb 22, 2018
9b2a21c
Fix numbering in Backend API readme
Feb 23, 2018
a5fc170
Refactor onnxGetBackendInfo interface
Feb 24, 2018
4a54b06
CMake target for backend interface header
Feb 26, 2018
4e4f26f
Fix include directory in CMake onnxbe_interface target
Feb 26, 2018
be54680
Remove limit on the number of tensor dimensions
Mar 2, 2018
6d08dc4
Generate compile warning for unchecked error code
Mar 2, 2018
371a51a
Add complex data types from TensorProto.DataType enum
Mar 7, 2018
bd4291e
Rename Backend API to ONNX Interface for Framework Integration (ONNXIFI)
Apr 2, 2018
8d9273a
Extend interface for additional use-cases
Apr 3, 2018
90761cd
Consistently refer to ONNXIFI objects as ONNXIFI backend/graph/event …
Apr 5, 2018
baa4316
Add missing event-related error codes:
Apr 10, 2018
6d38c59
Add new error code for unsupported tensor metainformation parameters
Apr 10, 2018
eca884b
More consistent handling of ONNXIFI_STATUS_INVALID_SHAPE and ONNXIFI_…
Apr 10, 2018
f4158e9
Add error codes for inconsistent shapes and data types
Apr 13, 2018
ea935da
Specify that onnxGraph and onnxEvent must be deinitialized before onn…
Apr 16, 2018
492a317
Clarify documentation that ModelProto message is not valid after onnx…
Apr 16, 2018
4e2353e
Introduce ONNXIFI_BACKEND_PROPERTY_OPTIMIZATION property for backend …
Apr 16, 2018
81aeb45
Support hot-pluggable backends
Apr 23, 2018
da8928c
Rename FRAMEWORK device type to HETEROGENEOUS device type
Apr 25, 2018
857c70a
Document ONNXIFI_STATUS_BACKEND_UNAVAILABLE in function return values
Apr 25, 2018
File filter
Filter file types
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.

Always

Just for now

@@ -315,3 +315,7 @@ endif()
if(APPLE)
set_target_properties(onnx PROPERTIES LINK_FLAGS "-undefined dynamic_lookup")
endif()

# ---[ ONNX Interface for Framework Integratin (ONNXIFI)
add_library(onnxifi INTERFACE)
target_include_directories(onnxifi INTERFACE onnx)
@@ -0,0 +1,52 @@
# ONNX Interface for Framework Integration (ONNXIFI) Proposal

We propose a cross-platform API for loading and executing ONNX graphs on optimized backends. High-level frameworks and applications can use this API to execute neural network and machine learning models. Hardware vendors can implement this API to expose specialized hardware accelerators and highly optimized software infrastructure to the users.

## Core Features

- Standardized interface for neural network inference on special-purpose accelerators, CPUs, GPUs, DSPs, and FPGAs
- Based on widely supported technologies
- C API for function calls
- ONNX format for passing model graphs
- NCHW tensor layout for passing inputs and outputs
- Dynamic discovery of available backends for model execution
- Multiple backends from different vendors can co-exist on the same system
- Dynamic discovery of supported ONNX Operators on each backend

### Optional Features:

- Graphs with variable-shape inputs and/or outputs
- Graphs with data-dependendent output shapes

## How to Use ONNX Interface for Framework Integration

0. (Optional) Use `onnxifi_library_load` to dynamically load the ONNX Interface for Framework Integration library.
1. Call `onnxGetNumBackends` to get the number of available backends. Note that it can be 0.
2. Call `onnxGetBackendInfo` to check additional information about any available backend.
3. Call `onnxGetBackendCompatibility` to check if your model, or parts of it, can run on the backend.
4. Call `onnxInitBackend` to initialize a backend, then call `onnxInitGraph` to offload one or more model graphs to the backend.
5. Call `onnxSetGraphIO` to set inputs and output for the graph, then call `onnxRunGraph` to execute the graph(s). If your model works with fixed-size inputs, one call to `onnxSetGraphIO` is sufficient for multiple `onnxRunGraph` calls. For models with variable-size inputs, you'd need to call `onnxSetGraphIO` before each `onnxRunGraph` call.
6. When done using the model, release the model graph(s) with `onnxReleaseGraph`, then release the backend with `onnxReleaseBackend`

## How to Implement ONNX Interface for Framework Integration

The minimum functionality an ONNXIFI implementation must provide is the following:

- Support ONNX 1.0 model format.
- There is no minimum list of Operators a backed has to support.

This comment has been minimized.

@dzhulgakov

dzhulgakov Apr 2, 2018
Member

maybe we should put basic CNN here - but it's a separate discussion. @houseroad

This comment has been minimized.

@Maratyszcza

Maratyszcza Apr 2, 2018
Author Contributor

I'll leave it to Compliance Working Group.

This comment has been minimized.

@houseroad

houseroad Apr 3, 2018
Member

@dzhulgakov it's a great topic for the test and compliance working group. I have recorded it.

This comment has been minimized.

@larroy

larroy Apr 25, 2018

backed -> backend

- Support graph inputs / outputs in CPU memory.
- Support graph inputs / outputs with fixed shape, specified in GraphProto message.

### Discovery
Vendor-provided libraries should adhere to some rules to ensure discovery by ONNX-supported frameworks and applications:
- Use `libonnxifi-<backend>.so` filename on Linux/Android
- Use `libonnxifi-<backend>.dylib` filename on macOS
- Use `onnxifi-<backend>.dll` filename on Windows
- Append ONNX function names with `<BACKEND>` (a vendor may define `ONNXIFI_LIBRARY_SUFFIX=<BACKEND>` and when using `onnxifi.h` header).
`<backend>` is the vendor-specific name in lowercase, and `<BACKEND>` is the same name in uppercase. E.g. a vendor **Gamma** would provide `libonnxifi-gamma.so` for Linux systems, and this library would implement functions `onnxGetNumBackendsGAMMA`, `onnxGetBackendInfoGAMMA`, etc.

### Extensions

Hardware vendors are welcome to add their own extensions to ONNX backend interface. The backend interface offers several extension mechanisms:
- Experimental, exotic, or vendor-specific operators can be supported in a private domain using NodeProto.domain attribute.
- Vendor-provided ONNXIFI implementation can expose additional functions

This comment has been minimized.

@dzhulgakov

dzhulgakov Apr 2, 2018
Member

maybe expand it a bit: "expose additional functions in the C API (for example separate calls to setup device-specific async execution)". Otherwise it's not clear which functions we're talking about

ProTip! Use n and p to navigate between commits in a pull request.