[Tracking] ONNX 1.9 work items

[askhade]

Creating this issue to collect and track all the work items for ONNX 1.9 release.
The release is planned for March 2021.

@onnx/sig-archinfra , @onnx/sig-operators , @onnx/steering-committee , @sveta-levitan, @wschin , @chinhuang007 and community members, please add work items here so that we can scope the release.

Thanks!

Currently planned workitems for 1.9

1. Removing Optimizers from onnx packages @daquexian
2. Updates to checker and shape inference exception handling @askhade
3. Selective schema loading @jcwchen
4. Opset 14 updates
5. Updates to external data helpers (add more options to control which tensors are serialized, add a new API to convert and serialize model in 1 step as opposed to the 2 steps required now) @annajung
6. Enhance ONNX type-checker: currently it does not check type-constraints specified in the op-schema spec @jcwchen
7. Enhance the sparse-tensor constants to allow block-sparse format? <if time permits; needs approval from community>
8. Should we migrate sparse-tensor type from ONNXML to ONNX?

[spandantiwari]

For opset 14, two possibilities to consider are:

1. Adding trilu op to spec.
2. Standardize specification of optional inputs/outputs.

[askhade]

@spandantiwari : Thanks for the inputs. Can you elaborate more on your 2nd bullet... I don't understand what is the exact work that needs to go in for this item.

[gramalingam]

Some possible enhancements (though not sure whether it should be in 1.9; depends on workload and prioritization).

1. Enhance ONNX type-checker: currently it does not check type-constraints specified in the op-schema spec (unlike ORT).
2. Enhance the sparse-tensor constants to allow block-sparse format?
3. Should we migrate sparse-tensor type from ONNXML to ONNX?

[jcwchen]

There is still a IR gap issue for version_converter: #2873. Anyone interested is welcome.

[daquexian]

Is there any plan to support python 3.9 in onnx 1.9 release?

[neginraoof]

Adding Trilu operator: #3291

[BowenBao]

Extending type support for math ops(Add, Sub, Mul, Div) #3334 (for HuggingFace/transfo_xl)

[jcwchen]

• Add weekly CI: provide ONNX TestPyPI packages for verification Add weekly CI: provide ONNX TestPyPI packages for verification #3283
• Selective load ONNX schema by specific opset_version Selective load ONNX schema by specific opset_version #3266

@daquexian According to the discussion with release manger of ONNX 1.9, Python 3.9 support is in the plan. Thanks.

[etusien]

Key Updates

• Removed Optimizers from ONNX packages remove optimizer related codes #3288
• Selective schema loading by specific opset_version Selective load ONNX schema by specific opset_version #3266
• Updates to external data helpers (more options to control which attributes are converted to external data and whether a model should be saved) Modify existing function in external data helper #3280
• New adapter for opset version converter New adapters #3343

Opset version 14

• New operator:
  • HardSwish Add HardSwish operator  #3332
  • Trilu Adding Trilu operator #3291
• Extended supported types to include uint8, int8, uint16, and int16. Extend dtype support for math ops #3334
  • Add
  • Div
  • Mul
  • Sub
• Added allowzero attribute to Reshape operator Update Reshape op: Add 'allowzero' attribute - Cont #3113
• Allowed recurrent operations to be batchwise Allow recurrent operations to be batchwise (reprise 2, signoff) #3217
  • GRU
  • LSTM
  • RNN
• Expanded Cum-Sum to float16 and bfloat16 data types CumSum-14: add data types float16 and bfloat16 #3195
• Expanded Relu to all signed data types Allow all signed types to be used with Relu #3141
• Added training-mode support to BatchNorm Updating BatchNorm to add training_mode #3333

API

• onnx.OnnxParser Parser for a textual syntax of ONNX models ONNX Parser #3194

Infrastructure

• Removed Python 3.5 in all release pipelines Deprecate Python 3.5 Release due to Python 3.5 incompatibility #3353
• Added Python 3.9 to all release pipelines Add Python 3.9 to release pipeline  #3352
• Added weekly CI: provide ONNX TestPyPI packages for verification Add weekly CI: provide ONNX TestPyPI packages for verification #3283
• Reduced binary size for Linux and Mac package Reduce binary size for Linux and Mac package #3337
• Added check for uploaded/generated backend test data in release CIs Check uploaded/generated backend test data in release CIs #3274
• Enabled no exception build and updated exception handling for DataTypeUtils
  enable no exception build and update exception handling for DataTypeUtils #3265

Bug fixes

• Added missing test for ConvInteger without padding remove optimizer related codes #3288
• Updated Resize op test to opset 13
• Expanded Ir_pb_converter to empty shape fix bug that ir_pb_converter introduces empty shape #3279

Notes

• Be aware of protobuf version gap issue (like building onnx with protobuf>=3.12 is not compatible with older protobuf)

[etusien]

In the comment above, I proposed the notes regarding the upcoming 1.9 release. Please let me know about any mistakes or missing scope items. Thanks!