Ops and Kernels

Understand the difference between Schema and Kernel

ONNX

• Read onnx.proto and onnx-ml.proto and understand the design principle behind it

• Get familiar with the operators

  • Must know: Dropout, MatMul, Gemm, Transpose, ReduceSum, Reshape

• Understand the concept and purpose of opset, domain

  • When to use which?

    • onnx domain
    • ms domain

• Understand the C++ data structure in onnx::TensorProto, onnx::AttributeProto, onnx::TypeProto

• Understand how Shape and Type Inference works in the schema definition

• Function Ops

Op Schema

• Understand the difference among the following 3 sets of schema. When to use which?

  • ONNX's op Schema in defs.cc

  • contrib ops in contrib_defs.cc

    • Good to know: LayerNorm, Gelu

  • training ops in training_op_defs.cc

• Action: Add an op or update an op's schema to get hands-on experience

Op Kernels

• Kernel Declaration and Registry

  • Understand when to use which registry for a kernel

  • Inference Kernels

    • ONNX Op Kernels

      • cpu_execution_provider.cc
      • cuda_execution_provider.cc

    • Contrib Op Kernels

      • cpu_contrib_kernels.cc
      • cuda_contrib_kernels.cc
      • Advanced: rocm_contrib_kernels.cc

  • Training Kernels

    • cpu_training_kernels.cc
    • cuda_training_kernels.cc
    • Advanced: rocm_training_kernels.cc

• Kernel Implementation

  • Tensor vs. OrtValue

    • Read tensor.h and ml_value.h
    • What's the difference between Tensor and OrtValue? Why we need two classes?
    • How to get a Tensor from OrtValue?
    • How to get data's raw pointer from a Tensor?

  • Kernel Definition

    • When to use Alias() and VariadicAlias()?
    • How to set TypeConstraint()?
    • When to use InputMemoryType<OrtMemTypeCPUInput>?

  • CPU Kernel vs. CUDA Kernel

    • What does it mean to have a CPU input/output for a CUDA kernel?

  • Gradient Kernels

    • Examples

      • Easy: DropoutGrad, GeluGrad
      • Medium: GatherGrad
      • Hard: LayerNormalizationGrad

    • Understand how to write unit tests to check gradient's correctness

  • Understand how to user OpTester in unit tests

  • Action: Implement a kernel to get hands-on experience