Skip to content
Sherlock edited this page Mar 26, 2021 · 2 revisions

Basic building blocks to describe a computation Graph

  • Node (graph.h/.cc)

    • What's the difference between an Op and a Node?

    • What are the common properties for a node?

      • Can a node's name be empty?
      • What's the identifier of a node in a graph? Index or Name?
    • Advance: Function Ops, node with FunctionBody

  • Graph (graph.h/.cc)

    • How to traverse from one node to another node?

    • What's the difference between a GraphInput and an Initializer?

    • Look for an example using GetProducerNode() and GetConsumerNodes()

    • What's the purpose of Graph::Resolve()?

      • How is ShapeAndTypeInference invoked?
  • NodeArg (node_arg.h)

    • What the relationship between a graph edge and a NodeArg?
    • What's the unique identifier of a NodeArg in a graph?
    • Action: Look for some example using Graph::GetOrCreateNodeArg() (You will need to use this at some point)

Graph Transformers

  • Understand the difference between GraphTransformer and RewriteRule

  • Understanding the purpose of GraphTransformerManager

    • How to register a set of graph transformers into a session?
  • Understanding the two versions of graph_transformer_utils.cc (onnxruntime/orttraining ones)

  • Get familiar with graph_utils.cc

  • Experiment with onnx.helper to compose a onnx model from the script (see transpose_matmul_gen.py for examples)

  • Action: Implement a graph transformer to get hands-on experience

Training Graph

  • Understand the workflow of training graph transformation

  • Understand GraphAugmenter (graph_augmenter.h/.cc)

  • GradientGraphBuilder

    • Understand the purpose/usage of STOP_GRADIENT_EDGES
    • Understand the meaning of x_node_args/y_node_args
    • Advance: Understand the back-propagation process in GradientGraphBuilder::Build()
  • Per Op GradientBuilder

    • Understand the Gradient Registry (gradient_builder_registry.cc)

    • Understand the Gradient Builder Declaration (gradient_builder.h)

    • Read a few examples in Gradient Builder Implementation (gradient_builder.cc)

      • Understand the shorthands of I, GI, O, GO (gradient_builder_base.h)

      • Understand how gradient subgraph is composed with existing ops, followings are good examples

        • Easy: GetDropoutGradient, GetSqrtGradient
        • Medium: GetAddSubGradient, GetMulGradient
        • Hard: GetMatMulGradient, GetGemmGradient
      • Understand how broadcasting is handled when building gradient graph GradientBuilderBase::HandleBroadcasting()

      • Action: Implement a gradient definition for an op to get hands-on experience

      • Good reference implementation: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/ops/array_grad.py