NNGraph input/output valid by register tensors

[chengtbf]

用于同时解决：

1. nn.Graph 非对称 Consistent Input/Output Tensor 在每个 rank 上有无分量 决定 是否 send Push/Pull CB 的 问题。
2. nn.Graph 检查每个 step 输入 tensor 的 meta 信息与编译时的 tensor meta 一致

• nn.Graph register_input/output_op_names -> register_input/output_op_names_and_tensors
• NNGraphIf 提供 input/output_valid 接口，用于表示每个 input/output tensor 有无在本 rank 上的分量
• RunLazyJobInstructionType::Compute 给 BufferMgr Send JobInstance 的 Push/Pull CB 时，根据 io valid 过滤跳过无分量的 op
  - [ ] 每次 RunLazyNNGraph 检查 input tensor meta 信息合法 NNGraph RunLazyJob check static input tensor meta #6243
• 非对称 超过 buffer size 的 nn.Graph 单测通过

[chengtbf]

核心逻辑，根据 NNGraphIf 提供的 inputs/outputs_valid 接口，决定是否跳过 Send Push/Pull CB

[chengtbf]

判断是否在本 rank 上有分量，表示是否 valid

[chengtbf]

这个分支可以 Review 了，解决了 issue：

• 流水并行出错记录： header_size == rhs->blob_desc().ByteSizeOfBlobHeader() 报错 #6226

发现的执行超过 128 iter 会死锁的 BUG

@leaves-zwx @strint @lixinqi @doombeaker

[strint]

parallel_id看起来是一个placement内的rank id编号，如果发现当前的全局rank不属于该tensor的placement，那么就会查到一个空的placement内的rank id编号，表示该tensor在该全局rank没有分量。

这个判断是否有本rank分量的接口后面貌似可以考虑包装下会更直接？

[chengtbf]

是的

[github-actions]

CI failed, removing label automerge

[github-actions]

Speed stats:

GPU Name: GeForce GTX 1080 

OneFlow resnet50 time: 128.0ms (= 6398.8ms / 50, input_shape=[16, 3, 224, 224])
PyTorch resnet50 time: 141.4ms (= 7069.4ms / 50, input_shape=[16, 3, 224, 224])
✔️ Relative speed: 1.10 (= 141.4ms / 128.0ms)

OneFlow resnet50 time: 74.2ms (= 3711.3ms / 50, input_shape=[8, 3, 224, 224])
PyTorch resnet50 time: 83.0ms (= 4147.6ms / 50, input_shape=[8, 3, 224, 224])
✔️ Relative speed: 1.12 (= 83.0ms / 74.2ms)

OneFlow resnet50 time: 47.9ms (= 2397.2ms / 50, input_shape=[4, 3, 224, 224])
PyTorch resnet50 time: 58.8ms (= 2942.0ms / 50, input_shape=[4, 3, 224, 224])
✔️ Relative speed: 1.23 (= 58.8ms / 47.9ms)

OneFlow resnet50 time: 41.5ms (= 2077.0ms / 50, input_shape=[2, 3, 224, 224])
PyTorch resnet50 time: 48.8ms (= 2442.1ms / 50, input_shape=[2, 3, 224, 224])
✔️ Relative speed: 1.18 (= 48.8ms / 41.5ms)

OneFlow resnet50 time: 34.6ms (= 1729.3ms / 50, input_shape=[1, 3, 224, 224])
PyTorch resnet50 time: 38.9ms (= 1946.9ms / 50, input_shape=[1, 3, 224, 224])
✔️ Relative speed: 1.13 (= 38.9ms / 34.6ms)

OneFlow resnet50 time: 160.4ms (= 8020.4ms / 50, input_shape=[16, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 166.0ms (= 8297.7ms / 50, input_shape=[16, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.03 (= 166.0ms / 160.4ms)

OneFlow resnet50 time: 102.9ms (= 5144.7ms / 50, input_shape=[8, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 111.8ms (= 5589.3ms / 50, input_shape=[8, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.09 (= 111.8ms / 102.9ms)

OneFlow resnet50 time: 75.3ms (= 3762.8ms / 50, input_shape=[4, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 80.9ms (= 4045.7ms / 50, input_shape=[4, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.08 (= 80.9ms / 75.3ms)

OneFlow resnet50 time: 70.1ms (= 3506.1ms / 50, input_shape=[2, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 67.2ms (= 3361.5ms / 50, input_shape=[2, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 0.96 (= 67.2ms / 70.1ms)

OneFlow resnet50 time: 72.6ms (= 3629.6ms / 50, input_shape=[1, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 65.6ms (= 3280.7ms / 50, input_shape=[1, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 0.90 (= 65.6ms / 72.6ms)