Fea/nn graph/warmup amp config

[strint]

• WarmUpLR
  • eager
  • graph
• amp & nn.Graph.config
• flow.config

[strint]

https://pytorch.org/docs/master/generated/torch.optim.lr_scheduler.WarmUpLR.html#torch.optim.lr_scheduler.WarmUpLR

和torch开发分支中一样的接口。

但是是个torch的增强版本：可以组合一个普通的LrScheduler。支持eager和graph。

[strint]

config接口调用的例子

[strint]

amp调用的例子

[chengtbf]

这个只是为了 debug 打印吧

[strint]

嗯，为了内部debug，加下滑线前缀的方法都是内部使用的私有方法，不保证稳定

[chengtbf]

LR 部分的内容 @leaves-zwx 文骁 Review 吧，通过了就能合并~

[chengtbf]

这里需要提供 key args 把，不然别人不知道 3000、2、1000 都是啥

[strint]

这里和torch接口对齐了，用户传参时也可以写 arg=val，也可以不写

[chengtbf]

对，但是我觉得我们的示例代码要写明白，参考：

# Assuming optimizer uses lr = 0.05 for all groups
# lr = 0.025    if epoch == 0
# lr = 0.03125  if epoch == 1
# lr = 0.0375   if epoch == 2
# lr = 0.04375  if epoch == 3
# lr = 0.005    if epoch >= 4
scheduler = WarmUpLR(self.opt, warmup_factor=0.5, warmup_iters=4, warmup_method="linear")
for epoch in range(100):
    train(...)
    validate(...)
    scheduler.step()

[chengtbf]

我们的测试脚本，用户也会尝试模仿。所以最好还是带上参数更明白

[strint]

有道理，已经加上

[github-actions]

Speed stats:

GPU Name: GeForce GTX 1080 

PyTorch resnet50 time: 140.1ms (= 7004.4ms / 50, input_shape=[16, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 128.4ms (= 6419.8ms / 50, input_shape=[16, 3, 224, 224], backward is enabled)
Relative speed: 1.09 (= 140.1ms / 128.4ms)

PyTorch resnet50 time: 84.5ms (= 4225.2ms / 50, input_shape=[8, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 74.6ms (= 3728.0ms / 50, input_shape=[8, 3, 224, 224], backward is enabled)
Relative speed: 1.13 (= 84.5ms / 74.6ms)

PyTorch resnet50 time: 57.6ms (= 2878.5ms / 50, input_shape=[4, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 47.3ms (= 2365.6ms / 50, input_shape=[4, 3, 224, 224], backward is enabled)
Relative speed: 1.22 (= 57.6ms / 47.3ms)

PyTorch resnet50 time: 49.2ms (= 2460.5ms / 50, input_shape=[2, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 41.4ms (= 2068.8ms / 50, input_shape=[2, 3, 224, 224], backward is enabled)
Relative speed: 1.19 (= 49.2ms / 41.4ms)

PyTorch resnet50 time: 44.7ms (= 2234.6ms / 50, input_shape=[1, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 35.3ms (= 1766.1ms / 50, input_shape=[1, 3, 224, 224], backward is enabled)
Relative speed: 1.27 (= 44.7ms / 35.3ms)

[chengtbf]

这里不是有吗？ 😂 @strint @leaves-zwx

flow.boxing.nccl.enable_use_compute_stream()