skip connection structure

[ZhenpengChenCode]

It seems that the OpCounter doesn't take into account the skip connection structure.

[ZhenpengChenCode]

Resnet flops calculated by the OpCOunter is smaller than the value in paper.

[Lyken17]

Yes, as pytorch is a dynamic graph library, it is impossible to determine skip-connection structure at nn.Module level.

One quick fix is to write a special judge for inverted residual block. I'll try to provide a more precise one via tracing computational graph.

[Kylin9511]

@Lyken17 In torch1.1, a new placeholder module is provided for this kind of awkward situation.

See torch.nn.Idnetity

As I can see, you should add this Module into count_hooks.py and profile.py, counting the extra ops for skip connection.

And update your readme for special notice is important as well that one must use the standard module provided by PyTorch for skip connection to get an accurate FLOPs.

[Lyken17]

not really. Most existing codebase still defines the identity connection in forward. The real problem, as I mentioned before, is pytorch's define-by-run philosophy. The only way to record all operations is to trace the DAG.

In this case, please have a look at my labmate's project zhijian-liu/onnx-profiler

[Kylin9511]

What you mentioned is quite right. I didn't say it would fix this problem without changing the codes. If someone would like to call your hook based op-counter while considering skip connection, he can represent skip connection with nn.Identity. And by adding this module to your profile we can fix the problem.

It is kind of a patch, not a global solution.

By unify the model to onnx mode is a very interesting idea, I may look deeper into it some time.

[Kylin9511]

It is kind of like a user-defined patch, so I don't think you should update your project.