FLOPs count seems to be off #17

matthewygf · 2019-06-05T15:42:44Z

Hi Luke, Thanks for your great work !

I am interested in the FLOPs of the models implemented.

I have always been using this to count FLOPs.

And for most cases, models from vision:

Resnet50
VGG19
Densenet121
Densenet169
Shufflenetv2_2_0

they all seems to match the FLOPs count from paper.

However in this case it is not. I can not think of a reason why, when i traverse the code down to each module, even added the FLOPs count for both padding inside the conv2dblock and Swish activation .

Do you have any idea?

Thanks in advance.

sdoria · 2019-06-05T16:53:30Z

Hi Matthew,
Could you share your FLOPs results? I am also wondering why EfficientnetB1 is taking longer to train (time per epoch) than resnet50, using FP16.
Thanks.

matthewygf · 2019-06-05T18:00:49Z

@sdoria

Resnet50: 4GFLOPs

Resnet50 is 6 times more flops than the EfficientNetB1 here.

EfficientNetB0: 4.1MFLOPs, Paper reported: 3.9MFLOPs
EfficientNetB1: 6.2 MFLOPs, Paper reported: 7MFLOPs
EfficientNetB2: 7.1 MFLOPs, Paper reported: 1GFLOPs
EfficientNetB3: 1GFLOPs, Paper reported: 1.8GFLOPs

Hope it helps.

matthewygf · 2019-06-05T21:37:20Z

@sdoria Actually i think one of the reason why EfficientNetB1 is slower than ResNet50, could be the control flow logic in Padding, deciding whether to pad.
GPU sync for diverge path is costly, that might be the case.

sdoria · 2019-06-05T22:59:09Z

I assume you are talking about the logic behind 'same' padding in Conv2dSamePadding. I don't think that's a problem. I have an alternative version that sets padding = kernel_size//2 instead and it trains at the same speed (time per epoch)

zhjpqq · 2019-06-12T07:42:10Z

My FLOPS calculation code, http://studyai.com/article/a718990b

It is more close to paper than your method, but still different.

zhjpqq · 2019-06-15T03:41:08Z

For some paper, the "multiply_adds==True",
while for other papers the "multiply_adds==False".
code is here: http://studyai.com/article/a718990b

Under "multiply_adds==False", the results are below, which are the same to paper.

#model depth param GFLOPs

effb0 82l 5.28M 0.393G

effb1 116L 7.79M 0.697G

effb2 116L 9.11M 1.007G

effb3 131L 12.23M 1.851G

effb4 161L 19.34M 4.443G

matthewygf · 2019-06-19T10:27:40Z

@zhjpqq thank you for you info :)

maoyichun · 2019-12-06T11:29:11Z

@zhjpqq hello, I run your code but the results is different from yours:
efficient b0
*** Number of layers: 82 , conv2d: 81, classifier: 1 ...

Number of FLOPs: 0.00802G

*** Number of params: 5.288548 million...

matkalinowski · 2020-08-22T15:40:42Z

@zhjpqq Could you share the code you used to receive those results here? I am not able to enter site from the link you have sent.

EDIT_0:
I found probably the source code that was mentioned earlier and it is here but I have the same results as @maoyichun....

EDIT_1:
Reason of low FLOPS looks to be a missing calculation for CONV operations. When running this script with:

multiply_adds=False
input_size set according to EfficientNet definition
it works as described.

EDIT_2:
After running some models I found out increased error in bigger models. It looks like the cause of it was error in static padding calculation going from _depthwise_conv in block 16. Run code below on model B1 to see what I am talking about.

dict(dict(model.named_modules())['_blocks.16'].named_modules())['_depthwise_conv']

I think I fixed it in this pull request: #223

I am also using different flop_count method that I recommend

shawnricecake · 2020-09-18T18:04:05Z

For some paper, the "multiply_adds==True",
while for other papers the "multiply_adds==False".
code is here: http://studyai.com/article/a718990b

Under "multiply_adds==False", the results are below, which are the same to paper.

#model depth param GFLOPs

effb0 82l 5.28M 0.393G

effb1 116L 7.79M 0.697G

effb2 116L 9.11M 1.007G

effb3 131L 12.23M 1.851G

effb4 161L 19.34M 4.443G

Hi,
I can not open the link u gave above.
Can you give me a new one for calculating flops of efficientnet?

matkalinowski · 2020-09-18T20:11:19Z

@shen494157765 Use this one: https://github.com/facebookresearch/fvcore/blob/ffd5dfff8ee6d5a88939376f208b08022562e789/fvcore/nn/flop_count.py#L28 it should work just fine.

shawnricecake · 2020-09-18T20:27:18Z

@shen494157765 Use this one: https://github.com/facebookresearch/fvcore/blob/ffd5dfff8ee6d5a88939376f208b08022562e789/fvcore/nn/flop_count.py#L28 it should work just fine.

Hi,

Yes, I used the package of fvcore, and my code is below:

from fvcore.nn import flop_count
from efficientnet_pytorch import EfficientNet
model = EfficientNet.from_name('efficientnet-b2')
netinput = torch.randn(1, 3, 224, 224)
final_count, skipped_ops = flop_count(model, (netinput, )) 
print(final_count)

but the result is

Skipped operation aten::batch_norm 69 time(s)
Skipped operation prim::PythonOp 69 time(s)
Skipped operation aten::adaptive_avg_pool2d 24 time(s)
Skipped operation aten::sigmoid 23 time(s)
Skipped operation aten::mul 39 time(s)
Skipped operation aten::rand 16 time(s)
Skipped operation aten::add 32 time(s)
Skipped operation aten::div 16 time(s)
Skipped operation aten::dropout 1 time(s)
defaultdict(<class 'float'>, {'conv': 0.65755544, 'addmm': 0.001408})

which is different from the result in paper. (efficientnet b2)
and for other model (efficientnet b0, b1 ,... b7), the results are also different.

Thanks

matkalinowski · 2020-09-19T08:07:25Z

There is ongoing work on pytorch side. There are multiple packages to calculate flops but this one is the closest to original I was able to find.

Other packages you can try:

shawnricecake · 2020-09-19T22:42:48Z

There is ongoing work on pytorch side. There are multiple packages to calculate flops but this one is the closest to original I was able to find.

Other packages you can try:

https://github.com/Lyken17/pytorch-OpCounter

https://github.com/1adrianb/pytorch-estimate-flops

Hi,

Thanks for your reply, I have tried that 'ongoing' work, and I got the results which were similar to the results I got from the tool 'fvcore' (https://github.com/facebookresearch/fvcore/blob/master/fvcore/nn/flop_count.py)

And now I just think that the only way to calculate the flops of this efficientnet is calculating by myself.....

Thanks

matkalinowski · 2020-09-20T12:54:43Z

Also @shen494157765 please make sure you are using the newest verion of this library. I have proposed fix (#223) to the architecture lately.

shawnricecake · 2020-09-20T18:44:06Z

Also @shen494157765 please make sure you are using the newest verion of this library. I have proposed fix (#223) to the architecture lately.

Hi,

Yes, I have used the latest version when I calculated the flops.
What I have done yesterday was that I just calculated the flops of efficientnet b0 by myself, which means I calculated conv2d in efficientnet b0 one by one. (if you want to see that I can share it to you)

The result is that the efficientnet b0 has 0.821842112 GFLOPs (about 0.41 GMACs), which is similar to 0.39 showed in paper.

(Just ignore the GFLOPs GMACs, I just think that the result in paper is GMACs not GFLOPs, buy anyway, many papers has same kind of definition.)

But for other kinds of efficientnet (b1 - b7), there are too much layers for them, so I did not choose them to verify the result.

For my own experiments, what I need is part of layers in efficientnet, and that is the reason that I need to know how to calculate the flops of efficientnet.

Now, I just calculated it manually with every conv2d.

If you know there is some tool that can get a similar result to the efficientnet paper, please let me know.

I do not know that if you have tried the tool you recommended above (can get the closet result) in effcientnet b6 or b7, that was a real disaster.

Thanks

Monkey-D-Luffy-star · 2021-08-09T13:17:03Z

@matthewygf I think your input size maybe (3,224,224) for all models(efficientnet-b0,efficientnet-b1,..etc.)，but the input size of paper is variable for different models

matthewygf closed this as completed Jun 19, 2019

matkalinowski mentioned this issue Aug 25, 2020

static padding fixed #223

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

FLOPs count seems to be off #17

FLOPs count seems to be off #17

matthewygf commented Jun 5, 2019

sdoria commented Jun 5, 2019

matthewygf commented Jun 5, 2019 •

edited

matthewygf commented Jun 5, 2019

sdoria commented Jun 5, 2019

zhjpqq commented Jun 12, 2019

zhjpqq commented Jun 15, 2019

matthewygf commented Jun 19, 2019

maoyichun commented Dec 6, 2019

matkalinowski commented Aug 22, 2020 •

edited

shawnricecake commented Sep 18, 2020

effb0 82l 5.28M 0.393G

effb1 116L 7.79M 0.697G

effb2 116L 9.11M 1.007G

effb3 131L 12.23M 1.851G

effb4 161L 19.34M 4.443G

matkalinowski commented Sep 18, 2020

shawnricecake commented Sep 18, 2020

matkalinowski commented Sep 19, 2020

shawnricecake commented Sep 19, 2020

matkalinowski commented Sep 20, 2020

shawnricecake commented Sep 20, 2020

Monkey-D-Luffy-star commented Aug 9, 2021

FLOPs count seems to be off #17

FLOPs count seems to be off #17

Comments

matthewygf commented Jun 5, 2019

sdoria commented Jun 5, 2019

matthewygf commented Jun 5, 2019 • edited

matthewygf commented Jun 5, 2019

sdoria commented Jun 5, 2019

zhjpqq commented Jun 12, 2019

zhjpqq commented Jun 15, 2019

effb0 82l 5.28M 0.393G

effb1 116L 7.79M 0.697G

effb2 116L 9.11M 1.007G

effb3 131L 12.23M 1.851G

effb4 161L 19.34M 4.443G

matthewygf commented Jun 19, 2019

maoyichun commented Dec 6, 2019

matkalinowski commented Aug 22, 2020 • edited

shawnricecake commented Sep 18, 2020

effb0 82l 5.28M 0.393G

effb1 116L 7.79M 0.697G

effb2 116L 9.11M 1.007G

effb3 131L 12.23M 1.851G

effb4 161L 19.34M 4.443G

matkalinowski commented Sep 18, 2020

shawnricecake commented Sep 18, 2020

matkalinowski commented Sep 19, 2020

shawnricecake commented Sep 19, 2020

matkalinowski commented Sep 20, 2020

shawnricecake commented Sep 20, 2020

Monkey-D-Luffy-star commented Aug 9, 2021

matthewygf commented Jun 5, 2019 •

edited

matkalinowski commented Aug 22, 2020 •

edited