Count number of floating point operations per layer #6203

aizvorski · 2017-04-10T01:31:51Z

This PR partially implements the feature requested in #5625.

Opening it here for feedback on the implementation strategy.

Supported layers: Conv1D, Conv2D, Conv3D, Dense. Other layers return None, and don't count towards the total.

Behavior with different padding and strides seems okay.

Questions:

Should this go in the Layer class (as here), or in each individual layer implementation?
Should this rely on some knowledge about the layers (as here), or on backend features for counting flops? If backend, how?
Should a multiply-add be counted as one flop (as here) or two? Common usage seems to be one flop, eg ResNet paper.
Should layers like pooling or activations even be counted at all? (How many flops is an ELU?) It seems that these are not usually counted at all, in any case typically a few convolutions account for the bulk of the calculation.

fchollet · 2017-04-10T02:21:55Z

keras/engine/topology.py

+                output_area = np.prod( self.output_shape[1:-1] )
+                filter_area = np.prod( self.kernel_size )
+            if output_area is not None:
+                flops = input_filters * output_filters * filter_area * output_area


Instead of this switch, this should be implemented as a recursive call to layer methods (with a reasonable default for the base layer, leveraging the ndim of the weights).

The base layer should raise an error actually or None. Depend if we want everyone to provide this service. Every layer in keras' repo should provide this

@fchollet I replaced the switch with a default based on the kernel shape. What do you think?

Any layer can override count_ops, but I think at the moment this handles most cases with no need for overriding.

fchollet · 2017-04-10T02:22:47Z

keras/engine/topology.py

@@ -1229,6 +1229,45 @@ def count_params(self):
                                   self.name + '.build(batch_input_shape)`.')
        return sum([K.count_params(p) for p in self.weights])

+    def count_flops(self):
+        """Count the total number of floating point operations for a _forward_ calculation of this layer.


Some of your lines are too long. We don't strictly enforce line length, but please do your best to follow PEP8.

@fchollet Fixed

fchollet · 2017-04-10T02:24:22Z

keras/engine/topology.py

@@ -1229,6 +1229,45 @@ def count_params(self):
                                   self.name + '.build(batch_input_shape)`.')
        return sum([K.count_params(p) for p in self.weights])

+    def count_flops(self):


flops is generally used for floating point operations per second. Since this is merely a flop count, maybe we need a different name. Maybe.

Maybe just count_ops ?

That sounds good. There may be confusion with the concept of op in TF though (e.g. a dot product). But maybe not a big deal.

Another possibility is count_flop (without the s which may be confused for "per second"). I was concerned about confusion with op in TF just as @fchollet said that's why I didn't got for count_ops initially. Happy to rename it to whatever you guys prefer.

I vote count_ops since it is more consistent with count_params

I also don't think the confusion with TF ops is a big deal. As long as the docstring makes it very clear what is being counted.

One more note: I don't think op counting is a common enough use case that it should be displayed in model.summary().

@fchollet @farizrahman4u Okay, I renamed it to count_ops.

@fchollet I would argue in favor of having op counting in model.summary().

Op counts are a good predictor of speed, and can be very unevenly distributed among layers (so it is not obvious where to optimize). People tend to optimize what they can see, optimizing param counts has been very easy with keras, op counts / speed not so much until now.

Also, most papers seem to report both param counts and op counts as a measure of the "size" of the network.

If you are stongly against having op counting as default in model.summary(), perhaps there could be an option to display it. I would really like it to be the default though, with an option to hide it.

fchollet

As we discussed before, this should be a layer method, implemented on every layer. You cannot come up with a unique implementation that will work for every layer; the current attempt would only return a result for a subset of layers, and would give the wrong answer in most cases... computing the proper answer requires knowing how each layer works.

fchollet · 2017-04-25T20:39:09Z

keras/engine/topology.py

+                                   'You can build it manually via: `' +
+                                   self.name + '.build(batch_input_shape)`.')
+        ops = None
+        if K.image_data_format() == 'channels_last':


So you always return None in mode channels_first? Why? That's not the expected behavior.

You're right. Just didn't have time to implement yet; will add that soon.

fchollet · 2017-04-25T20:40:26Z

keras/engine/topology.py

+        ops = None
+        if K.image_data_format() == 'channels_last':
+            output_area = np.prod(self.output_shape[1:-1])
+            output_area = int(output_area)


You are later multiplying with a float. This has no effect on the output

output_area is an int except when it is calculated as np.prod([]) which returns 1.0 as a float (for Dense layers). Somewhat surprising behavior from numpy, this is just a workaround for it.

It does; two lines later you are multiplying your int with float, which returns a float. Hence casting to int has no effect on the final type.

fchollet · 2017-04-25T20:40:41Z

keras/engine/topology.py

+        if K.image_data_format() == 'channels_last':
+            output_area = np.prod(self.output_shape[1:-1])
+            output_area = int(output_area)
+            if hasattr(self, 'kernel'):


This seems strangely ad-hoc

Additionally this will give the wrong number for RNN layers

fchollet · 2017-04-25T20:41:39Z

keras/utils/layer_utils.py

-        line_length = line_length or 65
-        positions = positions or [.45, .85, 1.]
+        line_length = line_length or 80
+        positions = positions or [.40, .72, .85, 1.]


Screen real estate is scarce, and op counting is too niche to be part of the model summary (as we previously discussed).

@fchollet Mos notable deep learning papers I can think of report op counts. Why do you say it's niche?

The papers that report op count are those that are trying to make a point about op count efficiency, which is fairly niche. Most DL papers barely report their model architectures...

fchollet · 2017-04-25T20:43:35Z

keras/engine/topology.py

+            output_area = np.prod(self.output_shape[1:-1])
+            output_area = int(output_area)
+            if hasattr(self, 'kernel'):
+                ops = np.prod(self.kernel._keras_shape) * output_area


You should not rely on private property _keras_shape which may not be set

I see, this should use K.int_shape(self.kernel) instead?

Yes, which will be defined at least for the built-in layers

fchollet · 2017-04-25T20:45:53Z

keras/engine/topology.py

+                                   self.name + '.build(batch_input_shape)`.')
+        ops = None
+        if K.image_data_format() == 'channels_last':
+            output_area = np.prod(self.output_shape[1:-1])


This formula is wrong for most convolution layer configs because it doesn't take into account padding and strides

@fchollet I've checked and it seems correct to me. The output area is what determines the number of positions at which the convolution has to be calculated; stride and padding affect that, but only by changing the output area. Could you show a counter-example? (for a convolutional layer)

The formula is wrong because np.prod(self.output_shape[1:-1]) is not the number of positions where the kernel is applied, it is the maximum number of positions.

@fchollet Does the output shape include padding? That may be the problem you're referring to. I tried to think of any other way the output shape would not be equal to the number of positions the kernel is applied, but as far as I can tell input stride and input padding are already taken into account by this. A specific counter example would be greatly appreciated.

fchollet · 2017-04-25T20:54:34Z

Writing a new method for every layer would be a lot of work, too. And it would still be fairly brittle because it would still fail for some layers, in particular Lambda layers.

Overall, computing the right answer at the network level requires a lot of knowledge about the network, and is not feasible in the general case. We may be better off leaving this outside of Keras. Thoughts?

stale · 2017-09-05T06:28:35Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 30 days if no further activity occurs, but feel free to re-open a closed issue if needed.

cwzat · 2018-04-28T07:14:58Z

How to deal with input_shape = (None, None,None,3). Now it cannot deal with the typeNoneType?Thank you!

ShuteLee · 2019-05-01T05:45:38Z

So, is it provided now??

aizvorski added 2 commits April 9, 2017 18:19

Add count_flops() as analog to count_params()

4347cd2

Print flops per layer and total in model summary

666c407

fchollet reviewed Apr 10, 2017

View reviewed changes

aizvorski added 4 commits April 22, 2017 20:57

Rename count_flops to count_ops

d908c9f

Fixes for pep8

6906343

Replace switch with default based on kernel shape

2cc8495

Manual merge

813d057

fchollet reviewed Apr 25, 2017

View reviewed changes

stale bot added the stale label Sep 5, 2017

stale bot closed this Oct 5, 2017

Count number of floating point operations per layer #6203

Count number of floating point operations per layer #6203

aizvorski commented Apr 10, 2017 •

edited

fchollet Apr 10, 2017

Dref360 Apr 10, 2017

aizvorski Apr 23, 2017 •

edited

fchollet Apr 10, 2017

aizvorski Apr 23, 2017

fchollet Apr 10, 2017

farizrahman4u Apr 10, 2017

fchollet Apr 10, 2017

aizvorski Apr 10, 2017 •

edited

farizrahman4u Apr 10, 2017

fchollet Apr 10, 2017

aizvorski Apr 23, 2017

aizvorski Apr 23, 2017

fchollet left a comment

fchollet Apr 25, 2017

aizvorski Apr 25, 2017 •

edited

fchollet Apr 25, 2017

aizvorski Apr 25, 2017 •

edited

fchollet Apr 25, 2017

fchollet Apr 25, 2017

fchollet Apr 25, 2017

fchollet Apr 25, 2017

aizvorski Apr 25, 2017

fchollet Apr 25, 2017

fchollet Apr 25, 2017

aizvorski Apr 25, 2017 •

edited

fchollet Apr 25, 2017

fchollet Apr 25, 2017

aizvorski Apr 25, 2017 •

edited

fchollet Apr 25, 2017

alex-at-thimble May 25, 2017 •

edited

fchollet commented Apr 25, 2017

stale bot commented Sep 5, 2017

cwzat commented Apr 28, 2018

ShuteLee commented May 1, 2019

Count number of floating point operations per layer #6203

Count number of floating point operations per layer #6203

Conversation

aizvorski commented Apr 10, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aizvorski Apr 23, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aizvorski Apr 10, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fchollet left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aizvorski Apr 25, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aizvorski Apr 25, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aizvorski Apr 25, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aizvorski Apr 25, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alex-at-thimble May 25, 2017 • edited

Choose a reason for hiding this comment

fchollet commented Apr 25, 2017

stale bot commented Sep 5, 2017

cwzat commented Apr 28, 2018

ShuteLee commented May 1, 2019

aizvorski commented Apr 10, 2017 •

edited

aizvorski Apr 23, 2017 •

edited

aizvorski Apr 10, 2017 •

edited

aizvorski Apr 25, 2017 •

edited

aizvorski Apr 25, 2017 •

edited

aizvorski Apr 25, 2017 •

edited

aizvorski Apr 25, 2017 •

edited

alex-at-thimble May 25, 2017 •

edited