TF 2.0 Feature: Flops calculation

[pzobel]

Please make sure that this is a feature request. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:feature_template

System information

• TensorFlow version (you are using): TF 2.0 RC2
• Are you willing to contribute it (Yes/No):

Describe the feature and the current behavior/state.

I am missing the opportunity to compute the number of floating point operations of a tf.keras Model in TF 2.0.
In TF 1.x tf.profiler was available see here but I can find anything equivalent for TF 2.0 yet.

Will this change the current api? How?

Who will benefit with this feature?

Everbody interested in the computational complexity of a TensorFlow model.

Any Other info.

[pzobel]

Any updates?

[qiuminxu]

We're working on adding cost model for tf 2.0. Since this is a pretty large feature, it will take more time to enable it.

[pzobel]

Have you made any progress yet? What do you expect when the feature will be available (in the nightly builds)?

[MikeOfZen]

I concur, this would be highly useful

[henglicad]

This feature would be very helpful for me too.

[MADHAVAN001]

I am also looking for this feature

[eduardo4jesus]

Please, I need it too.

[Li-markus]

Just make sure this way can still work in tf2.0:

import tensorflow as tf
import keras.backend as K

def get_flops(model):
run_meta = tf.RunMetadata()
opts = tf.profiler.ProfileOptionBuilder.float_operation()

# We use the Keras session graph in the call to the profiler.
flops = tf.profiler.profile(graph=K.get_session().graph,
                            run_meta=run_meta, cmd='op', options=opts)

return flops.total_float_ops  # Prints the "flops" of the model.

then it's already perfect 😄

[eduardo4jesus]

@Li-markus

I had to tweak it to avoid errors with TF 2.0, but I am still not able to get it working.

model = tf.keras.models.Sequential([
      InputLayer((32, 32, 1)),
      Conv2D(8, 5, padding='same', activation='relu'),
      MaxPool2D(2),
      Conv2D(16, 5, padding='same', activation='relu'),
      MaxPool2D(2),
      Flatten(),
      Dense(128, activation='relu'),
      Dense(64, activation='relu'),
      Dense(10, activation='softmax')
  ])

def get_flops(model):
  tf.compat.v1.disable_eager_execution()
  sess = tf.compat.v1.Session()

  run_meta = tf.compat.v1.RunMetadata()
  profiler = tf.compat.v1.profiler
  opts = profiler.ProfileOptionBuilder.float_operation()
  # We use the Keras session graph in the call to the profiler.
  flops = profiler.profile(graph=sess.graph, 
                           run_meta=run_meta, cmd='op', options=opts)

  return flops.total_float_ops  # Prints the "flops" of the model

The output was:

0

[eduardo4jesus]

I have a related question but I don't know if this kind of questions should be posted here on Git. Could someone give an opinion on it? Wondering if that could even be a bug.

[yxchng]

@qiuminxu any updates on this?

[driedler]

This works using TF 2.1

def get_flops(model_h5_path):
    session = tf.compat.v1.Session()
    graph = tf.compat.v1.get_default_graph()
        

    with graph.as_default():
        with session.as_default():
            model = tf.keras.models.load_model(model_h5_path)

            run_meta = tf.compat.v1.RunMetadata()
            opts = tf.compat.v1.profiler.ProfileOptionBuilder.float_operation()
        
            # We use the Keras session graph in the call to the profiler.
            flops = tf.compat.v1.profiler.profile(graph=graph,
                                                  run_meta=run_meta, cmd='op', options=opts)
        
            return flops.total_float_ops

[dansitu]

Note that the above doesn't seem to work when loading from a SavedModel; only from an h5.

[evgps]

I implemented small lib to calculate FLOPs/MACs: https://github.com/evgps/flopco-keras

[sendjasni]

@sendjasni I updated my comment; the inputs should be supplied in a tuple. The general idea is being, input_signature should match the structuring of the model's inputs

Btw, do you have an idea on how to compute the inference time?

[shaoxiang777]

I implemented small lib to calculate FLOPs/MACs: https://github.com/evgps/flopco-keras

This repo doesn't work anymore.

[bhack]

We need to "migrate" this ticket to https://github.com/keras-team/keras

[mohantym]

Hi @pzobel! I found this library keras-flops from an external contributor as work around for now. But could you post in Keras repo as a feature request. Thank you!

[google-ml-butler]

This issue has been automatically marked as stale because it has no recent activity. It will be closed if no further activity occurs. Thank you.

[ayulockin]

Is there any update on this issue?

[bhack]

We need to open this in the Keras repo

[ayulockin]

Do you mean the KerasCV repo? @bhack

[bhack]

Keras repo:

https://github.com/keras-team/keras

[google-ml-butler]

Closing as stale. Please reopen if you'd like to work on this further.

[mohantym]

Hi @pzobel @markub3327 @bhack ! I created a feature request in keras repo with ticket number #16699 on behalf of the users. Feel free to comment there. Thank you!

[albertmundu]

Refer to this https://github.com/wandb/wandb/blob/latest/wandb/integration/keras/keras.py#L1025-L1073 for the future visitors

import tensorflow as tf
import numpy as np

def get_flops(model, model_inputs) -> float:
        """
        Calculate FLOPS [GFLOPs] for a tf.keras.Model or tf.keras.Sequential model
        in inference mode. It uses tf.compat.v1.profiler under the hood.
        """
        # if not hasattr(model, "model"):
        #     raise wandb.Error("self.model must be set before using this method.")

        if not isinstance(
            model, (tf.keras.models.Sequential, tf.keras.models.Model)
        ):
            raise ValueError(
                "Calculating FLOPS is only supported for "
                "`tf.keras.Model` and `tf.keras.Sequential` instances."
            )

        from tensorflow.python.framework.convert_to_constants import (
            convert_variables_to_constants_v2_as_graph,
        )

        # Compute FLOPs for one sample
        batch_size = 1
        inputs = [
            tf.TensorSpec([batch_size] + inp.shape[1:], inp.dtype)
            for inp in model_inputs
        ]

        # convert tf.keras model into frozen graph to count FLOPs about operations used at inference
        real_model = tf.function(model).get_concrete_function(inputs)
        frozen_func, _ = convert_variables_to_constants_v2_as_graph(real_model)

        # Calculate FLOPs with tf.profiler
        run_meta = tf.compat.v1.RunMetadata()
        opts = (
            tf.compat.v1.profiler.ProfileOptionBuilder(
                tf.compat.v1.profiler.ProfileOptionBuilder().float_operation()
            )
            .with_empty_output()
            .build()
        )

        flops = tf.compat.v1.profiler.profile(
            graph=frozen_func.graph, run_meta=run_meta, cmd="scope", options=opts
        )

        tf.compat.v1.reset_default_graph()

        # convert to GFLOPs
        return (flops.total_float_ops / 1e9)/2
    
    
    
#Usage

if __name__ =="__main__":
    image_model = tf.keras.applications.EfficientNetB0(include_top=False, weights=None)
    
    x = tf.constant(np.random.randn(1,256,256,3))
    
    print(get_flops(image_model, [x]))

[sendjasni]

Refer to this https://github.com/wandb/wandb/blob/latest/wandb/integration/keras/keras.py#L1025-L1073 for the future visitors

import tensorflow as tf
import numpy as np

def get_flops(model, model_inputs) -> float:
        """
        Calculate FLOPS [GFLOPs] for a tf.keras.Model or tf.keras.Sequential model
        in inference mode. It uses tf.compat.v1.profiler under the hood.
        """
        # if not hasattr(model, "model"):
        #     raise wandb.Error("self.model must be set before using this method.")

        if not isinstance(
            model, (tf.keras.models.Sequential, tf.keras.models.Model)
        ):
            raise ValueError(
                "Calculating FLOPS is only supported for "
                "`tf.keras.Model` and `tf.keras.Sequential` instances."
            )

        from tensorflow.python.framework.convert_to_constants import (
            convert_variables_to_constants_v2_as_graph,
        )

        # Compute FLOPs for one sample
        batch_size = 1
        inputs = [
            tf.TensorSpec([batch_size] + inp.shape[1:], inp.dtype)
            for inp in model_inputs
        ]

        # convert tf.keras model into frozen graph to count FLOPs about operations used at inference
        real_model = tf.function(model).get_concrete_function(inputs)
        frozen_func, _ = convert_variables_to_constants_v2_as_graph(real_model)

        # Calculate FLOPs with tf.profiler
        run_meta = tf.compat.v1.RunMetadata()
        opts = (
            tf.compat.v1.profiler.ProfileOptionBuilder(
                tf.compat.v1.profiler.ProfileOptionBuilder().float_operation()
            )
            .with_empty_output()
            .build()
        )

        flops = tf.compat.v1.profiler.profile(
            graph=frozen_func.graph, run_meta=run_meta, cmd="scope", options=opts
        )

        tf.compat.v1.reset_default_graph()

        # convert to GFLOPs
        return (flops.total_float_ops / 1e9)/2
    
    
    
#Usage

if __name__ =="__main__":
    image_model = tf.keras.applications.EfficientNetB0(include_top=False, weights=None)
    
    x = tf.constant(np.random.randn(1,256,256,3))
    
    print(get_flops(image_model, [x]))

Thanks for sharing @albertmundu.
How much accurate is this ? I'have been using a different code (the solution provided here) and the results are way different.

[albertmundu]

@sendjasni I followed the link you provided; I used one of the working codes given by user https://stackoverflow.com/users/4619958/ch271828n to see how close the reading is between these two variants.

Using StackOverflow code

def flops():
    session = tf.compat.v1.Session()
    graph = tf.compat.v1.get_default_graph()

    with graph.as_default():
        with session.as_default():
            model = keras.applications.EfficientNetV2B0(weights=None, input_tensor=tf.compat.v1.placeholder('float32', shape=(1, 224, 224, 3)))

            run_meta = tf.compat.v1.RunMetadata()
            opts = tf.compat.v1.profiler.ProfileOptionBuilder.float_operation()

            # Optional: save printed results to file
            # flops_log_path = os.path.join(tempfile.gettempdir(), 'tf_flops_log.txt')
            # opts['output'] = 'file:outfile={}'.format(flops_log_path)

            # We use the Keras session graph in the call to the profiler.
            flops = tf.compat.v1.profiler.profile(graph=graph,
                                                  run_meta=run_meta, cmd='op', options=opts)

    tf.compat.v1.reset_default_graph()

    return flops.total_float_ops


flops()

And it prints the following

=========================Options=============================
-max_depth                  10000
-min_bytes                  0
-min_peak_bytes             0
-min_residual_bytes         0
-min_output_bytes           0
-min_micros                 0
-min_accelerator_micros     0
-min_cpu_micros             0
-min_params                 0
-min_float_ops              1
-min_occurrence             0
-step                       -1
-order_by                   float_ops
-account_type_regexes       .*
-start_name_regexes         .*
-trim_name_regexes          
-show_name_regexes          .*
-hide_name_regexes          
-account_displayed_op_only  true
-select                     float_ops
-output                     stdout:

==================Model Analysis Report======================
1454888026

Doc:
op: The nodes are operation kernel type, such as MatMul, Conv2D. Graph nodes belonging to the same type are aggregated together.
flops: Number of float operations. Note: Please read the implementation for the math behind it.

Profile:
node name | # float_ops
Conv2D                   1.41b float_ops (100.00%, 96.98%)
DepthwiseConv2dNative    22.61m float_ops (3.02%, 1.55%)
Mul                      17.02m float_ops (1.46%, 1.17%)
MatMul                   2.56m float_ops (0.29%, 0.18%)
Mean                     1.32m float_ops (0.12%, 0.09%)
Sub                      211.25k float_ops (0.03%, 0.01%)
RealDiv                  150.53k float_ops (0.01%, 0.01%)
BiasAdd                  14.52k float_ops (0.00%, 0.00%)
Softmax                  5.00k float_ops (0.00%, 0.00%)
Maximum                      3 float_ops (0.00%, 0.00%)

======================End of Report==========================

Using wandb code #32809 (comment)

x=tf.constant(np.random.randn(1,224,224,3))
model = tf.keras.applications.EfficientNetV2B0(weights=None)

get_flops(model, [x])

It prints

0.7238510575

This value is obtained using (flops.total_float_ops / 1e9)/2

If you apply the same for the above (1454888026/1e9)/2, you get 0.7274440130000001 which is almost the same up to 2 decimal points to 0.7238510575

[moonsh]

Hi, @srihari-humbarwadi

The code you shared is for getting MACs? or FLOPs? Seems like it's getting MACs because it's divided by 2.
#32809 (comment)

[srihari-humbarwadi]

Hi, @srihari-humbarwadi

The code you shared is for getting MACs? or FLOPs? Seems like it's getting MACs because it's divided by 2.

#32809 (comment)

It is MACs