Keras inference time optimizer #264

ZFTurbo · 2018-05-30T08:21:33Z

Reduce neural net structure (Conv + BN -> Conv)
Also works:
DepthwiseConv2D + BN -> DepthwiseConv2D
SeparableConv2D + BN -> SeparableConv2D

This code takes on input trained Keras model and optimize layer structure and weights in such a way that model became much faster (~30%), but works identically to initial model. It can be extremely useful in case you need to process large amount of images with trained model. Reduce operation was tested on all Keras models zoo. See comparison table and full description by link:
https://github.com/ZFTurbo/Keras-inference-time-optimizer

Reduce neural net structure (Conv + BN -> Conv) Also works: DepthwiseConv2D + BN -> DepthwiseConv2D SeparableConv2D + BN -> SeparableConv2D This code takes on input trained Keras model and optimize layer structure and weights in such a way that model became much faster (~30%), but works identically to initial model. It can be extremely useful in case you need to process large amount of images with trained model. Reduce operation was tested on all Keras models zoo. See comparison table and full description by link: https://github.com/ZFTurbo/Keras-inference-time-optimizer

Keras inference time optimizer

ahundt · 2018-06-20T03:57:10Z

The style here doesn't seem to match that of other keras code. Could you fix that?

titu1994 · 2018-06-20T05:13:01Z

Are there no tests other than the benchmark test from your repo that can be performed ? A benchmark test isn't what would be feasible inside Travis, especially with such large models.

Once the style fixes are finished, this can be merged if it includes at least some form of test and all tests pass.

ZFTurbo · 2018-06-20T10:21:00Z

@titu1994 I added "mobilenet_small" benchmark. It run several seconds on my machine. I think it can be used as test for kito.py code.

Should I include tests directly in kito.py file?

@ahundt: Regarding style. I saw other files in "utils" folder and didn't see any specific style pattern. Could you please clarify what exactly should I change? Do you mean comments in functions or something else?

ahundt · 2018-06-20T19:40:44Z

tests should go in the tests folder, take a look how those are written.

titu1994 · 2018-06-22T00:49:44Z

@ZFTurbo The test is short on Tensorflow, but I doubt its fast on Theano. In any case, it's better than no test at all.

As @ahundt mentioned, tests should go into the tests directory, specifically tests/keras_contrib/utils/*_test.py where you can replace * with some name (i suggest kito to be same with this file). For the format, please see other tests or tests from core Keras.

ZFTurbo · 2018-06-26T12:28:43Z

I added test.

ahundt · 2018-09-30T21:45:13Z

@titu1994 @ZFTurbo this looks pretty good to me now, think it should be merged?

ZFTurbo · 2018-10-02T19:27:17Z

@ahundt I'm totally fine with merge. )

ahundt

I looked through this again and had a couple additional questions

ahundt · 2018-10-04T16:55:06Z

keras_contrib/utils/kito.py

+
+    # DeepLabV3+ non-standard layer
+    if layer.__class__.__name__ == 'BilinearUpsampling':
+        from neural_nets.deeplab_v3_plus_model import BilinearUpsampling


Just noticed this, there is now a bilinear upsampling layer in keras itself.
keras-team/keras#10994

It has other name in Keras. This code made specially for DeepLab V3+. It still uses it's own implementation.

/cc @bonlime

We can't have specialized code for things not included in keras-contrib. You'll need to come up with a way to parameterize this, such as by passing functions, or a dictionary.

ahundt · 2018-10-04T16:56:12Z

tests/keras_contrib/utils/kito_test.py

+        print('Reduced model number layers: {}'.format(len(model_reduced.layers)))
+        print('Compare models...')
+        if model_name in ['nasnetlarge', 'deeplab_v3plus_mobile', 'deeplab_v3plus_xception']:
+            max_error = compare_two_models_results(model, model_reduced, test_number=10000, max_batch=128)


how long does this test take to run?

Several seconds

ahundt · 2018-10-04T16:57:06Z

keras_contrib/utils/kito.py

+    return outbound_layers
+
+
+def get_copy_of_layer(layer, verbose=False):


how do users do this for their own non-standard layers?

They need to change the code ( Like here:

# DeepLabV3+ non-standard layer if layer.__class__.__name__ == 'BilinearUpsampling': from neural_nets.deeplab_v3_plus_model import BilinearUpsampling

I'm not sure how to make it universal, since parameters are different.

They could pass a dict mapping from names to functions/object, so if it is in the dictionary you call that, otherwise you do some default behavior, and else raise an error saying what they need to do to fix the problem.

You can also have a look at keras itself registers functions. If you look at the python file for loss, at the very bottom there is some registration code.

ZFTurbo · 2018-10-12T11:39:46Z

Added more tests
Removed non-standard layers. Probably need to add custom_objects parameter. Will do it later.
Added support for layers of type "Model".

gabrieldemarmiesse · 2018-12-16T19:11:37Z

Thank you @ZFTurbo for your work.

I've looked at the result presented in your page and the number are impressive.
Since you're using tensorflow, I might add that there is XLA (maybe you already know about it) which does a similar job.

If it's not too much trouble, can I ask you to run your benchmarks with XLA enabled?

Also as a side note. I started using keras for the first time with your scripts on kaggle a few years ago. Thanks for sharing! (remember the fish competition?) :)

ZFTurbo · 2018-12-17T12:53:52Z

Thank you @ZFTurbo for your work.

I've looked at the result presented in your page and the number are impressive.
Since you're using tensorflow, I might add that there is XLA (maybe you already know about it) which does a similar job.

If it's not too much trouble, can I ask you to run your benchmarks with XLA enabled?

I heard about XLA but not tried it yet. Probably XLA will make KITO useless. I'll try it and post result here.

Also as a side note. I started using keras for the first time with your scripts on kaggle a few years ago. Thanks for sharing! (remember the fish competition?) :)

Good to know it )) That's bad, but looks like PyTorch becoming mainstream now, instead of Keras.

gabrieldemarmiesse · 2018-12-17T19:29:14Z

I heard about XLA but not tried it yet. Probably XLA will make KITO useless. I'll try it and post result here.

Every feature has a maintenance cost. The more complicated the feature, the higher the cost. We have to choose wisely depending on the manpower that we have (currently not much). I can see how this tool can be useful, but since it works on keras models with a known structure, it can be the job of the backend to optimise it (and it should, even if this is not currently the case with many backends).
So if XLA does a similar job, I'm not sure we should merge this. The maintenance cost would be high compare to, let's say, a layer or a callback.

That's bad, but looks like PyTorch becoming mainstream now, instead of Keras.

That's not bad, diversity is cool, I'm pretty sure we wouldn't have tf eager without pytorch. Furthermore, maybe there will be a pytorch backend for keras poping up somewhere on the internet. Pytorch and keras are not necessarily competitors.

ZFTurbo added 2 commits May 30, 2018 11:17

Merge pull request #1 from ZFTurbo/ZFTurbo-patch-1

dd69212

Keras inference time optimizer

titu1994 closed this Jun 20, 2018

titu1994 reopened this Jun 20, 2018

Added support for Keras 2.2.0

013e928

Small fixes to pass tests

c77b546

user-miner1 added 3 commits June 26, 2018 14:49

Kito test bench

ca56c75

Kito test bench

23e66e5

Kito test bench

6cc94b4

IDMIPPM added 4 commits July 24, 2018 14:51

Fixed bug with rescale bias formula (very rare case).

8ce70e1

Merge remote-tracking branch 'origin/master'

73fb420

Small fix for Keras 2.2.2

371b0c0

Added support for multi input and multi output models

2513ec8

ahundt suggested changes Oct 4, 2018

View reviewed changes

bhack mentioned this pull request Oct 7, 2018

Bilinear Upsample Shathe/Semantic-Segmentation-Tensorflow-Eager#4

Closed

Update for latest version. Removed non-standard layers

7eca212

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Keras inference time optimizer #264

Keras inference time optimizer #264

ZFTurbo commented May 30, 2018

ahundt commented Jun 20, 2018

titu1994 commented Jun 20, 2018

ZFTurbo commented Jun 20, 2018

ahundt commented Jun 20, 2018

titu1994 commented Jun 22, 2018 •

edited

ZFTurbo commented Jun 26, 2018

ahundt commented Sep 30, 2018

ZFTurbo commented Oct 2, 2018

ahundt left a comment

ahundt Oct 4, 2018

ZFTurbo Oct 4, 2018

bhack Oct 7, 2018

ahundt Oct 9, 2018

ahundt Oct 4, 2018

ZFTurbo Oct 4, 2018

ahundt Oct 4, 2018

ZFTurbo Oct 4, 2018

ahundt Oct 12, 2018 •

edited

ahundt Oct 12, 2018

ZFTurbo commented Oct 12, 2018

gabrieldemarmiesse commented Dec 16, 2018

ZFTurbo commented Dec 17, 2018

gabrieldemarmiesse commented Dec 17, 2018 •

edited

		return outbound_layers


		def get_copy_of_layer(layer, verbose=False):

Keras inference time optimizer #264

Are you sure you want to change the base?

Keras inference time optimizer #264

Conversation

ZFTurbo commented May 30, 2018

ahundt commented Jun 20, 2018

titu1994 commented Jun 20, 2018

ZFTurbo commented Jun 20, 2018

ahundt commented Jun 20, 2018

titu1994 commented Jun 22, 2018 • edited

ZFTurbo commented Jun 26, 2018

ahundt commented Sep 30, 2018

ZFTurbo commented Oct 2, 2018

ahundt left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ahundt Oct 12, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ZFTurbo commented Oct 12, 2018

gabrieldemarmiesse commented Dec 16, 2018

ZFTurbo commented Dec 17, 2018

gabrieldemarmiesse commented Dec 17, 2018 • edited

titu1994 commented Jun 22, 2018 •

edited

ahundt Oct 12, 2018 •

edited

gabrieldemarmiesse commented Dec 17, 2018 •

edited