🚀 Tenary Weight and DoReFa-Net in TensorFlow (TensorLayer) #440

XJTUWYD · 2018-03-22T03:16:02Z

I add two compress strategies tenary weight network and dorefa-net into tensorlayer, did two experiments to compare the accuracy of different compress strategies based on mnist and cifar-10.
the result of the experiment is below:

	BinaryNet	Tenary Weight	DoReFa-Net
MNIST	98.86%	99.27%	98.89%
CIFAR10	41.1%	80.6%	81.1%

related issue: #416

add four apis TenaryConv2d,TenaryDenseLayer,DorefaConv2d,DorefaDenselyLayer and build different tutorials for bnn,twn,dorefa based on mnist and cifar10 datasets

four apis

luomai · 2018-03-22T03:23:16Z

tensorlayer/layers/binary.py

 import tensorflow as tf
+from tensorflow.python.framework import ops
+
+bitW = 1


can you avoid using global variables to setup the parameters?

luomai · 2018-03-22T03:23:50Z

tensorlayer/layers/binary.py

+    with G.gradient_override_map({"Round": "Identity"}):
+        return tf.round(x * n) / n
+
+def fw(x, force_quantization=False):


can we use a more meaningful names? If this function is used only internally, we better to start the name with a underscore.

luomai · 2018-03-22T06:02:57Z

example/tutorial_tenarycifar10_tfrecord.py

+import os, time
+import tensorflow as tf
+import tensorlayer as tl
+from tensorlayer.layers import *


We prefer to avoiding "import *" in new committed files.

lgarithm · 2018-03-22T06:15:44Z

tensorlayer/layers/binary.py

@@ -1,11 +1,19 @@
 # -*- coding: utf-8 -*-
 from .core import *


don't use import * for new code.

fiexed some bugs of format

add bitw and bita for apis

luomai · 2018-03-22T09:53:14Z

@zsdonghao as we are having a large number of examples, shall we actually start organizing the examples into different folders?

zsdonghao · 2018-03-22T11:07:38Z

example/tutorial_binarycifar10_tfrecord.py

@@ -0,0 +1,324 @@
+#! /usr/bin/python
+# -*- coding: utf-8 -*-
+"""Reimplementation of the TensorFlow official CIFAR-10 CNN tutorials.


we can remove this comment.

zsdonghao · 2018-03-22T11:07:50Z

example/tutorial_binarycifar10_tfrecord.py

+data_to_tfrecord(images=X_test, labels=y_test, filename="test.cifar10")
+
+## Example to visualize data
+# img, label = read_and_decode("train.cifar10", None)


we can remove this comment.

zsdonghao · 2018-03-22T11:08:03Z

example/tutorial_dorefacifar10_tfrecord.py

@@ -0,0 +1,319 @@
+#! /usr/bin/python
+# -*- coding: utf-8 -*-
+"""Reimplementation of the TensorFlow official CIFAR-10 CNN tutorials.


we can remove this comment.

zsdonghao · 2018-03-22T11:08:15Z

example/tutorial_dorefacifar10_tfrecord.py

+data_to_tfrecord(images=X_train, labels=y_train, filename="train.cifar10")
+data_to_tfrecord(images=X_test, labels=y_test, filename="test.cifar10")
+
+## Example to visualize data


we can remove this comment.

zsdonghao · 2018-03-22T11:08:43Z

example/tutorial_tenarycifar10_tfrecord.py

@@ -0,0 +1,318 @@
+#! /usr/bin/python
+# -*- coding: utf-8 -*-
+"""Reimplementation of the TensorFlow official CIFAR-10 CNN tutorials.


we can remove this comment.

zsdonghao · 2018-03-22T11:08:49Z

example/tutorial_tenarycifar10_tfrecord.py

+data_to_tfrecord(images=X_test, labels=y_test, filename="test.cifar10")
+
+## Example to visualize data
+# img, label = read_and_decode("train.cifar10", None)


we can remove this comment.

zsdonghao · 2018-03-22T11:10:45Z

tensorlayer/layers/binary.py

+
+    Examples
+    ---------
+    >>> net = tl.layers.InputLayer(x, name='input')


the example here should be modified.

zsdonghao · 2018-03-22T11:11:15Z

tensorlayer/layers/binary.py

+
+    Examples
+    ---------
+    >>> net = tl.layers.InputLayer(x, name='input')


the example here should be modified

zsdonghao · 2018-03-22T11:15:22Z

tensorlayer/layers/binary.py

+        self.inputs = prev_layer.outputs
+        if act is None:
+            act = tf.identity
+        logging.info("BinaryConv2d %s: n_filter:%d filter_size:%s strides:%s pad:%s act:%s" % (self.name, n_filter, str(filter_size), str(strides), padding,


change BinaryConv2d to DorefaConv2d

zsdonghao · 2018-03-22T11:15:43Z

tensorlayer/layers/binary.py

+
+        n_in = int(self.inputs.get_shape()[-1])
+        self.n_units = n_units
+        logging.info("BinaryDenseLayer  %s: %d %s" % (self.name, self.n_units, act.__name__))


change BinaryDenseLayer to DorefaDenseLayer

zsdonghao · 2018-03-22T11:16:49Z

tensorlayer/layers/binary.py

+        self.inputs = prev_layer.outputs
+        if act is None:
+            act = tf.identity
+        logging.info("BinaryConv2d %s: n_filter:%d filter_size:%s strides:%s pad:%s act:%s" % (self.name, n_filter, str(filter_size), str(strides), padding,


change BinaryConv2d to TenaryConv2d

zsdonghao · 2018-03-22T11:17:09Z

tensorlayer/layers/binary.py

+
+        n_in = int(self.inputs.get_shape()[-1])
+        self.n_units = n_units
+        logging.info("BinaryDenseLayer  %s: %d %s" % (self.name, self.n_units, act.__name__))


change BinaryDenseLayer to TenaryDenseLayer

zsdonghao · 2018-03-22T11:17:52Z

tensorlayer/layers/binary.py

        return tf.sign(x)


+def _quantize_dorefa(x, k):


need comment

zsdonghao · 2018-03-22T11:17:58Z

tensorlayer/layers/binary.py

+        return tf.round(x * n) / n
+
+
+def _quantize_weight(x, bitW, force_quantization=False):


need comment

zsdonghao · 2018-03-22T11:18:03Z

tensorlayer/layers/binary.py

+    return 2 * _quantize_dorefa(x, bitW) - 1
+
+
+def _quantize_active(x, bitA):


need comment

zsdonghao · 2018-03-22T11:18:09Z

tensorlayer/layers/binary.py

+    return _quantize_dorefa(x, bitA)
+
+
+def _cabs(x):


need comment

zsdonghao · 2018-03-22T11:18:13Z

tensorlayer/layers/binary.py

+    return tf.minimum(1.0, tf.abs(x), name='cabs')
+
+
+def _compute_threshold(x):


need comment

zsdonghao · 2018-03-22T11:18:25Z

tensorlayer/layers/binary.py

+    return threshold
+
+
+def _compute_alpha(x):


need comment

zsdonghao · 2018-03-22T11:18:29Z

tensorlayer/layers/binary.py

+    return alpha
+
+
+def _tenary_opration(x):


need comment

lgarithm · 2018-03-22T13:25:05Z

example/tutorial_dorefanet_mnist_cnn.py

+    # ref: https://github.com/itayhubara/BinaryNet.tf/blob/master/models/BNN_cifar10.py
+    with tf.variable_scope("binarynet", reuse=reuse):
+        net = tl.layers.InputLayer(x, name='input')
+        net = tl.layers.DorefaConv2d(net, 32, (5, 5), (1, 1), padding='SAME', b_init=None, name='bcnn1')


How does DorefaConv2d know that 32 is for n_filter but not for bitW or bitA?

it crashed with

$ python3 example/tutorial_dorefanet_mnist_cnn.py [TL] Load or Download MNIST > data/mnist [TL] data/mnist/train-images-idx3-ubyte.gz [TL] data/mnist/t10k-images-idx3-ubyte.gz 2018-03-22 21:30:07.666529: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 [TL] InputLayer binarynet/input: (128, 28, 28, 1) Traceback (most recent call last): File "example/tutorial_dorefanet_mnist_cnn.py", line 48, in <module> net_train = model(x, is_train=True, reuse=False) File "example/tutorial_dorefanet_mnist_cnn.py", line 25, in model net = tl.layers.DorefaConv2d(net, 32, (5, 5), (1, 1), padding='SAME', b_init=None, name='bcnn1') File "/home/vagrant/.local/lib/python3.6/site-packages/tensorlayer/layers/binary.py", line 687, in __init__ act.__name__)) TypeError: %d format: a number is required, not tuple

I forgot it and I have fixed the bug

lgarithm · 2018-03-22T13:36:09Z

Since our functions have long argument lists, and many of them have default values, I suggest that we always use keyword style argument passing (except for small helper functions that 1,2 or 3 arguments).

zsdonghao · 2018-03-22T14:50:05Z

tensorlayer/layers/binary.py

 class DorefaConv2d(Layer):
-    """
-    The :class:`BinaryConv2d` class is a 2D binary CNN layer, which weights are either -1 or 1 while inferencing.
+    """The :class:`DorefaConv2d` class is a binary fully connected layer, which weights are 'bitW' bits and the input of the previous layer 


should be convolutional layer

fix some issue and delete some comment

XJTUWYD · 2018-03-22T16:25:15Z

bnn is a excellent work in the compression of neuron network but it can not get a satisfied accuracy on relative large datasets, in order to solve the problem, tenary weight networks and dorefa were put forward. I add 4 apis for tensorlayer, Tenary Denselayer, TenaryConv2d, DorefaDenselayer, and DorefaConv2d . I perform 6 experiment based on mnist and cifar10，the details are in thr titorials.
finally thank you very much for the help of HaoDong, LuoMai, and Igarithm.

zsdonghao · 2018-03-22T16:37:26Z

@XJTUWYD Hi, i suggest you to add this comment into this issue #416, more people can see it.

and include your previous comment:

I add two compress strategies tenary weight network and dorefa-net into tensorlayer, did two experiments to compare the accuracy of different compress strategies based on mnist and cifar-10.
the result of the experiment is below:

	BinaryNet	Tenary Weight	DoReFa-Net
MNIST	98.86%	99.27%	98.89%
CIFAR10	41.1%	80.6%	81.1%

* compress add four apis TenaryConv2d,TenaryDenseLayer,DorefaConv2d,DorefaDenselyLayer and build different tutorials for bnn,twn,dorefa based on mnist and cifar10 datasets * four apis four apis * fiexed some bugs of format fiexed some bugs of format * add bitw and bita for apis add bitw and bita for apis * Add files via upload * Add files via upload * Add files via upload * do some explain about twn and dorefa * fix some issue and delete some comment fix some issue and delete some comment * add some comment * use yapf

XJTUWYD added 2 commits March 22, 2018 10:59

compress

cc47ee3

add four apis TenaryConv2d,TenaryDenseLayer,DorefaConv2d,DorefaDenselyLayer and build different tutorials for bnn,twn,dorefa based on mnist and cifar10 datasets

four apis

c4571c7

four apis

XJTUWYD closed this Mar 22, 2018

luomai reviewed Mar 22, 2018

View reviewed changes

luomai reopened this Mar 22, 2018

Merge branch 'master' into master

61c85ba

luomai changed the title ~~add two compress strategies tenary weight and dorefa-net into tensorlayer~~ [wip] add two model compression models: tenary weight and dorefa-net. Mar 22, 2018

luomai reviewed Mar 22, 2018

View reviewed changes

luomai requested a review from zsdonghao March 22, 2018 06:03

lgarithm reviewed Mar 22, 2018

View reviewed changes

tensorlayer/layers/binary.py

@@ -1,11 +1,19 @@

# -*- coding: utf-8 -*-

from .core import *

Copy link

Member

lgarithm Mar 22, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't use import * for new code.

XJTUWYD added 2 commits March 22, 2018 17:39

fiexed some bugs of format

fcb2f01

fiexed some bugs of format

add bitw and bita for apis

6b9185e

add bitw and bita for apis

luomai changed the title ~~[wip] add two model compression models: tenary weight and dorefa-net.~~ add two model compression models: tenary weight and dorefa-net. Mar 22, 2018

luomai approved these changes Mar 22, 2018

View reviewed changes

tensorlayer deleted a comment Mar 22, 2018

XJTUWYD added 2 commits March 22, 2018 18:18

Add files via upload

2bfa5bc

Add files via upload

3ae52c3

tensorlayer deleted a comment Mar 22, 2018

tensorlayer deleted a comment from luomai Mar 22, 2018

tensorlayer deleted a comment Mar 22, 2018

zsdonghao reviewed Mar 22, 2018

View reviewed changes

tensorlayer/layers/binary.py

return tf.sign(x)

def _quantize_dorefa(x, k):

Copy link

Member

zsdonghao Mar 22, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

need comment

zsdonghao reviewed Mar 22, 2018

View reviewed changes

tensorlayer/layers/binary.py

return tf.round(x * n) / n

def _quantize_weight(x, bitW, force_quantization=False):

Copy link

Member

zsdonghao Mar 22, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

need comment

zsdonghao reviewed Mar 22, 2018

View reviewed changes

tensorlayer/layers/binary.py

return 2 * _quantize_dorefa(x, bitW) - 1

def _quantize_active(x, bitA):

Copy link

Member

zsdonghao Mar 22, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

need comment

zsdonghao reviewed Mar 22, 2018

View reviewed changes

tensorlayer/layers/binary.py

return _quantize_dorefa(x, bitA)

def _cabs(x):

Copy link

Member

zsdonghao Mar 22, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

need comment

zsdonghao reviewed Mar 22, 2018

View reviewed changes

tensorlayer/layers/binary.py

return tf.minimum(1.0, tf.abs(x), name='cabs')

def _compute_threshold(x):

Copy link

Member

zsdonghao Mar 22, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

need comment

zsdonghao reviewed Mar 22, 2018

View reviewed changes

tensorlayer/layers/binary.py

return threshold

def _compute_alpha(x):

Copy link

Member

zsdonghao Mar 22, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

need comment

zsdonghao reviewed Mar 22, 2018

View reviewed changes

tensorlayer/layers/binary.py

return alpha

def _tenary_opration(x):

Copy link

Member

zsdonghao Mar 22, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

need comment

lgarithm reviewed Mar 22, 2018

View reviewed changes

zsdonghao changed the title ~~add two model compression models: tenary weight and dorefa-net.~~ 🚀 add two model compression models: tenary weight and dorefa-net. Mar 22, 2018

zsdonghao changed the title ~~🚀 add two model compression models: tenary weight and dorefa-net.~~ 🚀 Tenary Weight and DoReFa-Net in TensorFlow (TensorLayer) Mar 22, 2018

XJTUWYD added 2 commits March 22, 2018 22:46

Add files via upload

105dde1

do some explain about twn and dorefa

df5a12f

zsdonghao reviewed Mar 22, 2018

View reviewed changes

XJTUWYD added 3 commits March 22, 2018 22:50

fix some issue and delete some comment

2bda19d

fix some issue and delete some comment

add some comment

4f98b7e

use yapf

3069f7b

zsdonghao merged commit 4aa66e3 into tensorlayer:master Mar 22, 2018

		return tf.round(x * n) / n


		def _quantize_weight(x, bitW, force_quantization=False):

		return 2 * _quantize_dorefa(x, bitW) - 1


		def _quantize_active(x, bitA):

		return tf.minimum(1.0, tf.abs(x), name='cabs')


		def _compute_threshold(x):

🚀 Tenary Weight and DoReFa-Net in TensorFlow (TensorLayer) #440

🚀 Tenary Weight and DoReFa-Net in TensorFlow (TensorLayer) #440

Uh oh!

Conversation

XJTUWYD commented Mar 22, 2018 • edited by zsdonghao Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

luomai commented Mar 22, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lgarithm commented Mar 22, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

XJTUWYD commented Mar 22, 2018

Uh oh!

zsdonghao commented Mar 22, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

XJTUWYD commented Mar 22, 2018 •

edited by zsdonghao

Loading

zsdonghao commented Mar 22, 2018 •

edited

Loading