Improve pruning module #2354

NHZlX · 2017-06-02T06:39:00Z

resolve #2284
add dynamicPruningHook in ParameterUpdaterHook.cpp
improve python v2 api

… improve_pruning

NHZlX · 2017-06-02T06:40:47Z

paddle/parameter/ParameterUpdaterHook.cpp

@@ -131,6 +134,73 @@ class StaticPruningHook : public IParameterUpdaterHook {
  std::vector<bool> mask_;
 };

+class DynamicPruningHook : public IParameterUpdaterHook {


add the dynamicpruning hook, which calculate mask according to sparsity_ratio

NHZlX · 2017-06-02T06:42:52Z

paddle/parameter/ParameterUpdaterHook.cpp


 /**
 * ParameterUpdaterHook actually factory method.
 */
 static IParameterUpdaterHook* createImpl(
    const ParameterUpdaterHookConfig& config) {
  auto& type = config.type();
-  if (type == "pruning") {
-    if (config.has_purning_mask_filename()) {
+  if (type == "pruning_static") {


the specific 'pruning' type is dynamic one, and 'pruning_static' is static which read mask from the file.

Xreki

新增的DynamicPruningHook，和原来StaticPruningHook的区别是：

DynamicPruningHook是根据config里面的sparsity_ratio生成一个mask
StaticPruningHook是从文件读入mask

二者对于Parameter的操作是一样的，只是初始化mask的方式不一样。我不认为这种实现是一种Dynamic Pruning。它还是Static Pruning，可以替换原来的Staic Pruning实现。

Dynamic Pruning是类似这篇文章Dynamic Network Surgery for Efficient DNNs中的方法，不需要指定sparsity_ratio，可以自动地设置和调整sparsity_ratio，直至达到最大的压缩率。

Xreki · 2017-06-02T06:58:38Z

paddle/parameter/ParameterUpdaterHook.cpp

@@ -25,6 +25,9 @@ limitations under the License. */
 #include "paddle/utils/Flags.h"
 #include "paddle/utils/Util.h"

+using std::vector;
+using std::pair;


不要使用using语句

NHZlX · 2017-06-02T07:29:12Z

@Xreki 嗯，那么就只保留sparsity_ratio 的这个

Xreki · 2017-06-02T07:32:39Z

@NHZlX

那么就只保留sparsity_ratio 的这个

只保留这种就可以了，原来的StaticPruningHook操作太不方便

… improve_pruning

Xreki · 2017-06-05T07:29:37Z

paddle/parameter/ParameterUpdaterHook.cpp

- * define which link/weight between neural is disabled.
+ * Static means user specific a sparsity_ratio map before training started. The
+ * network will
+ * hold the sparsity_ratio maximum numbers of parameters, and cut off the rest.


line 33，多于的map。line 35，不通顺。

Xreki · 2017-06-05T07:31:03Z

paddle/parameter/ParameterUpdaterHook.cpp

  SameThreadChecker updateThreadChecker_;
  std::atomic<size_t> initCount_;
  VectorPtr maskVec_;
-  std::vector<bool> mask_;
+  VectorPtr maskTemp_;


看看能不能不把maskTemp做成成员变量。

Xreki · 2017-06-05T07:31:47Z

paddle/parameter/ParameterUpdaterHook.cpp

  }
+
+  LOG(FATAL) << "Unknown Hook type:  " << type;


确认一下，如果没有配置hook时，是不会调用该函数的吧？

这块是为了保证如果指定的hook type不在我们提供的当中的话，会报错。 python v2 /python/paddle/trainer/config_parser.py ParameterHook 处有关于hook type的检测，但是未来也有其他不同过python方式来调用这个的吧。

Xreki · 2017-06-05T07:35:19Z

proto/ParameterConfig.proto

@@ -26,7 +26,8 @@ enum ParameterInitStrategy {

 message ParameterUpdaterHookConfig {
  required string type = 1;
-  optional string purning_mask_filename = 2;
+  //hook type such as  'pruning'
+  optional double sparsity_ratio = 3;


既然purning_mask_filename类型删掉了，sparsity_ratio应该设置成2。另外，sparsity_ratio若是可选的，则应该设置默认值。

Xreki · 2017-06-05T07:35:49Z

proto/ParameterConfig.proto

@@ -26,7 +26,8 @@ enum ParameterInitStrategy {

 message ParameterUpdaterHookConfig {
  required string type = 1;
-  optional string purning_mask_filename = 2;
+  //hook type such as  'pruning'


注释一般放在前面，并且//后要有一个空格。

Xreki · 2017-06-05T07:38:28Z

python/paddle/trainer_config_helpers/attrs.py

+    Hook Attribute object. The hook is an auxiliary operation that occurs 
+    during network propagation. Such as pruning operation, It will cut off 
+    redundant parameters in the network before training. More detail can see 
+    here paddle/parameter/ParameterUpdaterHook.cpp


这里可改成引用论文。另外，最后一句检查一下语法。

Xreki · 2017-06-05T07:39:58Z

python/paddle/trainer_config_helpers/attrs.py

+
+    :param sparsity_ratio: Must be specified if hook type is 'pruning',
+	                       the network will hold the sparsity_ratio maximum parameters, and cut off the rest. 
+    :type sparsity_ratio: float number between 0 and 1


:type xxx:后面只跟类型，范围限制应该放到:param xxx:后面。

… improve_pruning

hedaoyuan · 2017-06-08T09:09:08Z

paddle/parameter/ParameterUpdaterHook.cpp

+
+    for (size_t i = 0; i < para->getSize(); i++)
+      param.push_back(std::make_pair(fabs(vecCpu->getData()[i]), i));
+    std::sort(param.begin(), param.end(), sortPairAscend);


可以用std::partial_sort

@hedaoyuan 嗯，这里用这个排序会更好一些，我会修改一下

hedaoyuan · 2017-06-08T09:13:57Z

paddle/parameter/ParameterUpdaterHook.cpp

-        dataPtr[i++] = m ? 1.0 : 0.0;
-      }
-    }
-
    // Currently just use a mask vector for hack.
    // @TODO(yuyang18): Implemented the mask operation in vector.
    if (para->useGpu()) {


86-91这段代码可以挪到generateMask里面去吧。

@hedaoyuan 这个是已经删掉的代码

哦，我指的是修改后的文件的86-91行。if (para->useGpu())这段逻辑。

可以放到那里边去

hedaoyuan · 2017-06-08T09:16:46Z

paddle/parameter/ParameterUpdaterHook.cpp

-    if (config.has_purning_mask_filename()) {
-      return new StaticPruningHook(config.purning_mask_filename());
-    }
+    if (config.has_sparsity_ratio())


config.has_sparsity_ratio()的判断放到StaticPruningHook的构造里面去吧。另外，这个我看python里面是有default值的，这里为什么是报错，而不是加default值？

StaticPruningHook只是赋值没有判断。我指的是There must be sparsity_ratio parameter for pruning Hook.这个逻辑本身是属于StaticPruningHook的。

python端会修改一下：如果没有指定sparsit_ratio, 自动使用默认值， c++这块的判断将删掉

… improve_pruning

hedaoyuan · 2017-06-15T04:56:37Z

paddle/parameter/ParameterUpdaterHook.cpp

@@ -60,17 +61,28 @@ class StaticPruningHook : public IParameterUpdaterHook {
    maskTemp_ = Vector::create(para->getSize(), false);
    maskTemp_->zeroMem();
    real* dataPtr = maskTemp_->getData();
+    size_t sparsityNum = para->getSize() * (1 - sparsityRatio_);


sparsityRatio_这个指的是非零元还是零元的ratio？我看左边的第72行，原先的定义看起来是非零元的ratio，这里为什么换了？
···
for (size_t i = 0; i < para->getSize() * sparsityRatio_; i++)
···

是0元的ratio，mask初始值全为0，这里要做的是将非0元素的mask 设置为1，所以为（1 - sparsityRatio_）, 这里的sparsityNum确实起的不是太好，会修改一下

hedaoyuan

本定装一下pre-commit
https://github.com/PaddlePaddle/Paddle/blob/develop/doc/howto/dev/contribute_to_paddle_cn.md#使用-pre-commit-钩子

hedaoyuan · 2017-06-15T12:36:12Z

proto/ParameterConfig.proto

  required string type = 1;
-  optional string purning_mask_filename = 2;
+  optional double sparsity_ratio = 2 [default = 0.8];


这里要注释清楚sparsity_ratio指的是非零还是零元的。比如default=0.8指的是80%的零元？实际上我看到sparsity_ratio的第一反应以为是非零元的占比。

@hedaoyuan 嗯，好

… improve_pruning

Xreki · 2017-06-16T08:37:50Z

paddle/parameter/ParameterUpdaterHook.cpp

- * Static means user load a mask map before training started. This map will
- * define which link/weight between neural is disabled.
+ * Static means user specific a sparsity_ratio before training start, and the
+ * network will prune the parameters based on the sparsity_ratio. More deatils


typo: specific -> specify, start -> started
More deatils can see -> More details can be found

Xreki · 2017-06-16T08:40:49Z

paddle/parameter/ParameterUpdaterHook.cpp

-    SetDevice device(para->getDeviceId());
+  void generateMask(Parameter* para) {
+    VectorPtr vec = para->getBuf(PARAMETER_VALUE);
+    maskTemp_ = Vector::create(para->getSize(), false);


maskTemp_改成局部变量

Xreki · 2017-06-16T08:47:07Z

paddle/parameter/ParameterUpdaterHook.cpp

+
+    std::partial_sort(
+        param.begin(), param.begin() + nonZeroNum, param.end(), sortPairAscend);
+    for (size_t i = 0; i < nonZeroNum; i++) dataPtr[param[i].second] = 1.0;


一些变量的名字看能不能改下，比如dataPtr，我得往上找到63行才知道这个变量指的是什么内容，最好用一些能顾名思义的名字。包括vec、vecCpu等

Xreki · 2017-06-16T08:58:01Z

python/paddle/trainer_config_helpers/attrs.py

+    during network propagation.
+    NOTE: IT IS A HIGH LEVEL USER INTERFACE.
+
+    :param  type: Hook type, eg: 'pruning'


这里的注释会用来生成api文档，所以最好写详细点。比如所有支持的type类型，以及参考的论文工作。

Xreki · 2017-06-16T08:58:39Z

python/paddle/trainer_config_helpers/attrs.py

+    """
+    Hook Attribute object. The hook is an auxiliary operation that occurs 
+    during network propagation.
+    NOTE: IT IS A HIGH LEVEL USER INTERFACE.


最好能说明下Hook是用来干什么的，作用的对象。另外，我认为这个NOTE就没有必要了。

Xreki · 2017-06-16T09:03:08Z

python/paddle/trainer_config_helpers/attrs.py

+            assert is_compatible_with(
+                self.sparsity_ratio,
+                float), 'sparisity_ratio must be float type'
+            assert self.sparsity_ratio <= 1 and self.sparsity_ratio >= 0, 'sparisity must be a flaot between [0, 1] '


'sparisity must be a flaot between [0, 1] ' -> 'sparisity_ratio must be a float between [0, 1] '，错误提示与变量名保持一致，另外还有typo。

Xreki · 2017-06-16T09:04:59Z

git commit的信息最好能反映实际修改的内容， Update ParameterUpdaterHook.cpp看不出改了什么。

… improve_pruning

NHZlX added 2 commits June 2, 2017 14:28

imporve pruning module

5413af8

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

ca55a24

… improve_pruning

NHZlX commented Jun 2, 2017

View reviewed changes

NHZlX requested review from reyoung, hedaoyuan and Xreki June 2, 2017 06:45

Xreki requested changes Jun 2, 2017

View reviewed changes

Xreki added this to In Progress in Embedded and Mobile Deployment Jun 2, 2017

NHZlX added 2 commits June 2, 2017 16:31

modify the pruning from reading mask to specify sparsity_ratio

18435f2

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

b6eaed0

… improve_pruning

NHZlX mentioned this pull request Jun 2, 2017

Improve the pruning module of paddle #2284

Closed

Xreki reviewed Jun 5, 2017

View reviewed changes

NHZlX added 3 commits June 5, 2017 17:42

modify the doc of the interface

092828f

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

1a1c589

… improve_pruning

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

23a1480

… improve_pruning

NHZlX force-pushed the improve_pruning branch from 88d840b to 23a1480 Compare June 5, 2017 10:20

NHZlX added 2 commits June 6, 2017 19:01

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

1566848

… improve_pruning

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

a1e1472

… improve_pruning

hedaoyuan requested changes Jun 8, 2017

View reviewed changes

NHZlX added 3 commits June 12, 2017 19:51

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

97a2fde

… improve_pruning

tiny modify

997cef2

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

23b1a27

… improve_pruning

NHZlX force-pushed the improve_pruning branch from c845dcc to 23b1a27 Compare June 14, 2017 09:50

NHZlX added 3 commits June 14, 2017 18:48

Create ParameterUpdaterHook.cpp

98e4bb7

Update ParameterUpdaterHook.cpp

4fbec82

Create ParameterUpdaterHook.cpp

5405dc0

hedaoyuan reviewed Jun 15, 2017

View reviewed changes

merge from remote

6248e56

hedaoyuan reviewed Jun 15, 2017

View reviewed changes

NHZlX added 2 commits June 16, 2017 14:29

explain the sparsity ratio

fc9e3e4

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

a3ada68

… improve_pruning

hedaoyuan approved these changes Jun 16, 2017

View reviewed changes

Update ParameterUpdaterHook.cpp

885275e

Xreki reviewed Jun 16, 2017

View reviewed changes

NHZlX removed the request for review from reyoung June 17, 2017 08:03

NHZlX added 8 commits June 19, 2017 22:47

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

1a82e7d

… improve_pruning

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

15bf6e0

… improve_pruning

modify the annotations of HookAttribute, Variable declaration

1eab8cc

modify the format

aaf11fa

modify format

a266292

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

43771ad

… improve_pruning

modify some topo

fdde4ef

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

561c456

… improve_pruning

Xreki approved these changes Jun 23, 2017

View reviewed changes

modity topo

1d6b859

NHZlX merged commit 8b86624 into PaddlePaddle:develop Jun 23, 2017

NHZlX deleted the improve_pruning branch June 23, 2017 05:57

Xreki moved this from In Progress to Model Compression in Embedded and Mobile Deployment Jul 12, 2017

Improve pruning module #2354

Improve pruning module #2354

Conversation

NHZlX commented Jun 2, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Xreki left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NHZlX commented Jun 2, 2017

Xreki commented Jun 2, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hedaoyuan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Xreki commented Jun 16, 2017

Xreki left a comment •

edited

Loading