[slim] Refine framework of slim and add filter pruning strategy #16226

wanghaoshuang · 2019-03-15T09:35:26Z

Add the framework of paddle slim
Add filter pruning strategy

1. Add framework of paddle slim 2. Add filter pruning strategy test=develop

… release_slim_pruning test=develop

test=develop

… release_slim_pruning test=develop

test=develop

…ng/Paddle into release_slim_pruning

test=develop

… release_slim_pruning test=develop

…ng/Paddle into release_slim_pruning test=develop

test=develop

wzzju · 2019-03-20T09:11:46Z

python/paddle/fluid/contrib/slim/graph/graph_wrapper.py

+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+


不是剪切和蒸馏不使用graph了吗？还需要graph_wrapper吗？

为了以后方便切换到IrGraph

wzzju · 2019-03-20T09:12:59Z

python/paddle/fluid/contrib/slim/prune/pruner.py

@@ -13,9 +13,10 @@
 # limitations under the License.



Quantization Strategy is not included?

准备另起PR提交Quantization Strategy相关的内容

wzzju · 2019-03-20T09:22:52Z

python/paddle/fluid/contrib/slim/core/config.py

@@ -17,7 +17,7 @@
 import yaml
 from collections import OrderedDict
 from ..prune import *
-from .compress_pass import *
+from ..quantization import *


I don't see where is the quantization?

quantization是主干已经有的一个module: https://github.com/PaddlePaddle/Paddle/tree/develop/python/paddle/fluid/contrib/slim/quantization

SunGaofeng · 2019-03-20T06:57:02Z

python/paddle/fluid/contrib/slim/core/compress_pass.py

+        cached_id(int): The id of dataset sampled. Evaluations with same cached_id use the same sampled dataset. default: 0.
+    """
+    np.random.seed(cached_id)
+    cache_path = cache_path + "/" + str(cached_id)


建议使用os.path.join(dir0, dir1,..., file)，以自动补充"/"

Fixed. Thx.

SunGaofeng · 2019-03-20T07:15:30Z

python/paddle/fluid/contrib/slim/core/compress_pass.py

+        Load the context from file.
+        """
+        with open(file_name) as context_file:
+            data = pickle.load(context_file)


pickle在python3下面测试过吗，上次我们提交video的代码，在python3下面使用pickle.load报错，修改成下面的形式了
if python_ver < (3, 0):
data_loaded = pickle.load(open(pickle_path, 'rb'))
else:
data_loaded = pickle.load(open(pickle_path, 'rb'), encoding='bytes')

Fixed. Thx.

Fix log and comments. test=develop

qingqing01

Not finished yet

qingqing01 · 2019-03-21T00:34:03Z

python/paddle/fluid/contrib/slim/core/compress_pass.py

+                    batch += 1
+                    yield data
+
+    return s_reader


Why needs these readers?

为了提升data feeder的速度，但并不是所有情况都能提速。这个逻辑放在压缩工具内部确实不太合适。

feeder加速，用户可以外面做，建议去掉~

qingqing01 · 2019-03-21T00:35:34Z

python/paddle/fluid/contrib/slim/core/compress_pass.py

+            teacher_graphs: The teacher graphs used in distillation strategies.
+            train_optimizer: The optimizer used to append backward ops and
+                             optimization ops into train_graph.
+            distiller_optimizer: The optimizer used by distillation strategies.


In distill, how to set train_optimizer and distiller_optimizer?

在distiller框架中，没有统一的用户接口，不同的任务都有不同的入口，比如：分类任务使用接口, language model 使用接口

其中，分类任务接口提供了蒸馏功能，language model 没有提供蒸馏功能。
在分类任务接口compress_classifier.py中，optimizer是固定的SGD，而且蒸馏策略只能用该optimizer. 如果用户需要切换optimizer，只能修改compress_classifier.py文件，该文件夹杂了各种对压缩策略的调度逻辑，对用户非常不友好。

对于分类任务，蒸馏没有用独立的optimizer, 它做蒸馏的步骤如下：

调用knowledge distillation policy的forward进行前向计算

使用所有压缩策略共用的optimizer进行反向计算

综上，distiller的用户接口设计、optimizer的使用方式和蒸馏策略的调用方式都不具参考价值。

distiller给出的大部分demo都是基于compress_classifier.py的, 用户能调整的也就是SGD optimizer的一些参数。

抱歉，我理解错问题了。这里我会补充下注释，详细说明下两个optimizer各自的用途。多谢。

抱歉，我上面的意思是蒸馏压缩算法时，这两个optimizer如何设置需要解释清楚~

qingqing01 · 2019-03-21T00:36:15Z

python/paddle/fluid/contrib/slim/core/compress_pass.py

+        logger.info('Latest evaluations: {}'.format(results))
+        return abs(results[1] - results[0]) / results[0] < delta
+
+    def run_eval_graph(self, sampled_rate=None, cached_id=0):


Why the run_eval_graph in the Context ?

为了方便各种策略随时评估当前graph的性能，该方法可放在：

每个策略类中：重复实现，策略实现者比较麻烦。

CompressPass类中：每个策略访问不到

GraphWrapper中?

单独实现一个ExecutorHelper?

qingqing01 · 2019-03-21T00:39:58Z

python/paddle/fluid/contrib/slim/core/compress_pass.py

+        if 'init_model' in factory.compress_pass:
+            self.init_model = factory.compress_pass['init_model']
+
+    def _init_model(self, context):


As discuss before, Pass is only used to transform graph, no include RUNING(train/eval)

CompressPass其实不是pass，我把名字改成Compressor?

I agree to change name. Make the consistent meaning of Pass in Python and C++. How do you think? @panyx0718 @wzzju

Renamed CompressPass to Compressor.

panyx0718 · 2019-03-21T01:29:23Z

python/paddle/fluid/contrib/slim/prune/pruner.py

+    Pruner used to pruning parameters by groups.
+    """
+
+    def __init__(self, pruning_axis, criterions):


you might want to add some comments for this and many others. not easy to understand the codes.

Thx. Fixed.

panyx0718 · 2019-03-21T03:47:57Z

python/paddle/fluid/contrib/slim/tests/filter_pruning/compress.yaml

+        class: 'SensitivePruneStrategy'
+        pruner: 'pruner_1'
+        start_epoch: 1
+        delta_rate: 0.2


you might want to document this metrics

Thx. Fixed.

panyx0718 · 2019-03-21T03:48:25Z

python/paddle/fluid/contrib/slim/tests/filter_pruning/compress.yaml

+        num_steps: 1
+        eval_rate: 0.5
+        pruned_params: 'conv6_sep_weights'
+        sensitivities_file: 'mobilenet_acc_top1_sensitive.data'


Thx. Fixed.

panyx0718 · 2019-03-21T03:49:55Z

python/paddle/fluid/contrib/slim/tests/configs/filter_pruning.yaml

+        num_steps: 1
+        eval_rate: 0.5
+        pruned_params: '.*_sep_weights'
+        sensitivities_file: 'mobilenet_acc_top1_sensitive.data'


Thx. Fixed.

qingqing01 · 2019-03-21T05:17:52Z

python/paddle/fluid/contrib/slim/core/compress_pass.py

+        if 'init_model' in factory.compress_pass:
+            self.init_model = factory.compress_pass['init_model']
+
+    def _init_model(self, context):


I agree to change name. Make the consistent meaning of Pass in Python and C++. How do you think? @panyx0718 @wzzju

qingqing01 · 2019-03-21T05:25:07Z

python/paddle/fluid/contrib/slim/core/compress_pass.py

+        strategies = self.strategies
+        if self.checkpoint_path:
+            if not os.path.exists(self.checkpoint_path):
+                os.makedirs(self.checkpoint_path)


这里是load_checkpoint，路径还可能不存在，要makerdirs?

Thx. Fixed.

qingqing01 · 2019-03-21T05:26:44Z

python/paddle/fluid/contrib/slim/core/compress_pass.py


-        for epoch in range(self.epoch):
+        reader = feed_reader(


建议去掉feed_reader，用户外面传进来的reader，可能已经是多线程/进程了

Thx. Fixed.

qingqing01 · 2019-03-21T05:39:11Z

python/paddle/fluid/contrib/slim/core/compress_pass.py

+        """
+        Runing evaluation.
+        """
+        results, names = context.run_eval_graph()


这个class是各种压缩算法公用的吧？如果是Context里的run_eval_graph可以直接放这里吗？

Compressor(CompressPass)对象包含了strategy, 从层级关系上看，strategy不能直接访问Compressor对象的。
如果strategy.on_compression_epoch(compressor):

两个对象相互依赖

strategy能访问到的内容太多，比如可以直接调用compressor.run()形成死循环

context相当于把compressor中允许且需要strategy访问的信息和能力封装起来，供strategy使用。同时，compressor自己也可以使用，比如这一行。

qingqing01 · 2019-03-21T05:44:47Z

python/paddle/fluid/contrib/slim/graph/executor.py

-                graph.program.global_block().var(name) for name in fetches
-            ]
-        results = self.exe.run(graph.program,
+    def run(self, graph, scope, data=None, feed=None, fetches=None):


add comments, data是什么类型？

Thx. Fixed.

qingqing01 · 2019-03-21T05:44:49Z

python/paddle/fluid/contrib/slim/graph/executor.py

-                graph.program.global_block().var(name) for name in fetches
-            ]
-        results = self.exe.run(graph.program,
+    def run(self, graph, scope, data=None, feed=None, fetches=None):


add comments, data, feed, fetches是什么类型？

qingqing01 · 2019-03-21T07:54:41Z

python/paddle/fluid/contrib/slim/prune/prune_strategy.py

+        Args:
+            context(slim.core.Context): The context storing all information used to evaluate the current model.
+            sampled_rate(float): The sampled rate used to sample partial data for evaluation. None means using all data in eval_reader. default: None.
+            cached_id(int): The id of dataset sampled. Evaluations with same cached_id use the same sampled dataset. default: 0.


似乎不太明白cache_id的解释

cache_id是随机抽取的数据子集的唯一标识。

2. Fix cache reader 3. Rename CompressPass to Compressor 4. Add comments for distiller optimizer 5. Remove unused pruner currently 6. Add some comments. 7. Change API.spec test=develop

test=develop

chengduoZH

LG api.spec

First pr of paddle slim.

087d228

1. Add framework of paddle slim 2. Add filter pruning strategy test=develop

wanghaoshuang requested review from qingqing01, wzzju and panyx0718 March 15, 2019 09:39

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

cbab069

… release_slim_pruning test=develop

wanghaoshuang force-pushed the release_slim_pruning branch from dab6dea to cbab069 Compare March 15, 2019 09:41

Rename unitest to tests.

fa2c64b

test=develop

wanghaoshuang force-pushed the release_slim_pruning branch from 6918572 to fa2c64b Compare March 15, 2019 09:52

wanghaoshuang added 6 commits March 18, 2019 02:06

Add prettytable into requirements.

d424944

test=develop

Change in_nodes and out_nodes to odered dict.

5f7e070

test=develop

Remove distillation.

03accd7

test=develop

Fix API.spec

1859c86

test=develop

Fix unitest.

7b074e4

test=develop

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

3679af9

… release_slim_pruning test=develop

wanghaoshuang force-pushed the release_slim_pruning branch from d04fabb to 3679af9 Compare March 18, 2019 11:29

wanghaoshuang added 3 commits March 18, 2019 21:18

Fix unitest in windows.

7798b3a

test=develop

Merge branch 'release_slim_pruning' of https://github.com/wanghaoshua…

6425ffb

…ng/Paddle into release_slim_pruning

Fix unitest in windows.

cac2867

test=develop

wanghaoshuang force-pushed the release_slim_pruning branch from b707d6f to cac2867 Compare March 18, 2019 16:16

wanghaoshuang added 4 commits March 19, 2019 01:40

Fix unitest.

c2b1a52

test=develop

Hide some functions.

f4de736

test=develop

Fix python import in python3.5

0221245

test=develop

Fix compress pass.

49ed631

test=develop

wanghaoshuang mentioned this pull request Mar 20, 2019

test_dist_ctr failed in ci task #16312

Closed

wanghaoshuang added 3 commits March 20, 2019 13:19

Merge branch 'develop' of https://github.com/PaddlePaddle/paddle into…

8926673

… release_slim_pruning test=develop

Merge branch 'release_slim_pruning' of https://github.com/wanghaoshua…

9d955d3

…ng/Paddle into release_slim_pruning test=develop

Fix unitest of test_dist_ctr.

4edb841

test=develop

wzzju reviewed Mar 20, 2019

View reviewed changes

SunGaofeng reviewed Mar 20, 2019

View reviewed changes

wanghaoshuang added 3 commits March 20, 2019 22:25

Enhence flops.

0af2e2d

use os.path.join

8d02dc3

Fix pickle for python3

6113fff

Fix log and comments. test=develop

wanghaoshuang force-pushed the release_slim_pruning branch from 6d075e2 to 6113fff Compare March 20, 2019 15:34

qingqing01 reviewed Mar 21, 2019

View reviewed changes

panyx0718 previously approved these changes Mar 21, 2019

View reviewed changes

panyx0718 reviewed Mar 21, 2019

View reviewed changes

qingqing01 reviewed Mar 21, 2019

View reviewed changes

1. Remove feed_reader in compress pass

7349475

2. Fix cache reader 3. Rename CompressPass to Compressor 4. Add comments for distiller optimizer 5. Remove unused pruner currently 6. Add some comments. 7. Change API.spec test=develop

wanghaoshuang dismissed panyx0718’s stale review via 7349475 March 21, 2019 17:26

Fix pruning in python3.

6128255

test=develop

qingqing01 previously approved these changes Mar 22, 2019

View reviewed changes

wanghaoshuang dismissed qingqing01’s stale review via 2522980 March 22, 2019 08:42

Fix unitest in python3.

de96f94

test=develop

wanghaoshuang force-pushed the release_slim_pruning branch 2 times, most recently from 23bd353 to b7f8b4f Compare March 22, 2019 14:59

Fix format in python3.

071accc

test=develop

wanghaoshuang force-pushed the release_slim_pruning branch from b7f8b4f to 071accc Compare March 22, 2019 23:52

wanghaoshuang requested a review from shanyi15 March 23, 2019 02:12

shanyi15 approved these changes Mar 23, 2019

View reviewed changes

chengduoZH approved these changes Mar 23, 2019

View reviewed changes

qingqing01 approved these changes Mar 23, 2019

View reviewed changes

XiaoguangHu01 approved these changes Mar 23, 2019

View reviewed changes

wanghaoshuang merged commit 2e5831f into PaddlePaddle:develop Mar 23, 2019

wanghaoshuang deleted the release_slim_pruning branch May 20, 2022 03:55

[slim] Refine framework of slim and add filter pruning strategy #16226

[slim] Refine framework of slim and add filter pruning strategy #16226

Conversation

wanghaoshuang commented Mar 15, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

qingqing01 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wanghaoshuang Mar 21, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

qingqing01 Mar 21, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

qingqing01 Mar 21, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chengduoZH left a comment

Choose a reason for hiding this comment

wanghaoshuang Mar 21, 2019 •

edited

qingqing01 Mar 21, 2019 •

edited

qingqing01 Mar 21, 2019 •

edited