Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DisChnPrunedLearner with resnet18 on ImageNet can't converge in local mode #85

Closed
GoldenSpark opened this issue Nov 20, 2018 · 7 comments
Closed

DisChnPrunedLearner with resnet18 on ImageNet can't converge in local mode #85

GoldenSpark opened this issue Nov 20, 2018 · 7 comments
Assignees
@jiaxiang-wu
Labels
bug Something isn't working

Comments

@GoldenSpark
Copy link

No description provided.

@GoldenSpark
Copy link
Author

GoldenSpark commented Nov 20, 2018 •
edited
Loading

When I run the demo in tutorial with command ./scripts/run_local.sh nets/resnet_at_ilsvrc12_run.py
--learner dis-chn-pruned.
The log shows info with two lines over and over again just like this:
INFO:tensorflow : layer #18： prune_ratio=0.5879
INFO:tensorflow: adding channel #0 to the non-pruned set
INFO:tensorflow : layer #18： prune_ratio=0.5879
INFO:tensorflow: adding channel #0 to the non-pruned set
...
This situation has been lasting for several hours . I think it may be abnormal. Please help me check it . Thanks!

@jiaxiang-wu jiaxiang-wu self-assigned this Nov 21, 2018
@jiaxiang-wu jiaxiang-wu added the bug Something isn't working label Nov 21, 2018
@jiaxiang-wu
Copy link
Contributor

@GoldenSpark
Could you please post the full log, especially the part before "INFO:tensorflow : layer #...： prune_ratio = ..."?

@GoldenSpark
Copy link
Author

GoldenSpark commented Nov 21, 2018 •
edited
Loading

@jiaxiang-wu
goldenspark@node7743:~/pycharm_project/pocketflow$ ./scripts/run_local.sh nets/resnet_at_ilsvrc12_run.py --learner dis-chn-pruned
Python script: nets/resnet_at_ilsvrc12_run.py
#of GPUs: 1
extra arguments: --learner dis-chn-pruned --model_http_url https://api.ai.tencent.com/pocketflow --data_dir_local /data/goldenspark/TFRecord/
'nets/resnet_at_ilsvrc12_run.py' -> 'main.py'
multi-GPU training disabled
/home/users/goldenspark/anaconda3/lib/python3.6/site-packages/h5py/init.py:36: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type.
from ._conv import register_converters as _register_converters
[WARNING] TF-Plus & Horovod cannot be imported; multi-GPU training is unsupported
INFO:tensorflow:FLAGS:
INFO:tensorflow:data_disk: local
INFO:tensorflow:data_hdfs_host: None
INFO:tensorflow:data_dir_local: /data/goldenspark/TFRecord/
INFO:tensorflow:data_dir_hdfs: None
INFO:tensorflow:cycle_length: 4
INFO:tensorflow:nb_threads: 8
INFO:tensorflow:buffer_size: 1024
INFO:tensorflow:prefetch_size: 8
INFO:tensorflow:nb_classes: 1001
INFO:tensorflow:nb_smpls_train: 1281167
INFO:tensorflow:nb_smpls_val: 10000
INFO:tensorflow:nb_smpls_eval: 50000
INFO:tensorflow:batch_size: 64
INFO:tensorflow:batch_size_eval: 100
INFO:tensorflow:resnet_size: 18
INFO:tensorflow:lrn_rate_init: 0.1
INFO:tensorflow:batch_size_norm: 256.0
INFO:tensorflow:momentum: 0.9
INFO:tensorflow:loss_w_dcy: 0.0001
INFO:tensorflow:model_http_url: https://api.ai.tencent.com/pocketflow
INFO:tensorflow:summ_step: 100
INFO:tensorflow:save_step: 10000
INFO:tensorflow:save_path: ./models/model.ckpt
INFO:tensorflow:save_path_eval: ./models_eval/model.ckpt
INFO:tensorflow:enbl_dst: False
INFO:tensorflow:enbl_warm_start: False
INFO:tensorflow:loss_w_dst: 4.0
INFO:tensorflow:tempr_dst: 4.0
INFO:tensorflow:save_path_dst: ./models_dst/model.ckpt
INFO:tensorflow:nb_epochs_rat: 1.0
INFO:tensorflow:ddpg_actor_depth: 2
INFO:tensorflow:ddpg_actor_width: 64
INFO:tensorflow:ddpg_critic_depth: 2
INFO:tensorflow:ddpg_critic_width: 64
INFO:tensorflow:ddpg_noise_type: param
INFO:tensorflow:ddpg_noise_prtl: tdecy
INFO:tensorflow:ddpg_noise_std_init: 1.0
INFO:tensorflow:ddpg_noise_dst_finl: 0.01
INFO:tensorflow:ddpg_noise_adpt_rat: 1.03
INFO:tensorflow:ddpg_noise_std_finl: 1e-05
INFO:tensorflow:ddpg_rms_eps: 0.0001
INFO:tensorflow:ddpg_tau: 0.01
INFO:tensorflow:ddpg_gamma: 0.9
INFO:tensorflow:ddpg_lrn_rate: 0.001
INFO:tensorflow:ddpg_loss_w_dcy: 0.0
INFO:tensorflow:ddpg_record_step: 1
INFO:tensorflow:ddpg_batch_size: 64
INFO:tensorflow:ddpg_enbl_bsln_func: True
INFO:tensorflow:ddpg_bsln_decy_rate: 0.95
INFO:tensorflow:ws_save_path: ./models_ws/model.ckpt
INFO:tensorflow:ws_prune_ratio: 0.75
INFO:tensorflow:ws_prune_ratio_prtl: optimal
INFO:tensorflow:ws_nb_rlouts: 200
INFO:tensorflow:ws_nb_rlouts_min: 50
INFO:tensorflow:ws_reward_type: single-obj
INFO:tensorflow:ws_lrn_rate_rg: 0.03
INFO:tensorflow:ws_nb_iters_rg: 20
INFO:tensorflow:ws_lrn_rate_ft: 0.0003
INFO:tensorflow:ws_nb_iters_ft: 400
INFO:tensorflow:ws_nb_iters_feval: 25
INFO:tensorflow:ws_prune_ratio_exp: 3.0
INFO:tensorflow:ws_iter_ratio_beg: 0.1
INFO:tensorflow:ws_iter_ratio_end: 0.5
INFO:tensorflow:ws_mask_update_step: 500.0
INFO:tensorflow:cp_lasso: True
INFO:tensorflow:cp_quadruple: False
INFO:tensorflow:cp_reward_policy: accuracy
INFO:tensorflow:cp_nb_points_per_layer: 10
INFO:tensorflow:cp_nb_batches: 60
INFO:tensorflow:cp_prune_option: auto
INFO:tensorflow:cp_prune_list_file: ratio.list
INFO:tensorflow:cp_best_path: ./models/best_model.ckpt
INFO:tensorflow:cp_original_path: ./models/original_model.ckpt
INFO:tensorflow:cp_preserve_ratio: 0.5
INFO:tensorflow:cp_uniform_preserve_ratio: 0.6
INFO:tensorflow:cp_noise_tolerance: 0.15
INFO:tensorflow:cp_lrn_rate_ft: 0.0001
INFO:tensorflow:cp_nb_iters_ft_ratio: 0.2
INFO:tensorflow:cp_finetune: False
INFO:tensorflow:cp_retrain: False
INFO:tensorflow:cp_list_group: 1000
INFO:tensorflow:cp_nb_rlouts: 200
INFO:tensorflow:cp_nb_rlouts_min: 50
INFO:tensorflow:dcp_save_path: ./models_dcp/model.ckpt
INFO:tensorflow:dcp_save_path_eval: ./models_dcp_eval/model.ckpt
INFO:tensorflow:dcp_prune_ratio: 0.5
INFO:tensorflow:dcp_nb_stages: 3
INFO:tensorflow:dcp_lrn_rate_adam: 0.001
INFO:tensorflow:dcp_nb_iters_block: 10000
INFO:tensorflow:dcp_nb_iters_layer: 500
INFO:tensorflow:uql_equivalent_bits: 4
INFO:tensorflow:uql_nb_rlouts: 200
INFO:tensorflow:uql_w_bit_min: 2
INFO:tensorflow:uql_w_bit_max: 8
INFO:tensorflow:uql_tune_layerwise_steps: 100
INFO:tensorflow:uql_tune_global_steps: 2000
INFO:tensorflow:uql_tune_save_path: ./rl_tune_models/model.ckpt
INFO:tensorflow:uql_tune_disp_steps: 300
INFO:tensorflow:uql_enbl_random_layers: True
INFO:tensorflow:uql_enbl_rl_agent: False
INFO:tensorflow:uql_enbl_rl_global_tune: True
INFO:tensorflow:uql_enbl_rl_layerwise_tune: False
INFO:tensorflow:uql_weight_bits: 4
INFO:tensorflow:uql_activation_bits: 32
INFO:tensorflow:uql_use_buckets: False
INFO:tensorflow:uql_bucket_size: 256
INFO:tensorflow:uql_quant_epochs: 60
INFO:tensorflow:uql_save_quant_model_path: ./uql_quant_models/uql_quant_model.ckpt
INFO:tensorflow:uql_quantize_all_layers: False
INFO:tensorflow:uql_bucket_type: channel
INFO:tensorflow:uqtf_save_path: ./models_uqtf/model.ckpt
INFO:tensorflow:uqtf_save_path_eval: ./models_uqtf_eval/model.ckpt
INFO:tensorflow:uqtf_weight_bits: 8
INFO:tensorflow:uqtf_activation_bits: 8
INFO:tensorflow:uqtf_quant_delay: 0
INFO:tensorflow:uqtf_freeze_bn_delay: None
INFO:tensorflow:uqtf_lrn_rate_dcy: 0.01
INFO:tensorflow:nuql_equivalent_bits: 4
INFO:tensorflow:nuql_nb_rlouts: 200
INFO:tensorflow:nuql_w_bit_min: 2
INFO:tensorflow:nuql_w_bit_max: 8
INFO:tensorflow:nuql_tune_layerwise_steps: 100
INFO:tensorflow:nuql_tune_global_steps: 2101
INFO:tensorflow:nuql_tune_save_path: ./rl_tune_models/model.ckpt
INFO:tensorflow:nuql_tune_disp_steps: 300
INFO:tensorflow:nuql_enbl_random_layers: True
INFO:tensorflow:nuql_enbl_rl_agent: False
INFO:tensorflow:nuql_enbl_rl_global_tune: True
INFO:tensorflow:nuql_enbl_rl_layerwise_tune: False
INFO:tensorflow:nuql_init_style: quantile
INFO:tensorflow:nuql_opt_mode: weights
INFO:tensorflow:nuql_weight_bits: 4
INFO:tensorflow:nuql_activation_bits: 32
INFO:tensorflow:nuql_use_buckets: False
INFO:tensorflow:nuql_bucket_size: 256
INFO:tensorflow:nuql_quant_epochs: 60
INFO:tensorflow:nuql_save_quant_model_path: ./nuql_quant_models/model.ckpt
INFO:tensorflow:nuql_quantize_all_layers: False
INFO:tensorflow:nuql_bucket_type: split
INFO:tensorflow:log_dir: ./logs
INFO:tensorflow:enbl_multi_gpu: False
INFO:tensorflow:learner: dis-chn-pruned
INFO:tensorflow:exec_mode: train
INFO:tensorflow:debug: False
INFO:tensorflow:h: False
INFO:tensorflow:help: False
INFO:tensorflow:helpfull: False
INFO:tensorflow:helpshort: False
download_model: ./models
INFO:tensorflow:model files: checkpoint, model.ckpt-250227.meta, model.ckpt-250227.data-00000-of-00001, model.ckpt-250227.index
2018-11-19 14:43:11.243564: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 AVX512F FMA
2018-11-19 14:43:18.632748: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1405] Found device 0 with properties:
name: Tesla V100-SXM2-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:8a:00.0
totalMemory: 31.75GiB freeMemory: 31.34GiB
2018-11-19 14:43:18.632794: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1484] Adding visible gpu devices: 0
2018-11-19 14:43:18.936554: I tensorflow/core/common_runtime/gpu/gpu_device.cc:965] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-11-19 14:43:18.936605: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] 0
2018-11-19 14:43:18.936610: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 0: N
2018-11-19 14:43:18.936989: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30404 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-32GB, pci bus id: 0000:8a:00.0, compute c
apability: 7.0)
INFO:tensorflow:layer-to-block mapping: [0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 3, 3, 3]
2018-11-19 14:43:27.476371: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1484] Adding visible gpu devices: 0
2018-11-19 14:43:27.476424: I tensorflow/core/common_runtime/gpu/gpu_device.cc:965] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-11-19 14:43:27.476429: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] 0
2018-11-19 14:43:27.476434: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 0: N
2018-11-19 14:43:27.476619: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30404 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-32GB, pci bus id: 0000:8a:00.0, compute c
apability: 7.0)
INFO:tensorflow:Restoring parameters from ./models/model.ckpt-250227
INFO:tensorflow:layer #1: mask's shape is [ 1 1 64 64]
INFO:tensorflow:layer #1: prune_ratio = 0.0000
INFO:tensorflow:adding channel #42 to the non-pruned set
INFO:tensorflow:layer #1: prune_ratio = 0.9844
INFO:tensorflow:adding channel #60 to the non-pruned set
...
...
...
INFO:tensorflow:adding channel #149 to the non-pruned set
INFO:tensorflow:layer #17: prune_ratio = 0.5156
INFO:tensorflow:adding channel #8 to the non-pruned set
INFO:tensorflow:layer #17: prune_ratio = 0.5117
INFO:tensorflow:adding channel #156 to the non-pruned set
INFO:tensorflow:layer #17: prune_ratio = 0.5078
INFO:tensorflow:adding channel #223 to the non-pruned set
INFO:tensorflow:layer #17: prune_ratio = 0.5039
INFO:tensorflow:adding channel #16 to the non-pruned set
INFO:tensorflow:layer #17: prune_ratio = 0.5000
INFO:tensorflow:block #3: lr = 2.5000e-02 | loss = 7.4393e+00 | pr_trn = 1.7463e-01 | pr_msk = 1.8278e-01 | accuracy = 3.1250e-02 | acc_top1 = 0.0000e+00 | acc_top5 = 3.1250e-02
INFO:tensorflow:layer #18: mask's shape is [ 3 3 512 512]
INFO:tensorflow:layer #18: prune_ratio = 0.0000
INFO:tensorflow:adding channel #451 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.9980
INFO:tensorflow:adding channel #452 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.9961
INFO:tensorflow:adding channel #262 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.9941
INFO:tensorflow:adding channel #396 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.9922
INFO:tensorflow:adding channel #367 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.9902
INFO:tensorflow:adding channel #444 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.9883
INFO:tensorflow:adding channel #285 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.9863
INFO:tensorflow:adding channel #332 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.9844
INFO:tensorflow:adding channel #57 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.9824
INFO:tensorflow:adding channel #260 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.9805
INFO:tensorflow:adding channel #272 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.9785
INFO:tensorflow:adding channel #384 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.9766
INFO:tensorflow:adding channel #92 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.9746
INFO:tensorflow:adding channel #346 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.9727
INFO:tensorflow:adding channel #42 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.9707
INFO:tensorflow:adding channel #66 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.9688
INFO:tensorflow:adding channel #159 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.9668
INFO:tensorflow:adding channel #314 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.9648
INFO:tensorflow:adding channel #343 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.9629
INFO:tensorflow:adding channel #265 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.9609
INFO:tensorflow:adding channel #378 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.9590
INFO:tensorflow:adding channel #351 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.9570
INFO:tensorflow:adding channel #340 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.9551
INFO:tensorflow:adding channel #379 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.9531
INFO:tensorflow:adding channel #425 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.9512
INFO:tensorflow:adding channel #186 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.9492
INFO:tensorflow:adding channel #203 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.9473
INFO:tensorflow:adding channel #193 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.9453
INFO:tensorflow:adding channel #140 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.9434
INFO:tensorflow:adding channel #271 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.9414
INFO:tensorflow:adding channel #254 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.9395
INFO:tensorflow:adding channel #334 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.9375
INFO:tensorflow:adding channel #166 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.9355
INFO:tensorflow:adding channel #501 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.9336
INFO:tensorflow:adding channel #165 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.9316
INFO:tensorflow:adding channel #52 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.9297
INFO:tensorflow:adding channel #296 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.9277
INFO:tensorflow:adding channel #40 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.9258
INFO:tensorflow:adding channel #30 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.9238
INFO:tensorflow:adding channel #148 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.9219
INFO:tensorflow:adding channel #386 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.9199
INFO:tensorflow:adding channel #276 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.9180
INFO:tensorflow:adding channel #88 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.9160
INFO:tensorflow:adding channel #114 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.9141
INFO:tensorflow:adding channel #44 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.9121
INFO:tensorflow:adding channel #177 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.9102
INFO:tensorflow:adding channel #365 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.9082
INFO:tensorflow:adding channel #311 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.9062
INFO:tensorflow:adding channel #97 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.9043
INFO:tensorflow:adding channel #303 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.9023
INFO:tensorflow:adding channel #2 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.9004
INFO:tensorflow:adding channel #328 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.8984
INFO:tensorflow:adding channel #238 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.8965
INFO:tensorflow:adding channel #55 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.8945
INFO:tensorflow:adding channel #230 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.8926
INFO:tensorflow:adding channel #34 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.8906
INFO:tensorflow:adding channel #161 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.8887
INFO:tensorflow:adding channel #104 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.8867
INFO:tensorflow:adding channel #484 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.8848
INFO:tensorflow:adding channel #269 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.8828
INFO:tensorflow:adding channel #467 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.8809
INFO:tensorflow:adding channel #250 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.8789
INFO:tensorflow:adding channel #103 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.8770
INFO:tensorflow:adding channel #419 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.8750
INFO:tensorflow:adding channel #7 to the non-pruned set
...
INFO:tensorflow:layer #18: prune_ratio = 0.6758
INFO:tensorflow:adding channel #274 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.6738
INFO:tensorflow:adding channel #45 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.6719
INFO:tensorflow:adding channel #25 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.6699
INFO:tensorflow:adding channel #418 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.6680
INFO:tensorflow:adding channel #400 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.6660
INFO:tensorflow:adding channel #257 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.6641
INFO:tensorflow:adding channel #239 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.6621
INFO:tensorflow:adding channel #102 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.6602
INFO:tensorflow:adding channel #295 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.6582
INFO:tensorflow:adding channel #28 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.6562
INFO:tensorflow:adding channel #377 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.6543
INFO:tensorflow:adding channel #465 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.6523
INFO:tensorflow:adding channel #27 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.6504
INFO:tensorflow:adding channel #495 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.6484
INFO:tensorflow:adding channel #188 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.6465
INFO:tensorflow:adding channel #15 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.6445
INFO:tensorflow:adding channel #228 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.6426
INFO:tensorflow:adding channel #309 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.6406
INFO:tensorflow:adding channel #59 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.6387
INFO:tensorflow:adding channel #175 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.6367
INFO:tensorflow:adding channel #404 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.6348
INFO:tensorflow:adding channel #416 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.6328
INFO:tensorflow:adding channel #74 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.6309
INFO:tensorflow:adding channel #121 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.6289
INFO:tensorflow:adding channel #196 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.6270
INFO:tensorflow:adding channel #438 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.6250
INFO:tensorflow:adding channel #72 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.6230
INFO:tensorflow:adding channel #486 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.6211
INFO:tensorflow:adding channel #448 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.6191
INFO:tensorflow:adding channel #137 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.6172
INFO:tensorflow:adding channel #46 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.6152
INFO:tensorflow:adding channel #213 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.6133
INFO:tensorflow:adding channel #174 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.6113
INFO:tensorflow:adding channel #167 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.6094
INFO:tensorflow:adding channel #245 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.6074
INFO:tensorflow:adding channel #70 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.6055
INFO:tensorflow:adding channel #373 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.6035
INFO:tensorflow:adding channel #126 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.6016
INFO:tensorflow:adding channel #17 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.5996
INFO:tensorflow:adding channel #0 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.5977
INFO:tensorflow:adding channel #0 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.5977
INFO:tensorflow:adding channel #0 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.5977
INFO:tensorflow:adding channel #0 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.5977
INFO:tensorflow:adding channel #0 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.5977
INFO:tensorflow:adding channel #304 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.5957
INFO:tensorflow:adding channel #0 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.5957
INFO:tensorflow:adding channel #0 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.5957
INFO:tensorflow:adding channel #0 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.5957
INFO:tensorflow:adding channel #0 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.5957
INFO:tensorflow:adding channel #0 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.5957
INFO:tensorflow:adding channel #0 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.5957
INFO:tensorflow:adding channel #0 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.5957
INFO:tensorflow:adding channel #125 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.5938
INFO:tensorflow:adding channel #0 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.5938
INFO:tensorflow:adding channel #0 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.5938
INFO:tensorflow:adding channel #0 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.5938
INFO:tensorflow:adding channel #0 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.5938
INFO:tensorflow:adding channel #0 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.5938
INFO:tensorflow:adding channel #0 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.5938
INFO:tensorflow:adding channel #0 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.5938
INFO:tensorflow:adding channel #0 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.5938
INFO:tensorflow:adding channel #0 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.5938
INFO:tensorflow:adding channel #0 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.5938
INFO:tensorflow:adding channel #0 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.5938
INFO:tensorflow:adding channel #0 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.5938
INFO:tensorflow:adding channel #0 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.5938
INFO:tensorflow:adding channel #0 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.5938
INFO:tensorflow:adding channel #0 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.5938
INFO:tensorflow:adding channel #0 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.5938
INFO:tensorflow:adding channel #0 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.5938
INFO:tensorflow:adding channel #0 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.5938
INFO:tensorflow:adding channel #0 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.5938
INFO:tensorflow:adding channel #0 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.5938
INFO:tensorflow:adding channel #0 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.5938
INFO:tensorflow:adding channel #0 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.5938
INFO:tensorflow:adding channel #0 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.5938
INFO:tensorflow:adding channel #32 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.5918
INFO:tensorflow:adding channel #0 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.5918
INFO:tensorflow:adding channel #0 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.5918
INFO:tensorflow:adding channel #0 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.5918
INFO:tensorflow:adding channel #56 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.5898
INFO:tensorflow:adding channel #0 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.5898
INFO:tensorflow:adding channel #0 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.5898
INFO:tensorflow:adding channel #0 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.5898
INFO:tensorflow:adding channel #0 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.5898
INFO:tensorflow:adding channel #0 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.5898
INFO:tensorflow:adding channel #0 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.5898
INFO:tensorflow:adding channel #0 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.5898
INFO:tensorflow:adding channel #0 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.5898
INFO:tensorflow:adding channel #0 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.5898
INFO:tensorflow:adding channel #0 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.5898
INFO:tensorflow:adding channel #410 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.5879
INFO:tensorflow:adding channel #0 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.5879
INFO:tensorflow:adding channel #0 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.5879
INFO:tensorflow:adding channel #0 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.5879
INFO:tensorflow:adding channel #0 to the non-pruned set
INFO:tensorflow:layer #18: prune_ratio = 0.5879
INFO:tensorflow:adding channel #0 to the non-pruned set
...
Then it had been repeating the two lines above for more than ten hours until I stopped the program...

@jiaxiang-wu
Copy link
Contributor

@GoldenSpark Thanks for your information. We are investigating this issue and will push a fix ASAP.

@jiaxiang-wu
Copy link
Contributor

Similar problem reported in issue #143. Hotfix needed.

@jiaxiang-wu jiaxiang-wu mentioned this issue Dec 16, 2018
Merged
@jiaxiang-wu
Copy link
Contributor

@GoldenSpark
We have provided a hotfix in PR #147. Please verify whether the latest code has resolved your issues.

@jiaxiang-wu
Copy link
Contributor

Closing. Please reopen it if the bug still exists.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jiaxiang-wu jiaxiang-wu

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@GoldenSpark @jiaxiang-wu