Automated Deep Compression status #64
Comments
Hi, Currently the status of ADC (now AMC: https://arxiv.org/abs/1802.03494) is unchanged. I'll update when we have something that can be shared. Cheers |
Thanks for the response :) Cheers, |
Hi Amjad, I'm happy to hear that you're using Distiller and find it useful! Cheers, |
Hey Neta, Cheers, |
Sorry Amjad, I still haven't completed the move to the public v.0.11.0 Coach. I'm currently pushing code that's still integrated with an older, private branch, of Coach. |
Hi Amjad, Cheers, |
@nzmora Does that mean I have to install cuda 9.0 if I want to try AMC? |
Hi @HKLee2040 Cheers |
Work on AMC currently takes place in branch 'amc'. Your help is more than welcome. |
After switching to using Clipped PPO I'm getting very encouraging results. See: https://github.com/NervanaSystems/distiller/wiki/AutoML-for-Model-Compression-(AMC):-Trials-and-Tribulations |
@huxianer the schedule file for training Plain20 is here. It took me about 33 minutes on 4-GPUs. However, since you've asked :-), I've also uploaded the image here: Cheers, |
@nzmora Thank you very much! I have another question to ask you,I found the top1 performance is really unchanged when I don |
Hi @huxianer, I think you are asking how to train using AMC if we don't have a pre-trained model of the network we are compressing. I hope this helps, |
Why the smooth_top1 and smooth_reward are overlapping in my "Performance Data" diagram? And args.amc_target_density = None, so I add |
Hi @HKLee2040
I will need to fix the code for the case of one GPU.
I don't know which protocol you are using ("mac-constrained" or "accuracy-guaranteed"), but both are highly correlated to the Top1 accuracy: So it makes sense that you will see an overlap when the graphs are smoothed (I smoothed using a simple moving average) because the signal noise is made less noticeable in both the reward and accuracy signals. You can see an example here. Having said that, I think that you ask a good question. I think that this is a clue as to why the reward defined in the AMC paper, for accuracy-guaranteed-compression, is not so good. The solutions converge on maximum density for all layers (you can see this in the green bars here) - probably because the agent tries to maximize the Top1 accuracy - and not enough weight is given to the MACs (FLOPs) in the reward (5). Thanks, |
Hi @HKLee2040,
Thanks for the persistency. The shift you see is an illusion (and causes confusion, I guess) and is caused by the fact that the reward and Top1 accuracy use different axes (top1 on the right; reward on the left). The reward's range is [0..1] and the accuracy is [0..100] and because their values are correlated exactly (reward = 1/100 as you wrote above) they should align. However, when we draw the MAC values, also on the left axis, they distort the relativity of the axes (they shift relative to one another). You can see this if you disable the rendering of the MACs graphs, or if you set the
I uploaded my raw log files to here and you can load and try them. Still, you ask why for you the graphs overlap and for me they don't. This is because, in my files, the big drop in the MACs (at episode 3474; to ~5%) causes the left and right axes to shift and they become unaligned. Cheers |
Hi @nzmora Got it! It's my carelessness. I didn't check the scale of axes. |
Hi @nzmora May I know why you set pi_lr = 1e-4, q_lr = 1e-3 in ddpg?
|
Hi @HKLee2040, |
@nzmora Hi,How do you get the YAML file of pruning schedule,Could you share the pruning schedule YAML file of resnet trained in IMAGENET,THKS! |
@nzmora @HKLee2040 I refer to every YAML file,here give it directly,but it does not say how to get it.You say AMC/ADC currently works w/o YAML,could you give an example which without YAML file,Thank you for your help! |
Hi @huxianer You can refer to nzmora's message The command-line is: |
@nzmora Hi,whether this Distiller supports detection model,and if not,do you have any intention to support it? |
I am also interested in using AMC for detection models. How about the progress now? |
Hi @huxianer , @RizhaoCai , I merged the revised AMC implementation to 'master'. You can now try our auto-compression code. It currently doesn't support object detection. @levzlotnik is working on adding an example of object detection, after which we will consider automating. If you happen to integrate object-detection with AMC, we'd be interested in considering it for integration into the Distiller code-base. Cheers |
Hi @levzlotnik @nzmora |
Hello there,
I am wondering about the state of the ADC implementation, and what remains to bring it to a functional state.
In the ADC merge commit message, you mentioned that it is still WiP and that it is using an unreleased version of Coach. Is that still the case?
Also, is there any documentation for how to use ADC in Distiller?
Thanks
The text was updated successfully, but these errors were encountered: