训练速度过慢的问题？

[FLHonker]

MNIST从原本的10epcoh在DAFL框架下要训练200epoch，CIFAR-10要训练2000epoch。而且GAN的训练也耗时，经过实验比标准KD训练时间长了20-30倍。

[MingSun-Tse]

(I am not among the authors. Just for discussion.) I think this could be normal. One possible reason is that the data is not real. The information per sample can be limited, so basically the student network needs to see many more samples than the training on real data.

[FLHonker]

But I think this speed is unacceptable in actual use. Moreover, only small data sets are used in the experiment. If semantic segmentation, imagenet, and high-resolution images tasks are used, the computational complexity is very large. It is estimated that GAN cannot reasonably infer the distribution equivalent to real data.

[FLHonker]

(I am not among the authors. Just for discussion.) I think this could be normal. One possible reason is that the data is not real. The information per sample can be limited, so basically the student network needs to see many more samples than the training on real data.

欢迎star我的仓库一起交流KD：https://github.com/FLHonker/Awesome-Knowledge-Distillation

[MingSun-Tse]

But I think this speed is unacceptable in actual use. Moreover, only small data sets are used in the experiment. If semantic segmentation, imagenet, and high-resolution images tasks are used, the computational complexity is very large. It is estimated that GAN cannot reasonably infer the distribution equivalent to real data.

Yeah, you've made a point. ImageNet would be substantially harder. It definitely has a long road before practical use. But Rome is not built in one day. I think this paper can be a good start.

[FLHonker]

But I think this speed is unacceptable in actual use. Moreover, only small data sets are used in the experiment. If semantic segmentation, imagenet, and high-resolution images tasks are used, the computational complexity is very large. It is estimated that GAN cannot reasonably infer the distribution equivalent to real data.

Yeah, you've made a point. ImageNet would be substantially harder. It definitely has a long road before practical use. But Rome is not built in one day. I think this paper can be a good start.

我也一直试图改进这个问题，除非抛弃GAN，GAN的训练是个痛点。data-free是个很有意思的topic。

[HantingChen]

Thanks for MingSun-Tse's answer. That's right. We will develop a more efficient data-free learning method in the future work.