Question about search on custom dataset

[KleinXin]

First of all, thank you for your greate work.

I have a gender recognition project. The dataset I am using now is Celeba. I divide the dataset to female and male according to their lables, then follow the parameters you used to search on imagenet. As mentioned in the paper, during the first 35 of 50 epochs , only weights of the network are updated while the architechture is not changed. Then in the last 15 epochs, the architechture search is performed.

However, the results show that after 35 epochs, the accuracy of trainning/validation/ test are decressed dramatically. The architecture search does not bring any improvement but is harmful to the accuracy. The figure of the accuracy is shown below.

The parameters I used are shown below. Any parameters that are not shown are the same as the default values in file "train_search_imagenet.py"

[yuhuixu1993]

Hi, @KleinXin , thanks for your interest of our work. As you mentioned that it is a different dataset and task. the hyperparamters need to be consistent with your usual hyper parameters in your task which may have some difference with the training on imagenet. Besides, the accuracy of the search phase is not informative enough to judge the goodness of the searched architectures. You need to train from scratch with the final searched architectures. Besides, the stop epoch is also changeable.

[KleinXin]

Hi, @KleinXin , thanks for your interest of our work. As you mentioned that it is a different dataset and task. the hyperparamters need to be consistent with your usual hyper parameters in your task which may have some difference with the training on imagenet. Besides, the accuracy of the search phase is not informative enough to judge the goodness of the searched architectures. You need to train from scratch with the final searched architectures. Besides, the stop epoch is also changeable.

Do you think it is possible to complete searching and training in one-shot?
In fact, now we are trying to finish the recognition job in the searching step instead of searching architecture followed by training model.

[KleinXin]

Another question is that in 'train_search_imagenet.py' line 178, the parameter 'architect' is not transferred into train function as in 'train_searcch.py'.
In line 206 of 'train_search_imagenet.py', architect.step is commented.

Does this mean the architecture is not updated in 'train_search_imagenet.py' ？

[yuhuixu1993]

@KleinXin, in fact the architecture is updated. I just moved the update codes from architecture.py towards this file. The update code is the corresponding lines: 200~205. Hyperpaprameter args.begin controls from which epoch the arch paras begin to update. Besides, you can double check by see the printed arch para.

[rrryan2016]

Hi, @KleinXin , thanks for your interest of our work. As you mentioned that it is a different dataset and task. the hyperparamters need to be consistent with your usual hyper parameters in your task which may have some difference with the training on imagenet. Besides, the accuracy of the search phase is not informative enough to judge the goodness of the searched architectures. You need to train from scratch with the final searched architectures. Besides, the stop epoch is also changeable.

Thanks for your kind reply, really helpful.

I came across the same problem when applying to another classification task, and I just wonder do you meet similar problem that accuracy significantly decreases after epoch args.begin, when searching on CIFAR10 or ImageNet?

And I am interested in your log files when searching on CIFAR10 or ImageNet, just for reference. Would you like to share it ?