Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improved training speed of GHNs, extra results for CIFAR-10 #2

Merged
merged 3 commits into from
Jan 12, 2022

Conversation

bknyaz
Copy link
Contributor

@bknyaz bknyaz commented Jan 12, 2022

Training times

Implementation of some steps in the decoder of GHNs is improved to speed up the training time of GHNs without altering their overall behavior. These improvements mainly affect the speed when a meta-batch size > 1 is used (see the tables below).

Speed is measured on NVIDIA Quadro RTX 6000 in terms of seconds per training iteration (averaged for the first 100 iterations).

CIFAR-10

Model Current version Our PR Estimated total speed up for 300 epochs
MLP with meta-batch size bm = 1 0.21 sec/iter 0.13 sec/iter 0.5 days -> 0.3 days
MLP with meta-batch size bm = 8 6.35 sec/iter 0.89 sec/iter 15.5 days -> 2.2 days
GHN-2 with meta-batch size bm = 1 0.77 sec/iter 0.72 sec/iter 1.9 days -> 1.8 days
GHN-2 with meta-batch size bm = 8 7.74 sec/iter 1.99 sec/iter 18.9 days -> 4.9 days

ImageNet

Model Current version Our PR Estimated total speed up for 300/150 epochs*
MLP with bm = 1 0.53 sec/iter 0.37 sec/iter 8.9 days -> 6.2 days
MLP with bm = 8 (4 GPUs) 1.78 sec/iter 1.36 sec/iter 14.9 days -> 11.4 days
GHN-2 with bm = 1 1.08 sec/iter 0.92 sec/iter 18.0 days -> 15.4 days
GHN-2 with bm = 8 (4 GPUs) 3.78 sec/iter 3.50 sec/iter 31.6 days -> 29.2 days

*To estimate the total training time, 300 epochs is used for bm=1 and 150 epochs is used for bm=8 (according to the paper).

When 4 GPUs and bm = 8 is used, the speed up is not significant, because each GPU receives only two architectures.

Evaluation of GHNs

To make sure that the evaluation results (classification accuracies of predicted parameters) reported in the paper are the same as in this PR, the GHNs were evaluated on selected architectures and the same results were obtained (see the table below).

Model ResNet-50 ViT Test Architecture (index in the test split)
GHN-2-CIFAR-10 58.6 11.4 77.1 (210)
GHN-2-ImageNet 5.3 4.4 48.3 (85)

Extra results on CIFAR-10

  • Added file data/results_cifar10_extended.json containing training and evaluation curves (when training with SGD) for 1402 evaluation architectures of DeepNets-1M.
  • The results of evaluation GHNs on validation and noisy images of CIFAR-10 are added.
  • Jupyter notebook examples/read_cifar10_results.ipynb is added to read and visualize the training/evaluation curves.
    The image below is obtained using this notebook.

c10_extended_results

Other minor updates

  • --amp flag was added that can be used to decrease GPU memory consumption and, in some cases, improve speed (this flag was used to measure speed on ImageNet with 4 GPUs)
  • md5sum values of the DeepNets-1M files were added to make it easier to verify the dataset

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jan 12, 2022
@michaldrozdzal michaldrozdzal merged commit 9b2a4b3 into facebookresearch:main Jan 12, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants