DARTS for RNN with fastai
Language model on Penn Treebank using Differentiable Architecture Search (DARTS) and fastai library.
Based on DARTS: Differentiable Architecture Search by Hanxiao Liu, Karen Simonyan, Yiming Yang.
Check out the original implementation.
fastai 1.0.52.dev0 (latest as of 10th April 2019), PyTorch 1.0.
- Run databunch_nb.ipynb to create databunch
- Run train_search_nb.ipynb to search for genotype. ~5 hours on 1 v100 gpu for 1 run.
RNN search is sensitive to initialization so there should be several runs with different seed
- Train that genotype from scratch on train_nb.ipynb. ~1.5 days for 1600 epochs.
- Test a model using test_nb.ipynb
Pretrained model of DARTS_V1 genotype after 600 epochs
Place the file at data/models and run test_nb.ipynb. Loss ~4.22, 68.0 perplexity.
Caveat: I haven't been able to get ~58.0 test perplexity like the original implementation.
fastai dev version installation
git clone https://github.com/fastai/fastai cd fastai tools/run-after-git-clone pip install -e ".[dev]" git pull