Skip to content
Implement Differentiable Architecture Search (DARTS) for RNN with fastai
Branch: master
Clone or download
Latest commit 7f214b1 Apr 16, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
data/penn
README.md
darts_callbacks.py
databunch_nb.ipynb init Apr 15, 2019
test_nb.ipynb init Apr 15, 2019
train.py init Apr 15, 2019
train_nb.ipynb init Apr 15, 2019
train_search.py
train_search_nb.ipynb init Apr 15, 2019

README.md

DARTS for RNN with fastai

Language model on Penn Treebank using Differentiable Architecture Search (DARTS) and fastai library.
Blog post.
Based on DARTS: Differentiable Architecture Search by Hanxiao Liu, Karen Simonyan, Yiming Yang.
Check out the original implementation.

Requirements

fastai 1.0.52.dev0 (latest as of 10th April 2019), PyTorch 1.0.

Instructions

  1. Run databunch_nb.ipynb to create databunch
  2. Run train_search_nb.ipynb to search for genotype. ~5 hours on 1 v100 gpu for 1 run.
    RNN search is sensitive to initialization so there should be several runs with different seed
  3. Train that genotype from scratch on train_nb.ipynb. ~1.5 days for 1600 epochs.
  4. Test a model using test_nb.ipynb

Pretrained model

Pretrained model of DARTS_V1 genotype after 600 epochs darts_V1.pth.
Place the file at data/models and run test_nb.ipynb. Loss ~4.22, 68.0 perplexity.
Caveat: I haven't been able to get ~58.0 test perplexity like the original implementation.

fastai dev version installation

git clone https://github.com/fastai/fastai
cd fastai
tools/run-after-git-clone
pip install -e ".[dev]"
git pull 
You can’t perform that action at this time.