These are papers which I think are worth reading. No particular order. My interests focus on hyperparameter/architecture optimization, semi-supervised learning, and computer vision. NF is not found, WBR is will be released. Importance is just a vague estimate of the importance to me. TODO means I haven't read in sufficient detail yet. I haven't always looked very hard for all the associated links and I might have messed up some of them. PRs with more papers or fixs are welcome.
| name | arXiv | github | open review | desc | misc | importance | ||
|---|---|---|---|---|---|---|---|---|
| AdaBound | arxiv | github | open review | optimizer | new, not proven, TODO | 6/10 | ||
| Evaluating the Search Phase of NAS | arxiv | WBR | NF | evaluating NAS approaches vs random search | TODO | 8/10 | ||
| Random Search and Reproducibility for NAS | arxiv | github | NF | evaluating NAS approaches vs random search | TODO | 8/10 | ||
| Neural Architecture Optimization | arxiv | github | NF | Uses encoder decoder for gradient based NAS | 4/10 | |||
| DARTS: Differentiable Architecture Search | arxiv | original alternative | open review | Uses scalar multiples of branches for continuous relaxation | Alternate implemention is cleaner, but less verified | 8/10 | ||
| Efficient Neural Architecture Search via Parameter Sharing | arxiv | github | open review | Like original NAS/PNAS, but with weight sharing | implemention is in TF | 5/10 | ||
| Learning Transferable Architectures for Scalable Image Recognition | arxiv | just the model | NF | original NASNet | medium | 7/10 | ||
| ProxylessNAS | arxiv | WBR just the model | open review | architecture search which adjusts branch probabilities and optimizes for latency | 8/10 | |||
| Deep Residual Learning for Image Recognition | arxiv | github | NF | ResNets | medium | 10/10 | ||
| FractalNet | arxiv | github | open review | fractal architecture to reduce path length | residual nets won the war | 5/10 | ||
| Identity Mappings in Deep Residual Networks | arxiv | github | NF | ResNet v2 | 8/10 | |||
| Wide Residual Networks | arxiv | github | NF | ResNet variant with wider layers and less depth | 7/10 | |||
| EraseReLU | arxiv | NF | NF | ResNet variant with removed ReLU | 2/10 | |||
| Deep Networks with Stochastic Depth | arxiv | github | open review | introduces stochastic depth in resnets | 8/10 | |||
| Deep Pyramidal Residual Networks | arxiv | github | NF | optimizes resnet architecture including additive channels | medium best simple architecture for classification IMO | 9/10 | ||
| Deep Pyramidal Residual Networks with Separated Stochastic Depth | arxiv | github | NF | open review | tests ResDrop in pyramid net | not great paper or very innovative IMO, including for relevance to ShakeDrop | 1/10 | |
| Shake-Shake regularization | arxiv | github | open review | stochasticly combines output from multiple branches | 6/10 | |||
| ShakeDrop Regularization for Deep Residual Learning | arxiv | github | NF | open review | regularization technique for pyramid net | under rated IMO | 6/10 | |
| MobileNets | arxiv | github | NF | lightweight architecture for mobile, depthwise separable convolution | implemention is in TF blog | 6/10 | ||
| MobileNet v2 | arxiv | github | NF | improved mobilenet | implemention is in TF blog | 6/10 | ||
| ShuffleNet | arxiv | github | NF | lightweight architecture for mobile, introduces channel shuffle | 7/10 | |||
| ShuffleNet v2 | arxiv | github | NF | NF | specific hardware focused optimizations | 5/10 | ||
| Super-Convergence | arxiv | github | open review | approach for using learning rate schedule to speed up training | in practice leads to higher variance in final validation loss, works very well in general | 7/10 | ||
| Decoupled Weight Decay Regularization | arxiv | pr | initial open review second open review | adam works better with decoupled weight decay | always use decoupled weight decay | 7/10 | ||
| Parallel Architecture and Hyperparameter Search via Successive Halving and Classification | arxiv | github | NF | NF | search approach | simple, not best choice for architecture search IMO | 5/10 | |
| Fixup Initialization | arxiv | github | open review | initialization scheme which allows for no batchnorm | very architecture specific | 4/10 | ||
| Squeeze-and-Excitation Networks | arxiv | github | NF | novel block for CNN | seems like an easy win | 9/10 | ||
| Batch Renormalization | arxiv | github | NF | approach for using batch normalization when the batch size must be small | I found that it doesn't make much difference unless the batch size is very small, implemention is in keras | 5/10 | ||
| GroupNorm | arxiv | github | NF | alternative to batch normalization, better when batch must be very small | 5/10 | |||
| Attention Is All You Need | arxiv | github | NeurIPS Review | introduced transformer model for NLP | medium TODO | 9/10 | ||
| Deep Contextualized Word Representations (ELMo) | arxiv | github | NF | better embeddings | blog implemention is in TF, TODO | 5/10 | ||
| BERT | arxiv | github | NF | state of the art NLP model which is easy to fine tune for a variety of tasks | variety of explanations online, illustrated BERT, TODO | 9/10 | ||
| Transformer-XL | arxiv | github | open review | arbitrary sequence length transformer | TODO | 8/10 | ||
| Mean Teacher | arxiv | github | NF | open review | semi-supervised learning technique | 5/10 | ||
| Virtual Adversarial Training | arxiv | github | NF | NF | semi-supervised learning technique | TODO, right now SOTA | 7/10 | |
| Realistic Evaluation of Deep Semi-Supervised Learning Algorithms | arxiv | github | open review | creates evaluation framework for semi-supervised learning and possible errors when evaluating semi-supervised techniques | implemention is in TF | 7/10 |