Please test and add the AdaBound optimizer into the next stable release

[xuancong84]

Now we have another powerful general purpose optimizer called AdaBound that trains as fast as Adam and as good as SGD, see https://github.com/Luolc/AdaBound.

It appears to work better than Adam in several tasks including NLP and CV, with slightly higher accuracy and smoothier learning curve.

I think this would be a great and useful addition, thanks!

[vishwakftw]

New implementations such as these can go into pytorch/contrib initially.

[kayuksel]

I also agree with the request of adding AdaBound optimizer to contrib. The provided implementation by the author unfortunately does not have support for PyTorch 1.0

[soumith]

I'd agree, send a PR to https://github.com/pytorch/contrib with unit tests, I'm happy to merge