Thanks and Question

[LukeAI]

Thanks for putting this together! I'm giving DiffGrad a try now.

I know that you haven't run benchmarks, but you have obviously read a lot of papers and implemented a lot of optimizers so would be great to get your opinion on the most promising optimiser(s) to try for CNNs?

[jettify]

Hard to recommend any optimizer, best way to find one that works for your problem/data is to try. May be try Yogi or Lookahaed + Yogi or combine any other optimizer with Lookahaed :)