Use eltype(x) everywhere, ignore typeof(η)

[mcabbott]

First commit should fix #150, by working all in Float32. Chooses to do this by always following the momentum from setup, not the x, but perhaps that's messy? changed to eltype(x) now.

Since the type of η is now ignored, perhaps we should just remove all the type parameters? Then e.g. Adam(0) == Adam(0.0), fixes #119, closes #120. Also closes #86 (redundant).

Not all rules are changed yet, RFC?

[CarloLucibello]

we should just remove all the type parameters?

definitely, we also get nicer prints of the opt_state has a bonus

[ToucheSir]

If using the eltype of momentum is too restrictive, we could always change to some promoted eltype of various parts of input + state (e.g. momentum + param + gradient eltypes). Otherwise 👍 from me though, nice to see that this works!

[mcabbott]

All rules now changed, and tests pass. The big question is probably whether this counts as a breaking change, or not?

[mcabbott]

This doesn't prevent an error from Yota not compiling on master. Could revert as it's unrelated to this PR.

[CarloLucibello]

since this doesn't solve the CI problem let's remove the change from this PR

[mcabbott]

Reverted.

[darsnack]

We removed Descent{T}(…) type constructors, so unfortunately, yes.

[mcabbott]

In fact Descent{T} was the one I left, since IIRC someone had a paper doing "gradient descent" with complex learning rate.

I doubt anyone types that, but the concern is things like BSON, right?

[mcabbott]

I merge this now, we can think about whether anything else wants to be ganged into a breaking release.

[CarloLucibello]

Tagging a new breaking release now since this fixes a lot of problems and I don't see any breaking change on the horizon.