Clean up variable vs weight in base layer

[mattdangerw]

Clean up docstrings to clarify variables vs weights.

[fchollet]

Not quite! We still intend for our users to use add_weights, trainable_weights, set_weights, etc. as their primary accessor for weights and their values. Though some users (mainly JAX stateless API users) will resort to variables as their go-to.

There's a (subtle) difference between variables and weights.

Most users won't care about the difference or won't understand it -- so they should generally use weights because it will match their expectations.

Weights are variables that parameterize the forward pass. They are the parameters of the manifold learned by the model. For instance, a Conv kernel, or the BN statistics are weights.

Variables are more general than that. They include weights, but also various side-variables that aren't weights, such as RNG seeds, metric state variables (if any metric is attached), and possibly some metadata variables like counters.

You will use variables if you are referring to the "state" of a layer in general, as you would in JAX.
You will use weights if you are referring to forward pass parameters.

[mattdangerw]

Thanks! I still think there are a few cleanups to be made here, but good to know the distinction! I will rework this.

[mattdangerw]

OK! Update this to be a pure docstring fix. Tried to clarify a little more, and non_trainable_weights no longer claims to contain metrics and RNG state, which seems false.

Overall, this still feels a little confusing. We refer to weights and variables interchangeably for the most part. layer.add_weight() == layer.add_variable() == layer = backend.Variable(). Then we draw a distinction in one context. I might actually find it easier to grok if we had layer.trainable_state, layer.non_trainable_state and layer.stateless_call, because the API would align around a new term "state". weights is all Variables used by layers and sub-layers, state is all Variables used by layers, metrics and generators. But that is probably too disruptive of a change right now.

Do we want layer.get_variables() and layer.set_variables() as an analogue to layer.get_weights() and layer.set_weights()?

[fchollet]

Do we want layer.get_variables() and layer.set_variables() as an analogue to layer.get_weights() and layer.set_weights()?

I would say, probably not for now. We can revisit later though.

[fchollet]

Yeah this pretty confusing. Anyway, thanks for the fixes!