Controller selectors should be immutable after creation

[crimsonfaith91]

Background
As part of the greater controller GA effort, we have removed defaulting spec.selector to spec.template.labels values in apps/v1beta2 as the defaulting operation is semantically broken. In this issue, we propose to make spec.selector immutable after API object creation.

Motivation

1. Changing selectors leads to undefined behaviors - users are not expected to change the selectors
2. Having selectors immutable ensures they always match created children, preventing events such as accidental bulk orphaning
3. Having selectors mutable for GA is an immortal decision, which does not guarantee API backward compatibility

Immutability resolves some challenging issues that plague many teams for over a year e.g., #34292, and the current behavior for controllers with respect to selector mutations is undefined e.g., #26202.

In contrast, by making selectors immutable, validation ensures the selectors always match template labels of created children. This provides a risk-free, deterministic behavior to resolve the issue, and meet the general goal of declarative configuration.

We could lift the immutability in future after all controller types have well-defined behaviors (i.e., after GA). This provides ability to address new use cases while maintaining API backward compatibility. However, if we do not make selectors immutable prior to GA, we will have to live with mutable selectors for the lifetime of the v1 API and the controllers’ undefined behaviors until reasonable semantics are decided and implemented.

/kind bug
/sig apps
/sig api-machinery
@bgrant0607 @erictune @kow3ns @enisoc @janetkuo @foxish @liyinan926
@liggitt @smarterclayton @Kargakis @mikedanese

[smarterclayton]

Removing defaulting is a breaking API change? Can you link the issue where we removed it?

[crimsonfaith91]

@smarterclayton The server-side defaulting was removed for apps/v1beta2 only.
The (closed) issue: #50339
The (merged) PR: #50164

[liggitt]

Immutability resolves some challenging issues that plague many teams for over a year e.g., #34292, and the current behavior for controllers with respect to selector mutations is undefined e.g., #26202.

Both of those issues look like they were related to defaulting, not immutability.

[liggitt]

In contrast, by making selectors immutable, validation ensures the selectors always match template labels of created children.

For that to work, you would also have to make the template labels immutable, which is another significant change (updating template labels is currently allowed).

However, if we do not make selectors immutable prior to GA, we will have to live with mutable selectors for the lifetime of the v1 API

I don't see a clear way to enforce immutability for the v1 API only, while staying compatible with the existing beta API.

[0xmichalis]

@mfojtik @tnozicka

[kow3ns]

@smarterclayton defaulting is removed only in apps/v1beta2 not in apps/v1beta1. As you say, that is a breaking change that we can't make to the existing group version.

@liggitt if we make the selector immutable, and require via validation (as we do now) that the labels match the provided selector for the object to be valid, we get the desired behavior. This is a less strict requirement than making the labels immutable.

Wrt #26202, it contains a long discussion about the problems regarding selector mutation and links to other relevant issues. @bgrant0607's @smarterclayton's comments in this thread are still as true right now as they were at the time the issue was opened. Our controllers were not designed to handle selector changes gracefully, and whatever the user is attempting to achieve by mutating the selector they could achieve by other means (mostly).

I'm not trying to say that we should not support mutable selectors in the future, but we don't support it gracefully now, and we don't have a design for what the exact semantics of selector mutation are (and they may not uniform across all our controllers).

If we make the selectors immutable now, the community has time to offer designs for how selector mutations should work (assuming this feature generates enough interest), and we can make them mutable without breaking backward compatibility.

If we advance to apps/v1 with mutable selectors, we have to support the current undefined semantics for the lifetime of that group version.

To be clear we want to implement selector immutability in such a way apps/v1beta1 is not affected, and only API versions greater than this have immutable selectors.

[kow3ns]

@smarterclayton issue for defaulting #50339

[crimsonfaith91]

@liggitt @smarterclayton @Kargakis @mikedanese @mfojtik @tnozicka
I came out with a PR that makes the selector immutable without changing validation codes. Feel free to review it to ensure it looks good to you. Thanks!
#50719

[bgrant0607]

First of all, I'd like to find a way to make this happen. I think it will be less error-prone for users. I think there are some patterns we will want to allow, such as allowing new !key selector terms in order to facilitate adoption, canaries, and other evolution scenarios, but those can be pushed into the future.

Note that we still intend to make validation versioned eventually: #8550

Also, if we don't want to break old API versions now, we'd still need to allow mutation via old API paths, which means that clients of the new API will potentially observe changes but not be able to make them. I think that's fine, because we should clearly document that observers should be able to deal with selector changes, which will be necessary if we plan to allow mutations again in the future.