| title | authors | owning-sig | participating-sigs | reviewers | approvers | editor | creation-date | last-updated | status | see-also | replaces | superseded-by | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Apply |
|
sig-api-machinery |
|
|
|
TBD |
2018-03-28 |
2018-03-28 |
implementable |
|
|
|
kubectl apply is a core part of the Kubernetes config workflow, but it is
buggy and hard to fix. This functionality will be regularized and moved to the
control plane.
Example problems today:
- User does POST, then changes something and applies: surprise!
- User does an apply, then
kubectl edit, then applies again: surprise! - User does GET, edits locally, then apply: surprise!
- User tweaks some annotations, then applies: surprise!
- Alice applies something, then Bob applies something: surprise!
Why can't a smaller change fix the problems? Why hasn't it already been fixed?
- Too many components need to change to deliver a fix
- Organic evolution and lack of systematic approach
- It is hard to make fixes that cohere instead of interfere without a clear model of the feature
- Lack of API support meant client-side implementation
- The client sends a PATCH to the server, which necessitated strategic merge patch--as no patch format conveniently captures the data type that is actually needed.
- Tactical errors: SMP was not easy to version, fixing anything required client and server changes and a 2 release deprecation period.
- The implications of our schema were not understood, leading to bugs.
- e.g., non-positional lists, sets, undiscriminated unions, implicit context
- Complex and confusing defaulting behavior (e.g., Always pull policy from :latest)
- Non-declarative-friendly API behavior (e.g., selector updates)
"Apply" is intended to allow users and systems to cooperatively determine the desired state of an object. The resulting system should:
- Be robust to changes made by other users, systems, defaulters (including mutating admission control webhooks), and object schema evolution.
- Be agnostic about prior steps in a CI/CD system (and not require such a system).
- Have low cognitive burden:
- For integrators: a single API concept supports all object types; integrators have to learn one thing total, not one thing per operation per api object. Client side logic should be kept to a minimum; CURL should be sufficient to use the apply feature.
- For users: looking at a config change, it should be intuitive what the system will do. The “magic” is easy to understand and invoke.
- Error messages should--to the extent possible--tell users why they had a conflict, not just what the conflict was.
- Error messages should be delivered at the earliest possible point of intervention.
Goal: The control plane delivers a comprehensive solution.
Goal: Apply can be called by non-go languages and non-kubectl clients. (e.g., via CURL.)
- Multi-object apply will not be changed: it remains client side for now
- Some sources of user confusion will not be addressed:
- Changing the name field makes a new object rather than renaming an existing object
- Changing fields that can’t really be changed (e.g., Service type).
(Please note that when this KEP was started, the KEP process was much less well defined and we have been treating this as a requirements / mission statement document; KEPs have evolved into more than that.)
A brief list of the changes:
- Apply will be moved to the control plane.
- The original design is in a google doc; joining the kubernetes-dev or kubernetes-announce list will grant permission to see it. Since then, the implementation has changed so this may be useful for historical understanding. The test cases and examples there are still valid.
- Additionally, readable in the same way, is the original design for structured diff and merge; we found in practice a better mechanism for our needs (tracking field managers) but the formalization of our schema from that document is still correct.
- Apply is invoked by sending a certain Content-Type with the verb PATCH.
- Instead of using a last-applied annotation, the control plane will track a "manager" for every field.
- Apply is for users and/or ci/cd systems. We modify the POST, PUT (and non-apply PATCH) verbs so that when controllers or other systems make changes to an object, they are made "managers" of the fields they change.
- The things our "Go IDL" describes are formalized: structured merge and diff
- Existing Go IDL files will be fixed (e.g., by fixing the directives)
- Dry-run will be implemented on control plane verbs (POST, PUT, PATCH).
- Admission webhooks will have their API appended accordingly.
- An upgrade path will be implemented so that version skew between kubectl and the control plane will not have disastrous results.
The linked documents should be read for a more complete picture.
(TODO: update this section with current design)
Server-side apply has to understand the topology of the objects in order to make valid merging decisions. In order to reach that goal, some new Go markers, as well as OpenAPI extensions have been created:
Lists can behave in mostly 3 different ways depending on what their actual semantic is. New annotations allow API authors to define this behavior.
-
Atomic lists: The list is owned by only one person and can only be entirely replaced. This is the default for lists. It is defined either in Go IDL by pefixing the list with
// +listType=atomic, or in the OpenAPI with"x-kubenetes-list-type": "atomic". -
Sets: the list is a set (it has to be of a scalar type). Items in the list must appear at most once. Individual actors of the API can own individual items. It is defined either in Go IDL by pefixing the list with
// +listType=set, or in the OpenAPI with"x-kubenetes-list-type": "set". -
Associative lists: Kubernetes has a pattern of using lists as dictionary, with "name" being a very common key. People can now reproduce this pattern by using
// +listType=map, or in the OpenAPI with"x-kubernetes-list-type": "map"along with"x-kubernetes-list-map-keys": ["name"], or// +listMapKey=name. Items of an associative lists are owned by the person who applied the item to the list.
For compatibility with the existing markers, the patchStrategy and
patchMergeKey markers are automatically used and converted to the corresponding listType
and listMapKey if missing.
Maps and structures can behave in two ways:
- Each item in the map or field in the structure are independent from each
other. They can be changed by different actors. This is the default behavior,
but can be explicitly specified with
// +mapType=granularor// +structType=granularrespectively. They map to the same openapi extension:"x-kubernetes-map-type": "granular". - All the fields or item of the map are treated as one unit, we say the map/struct is
atomic. That can be specified with
// +mapType=atomicor// +structType=atomicrespectively. They map to the same openapi extension:"x-kubernetes-map-type": "atomic".
Since server-side apply is currently in the Alpha phase, it is not
enabled by default on kubectl. To use server-side apply on servers
with the feature, run the command
kubectl apply --experimental-server-side ....
If the feature is not available or enabled on the server, the command will fail rather than fall-back on client-side apply due to significant semantical differences.
As the feature graduates to the Beta phase, the flag will be renamed to --server-side.
The long-term plan for this feature is to be the default apply on all Kubernetes clusters. The semantical differences between server-side apply and client-side apply will make a smooth roll-out difficult, so the best way to achieve this has not been decided yet.
Right before being persisted to etcd, resources in the apiserver undergo a preparation mechanism that is custom for every resource kind. It takes care of things like incrementing object generation and status wiping. This happens through PrepareForUpdate and PrepareForCreate.
The problem status wiping at this level creates is, that when a user applies a field that gets wiped later on, it gets owned by said user. The apply mechanism (FieldManager) can not know which fields get wiped for which resource and therefor can not ignore those.
Additionally ignoring status as a whole is not enough, as it should be possible to own status (and other fields) in some occasions. More conversation on this can be found in the GitHub issue where the problem got reported.
Add an interface that resource strategies can implement, to provide field sets affected by status wiping.
# staging/src/k8s.io/apiserver/pkg/registry/rest/rest.go
// ResetFieldsProvider is an optional interface that a strategy can implement
// to expose a set of fields that get reset before persisting the object.
type ResetFieldsProvider interface {
// ResetFieldsFor returns a set of fields for the provided version that get reset before persisting the object.
// If no fieldset is defined for a version, nil is returned.
ResetFieldsFor(version string) *fieldpath.Set
}Additionally, this interface is implemented by registry.Store which forwards it to the corresponding strategy (if applicable).
If registry.Store can not provide a field set, it returns nil.
An example implementation for the interface inside the pod strategy could be:
# pkg/registry/core/pod/strategy.go
// ResetFieldsFor returns a set of fields for the provided version that get reset before persisting the object.
// If no fieldset is defined for a version, nil is returned.
func (podStrategy) ResetFieldsFor(version string) *fieldpath.Set {
set, ok := resetFieldsByVersion[version]
if !ok {
return nil
}
return set
}
var resetFieldsByVersion = map[string]*fieldpath.Set{
"v1": fieldpath.NewSet(
fieldpath.MakePathOrDie("status"),
),
}When creating the handlers in installer.go the current rest.Storage is checked to implement the ResetFieldsProvider interface and the result is passed to the FieldManager.
# staging/src/k8s.io/apiserver/pkg/endpoints/installer.go
var resetFields *fieldpath.Set
if resetFieldsProvider, isResetFieldsProvider := storage.(rest.ResetFieldsProvider); isResetFieldsProvider {
resetFields = resetFieldsProvider.ResetFieldsFor(a.group.GroupVersion.Version)
}When provided with a field set, the FieldManager strips all resetFields from incoming update and apply requests.
This causes the user/manager to not own those fields.
...
if f.resetFields != nil {
patchObjTyped = patchObjTyped.Remove(f.resetFields)
}
...We looked at a way to get the fields affected by status wiping without defining them separately.
Mainly by pulling the reset logic from the strategies PrepareForCreate and PrepareForUpdate methods into a new method ResetFields implementing an ObjectResetter interface.
This approach did not work as expected, because the strategy works on internal types while the FieldManager handles external api types. The conversion between the two and creating the diff was complex and would have caused a notable amount of allocations.
- 12/2019 #86083 implementing a poc for the described approach
We used a feature branch to ensure that no partial state of this feature would be in master. We developed the new "business logic" in a separate repo for velocity and reusability.
The specific logic of apply will be tested by extensive unit tests in the
structured merge and diff
repo. The integration between that repo and kubernetes/kubernetes will mainly
be tested by integration tests in test/integration/apiserver/apply
and test/cmd,
as well as unit tests where applicable. The feature will also be enabled in the
alpha-features e2e test suite,
which runs every hour and everytime someone types /test pull-kubernetes-e2e-gce-alpha-features
on a PR. This will ensure that the cluster can still start up and the other
endpoints will function normally when the feature is enabled.
Unit Tests in structured merge and diff repo for:
- Merge typed objects of the same type with a schema. link
- Merge deduced typed objects without a schema (for CRDs). link
- Convert a typed value to a field set. link
- Diff two typed values. link
- Validate a typed value against it's schema. link
- Get correct conflicts when applying. link
- Apply works for deduced typed objects. link
- Apply works for leaf fields with scalar values. link
- Apply works for items in associative lists of scalars. link
- Apply works for items in associative lists with keys. link
- Apply works for nested schemas, including recursive schemas. link
- Apply works for multiple appliers. link
- Apply works when the object conversion changes value of map keys. link
- Apply works when unknown/obsolete versions are present in managedFields (for when APIs are deprecated). link
Unit Tests for:
- Apply strips certain fields (like name and namespace) from managers. link
- ManagedFields API can be round tripped through the structured-merge-diff format. link
- Manager identifiers passed to structured-merge-diff are encoded as json. link
- Managers will be sorted by operation, then timestamp, then manager name. link
- Conflicts will be returned as readable status errors. link
- Fields API can be round tripped through the structured-merge-diff format. link
- Fields API conversion to and from the structured-merge-diff format catches errors. link
- Path elements can be round tripped through the structured-merge-diff format. link
- Path element conversion will ignore unknown qualifiers. link
- Path element confersion will fail if a known qualifier's value is invalid. link
- Can convert both built-in objects and CRDs to structured-merge-diff typed objects. link
- Can convert structured-merge-diff typed objects between API versions. link
Integration tests for:
- Creating an object with apply works with default and custom storage implementations. link
- Create is blocked on apply if uid is provided. link
- Apply has conflicts when changing fields set by Update, and is able to force. link
- There are no changes to the managedFields API. link
- ManagedFields has no entries for managers who manage no fields. link
- Apply works with custom resources. link
- Run kubectl apply tests with server-side flag enabled. link
An alpha version of this is targeted for 1.14.
This can be promoted to beta when it is a drop-in replacement for the existing kubectl apply, and has no regressions (which aren't bug fixes). This KEP will be updated when we know the concrete things changing for beta.
This will be promoted to GA once it's gone a sufficient amount of time as beta with no changes. A KEP update will precede this.
- Early 2018: @lavalamp begins thinking about apply and writing design docs
- 2018Q3: Design shift from merge + diff to tracking field managers.
- 2019Q1: Alpha.
(For more details, one can view the apply-wg recordings, or join the mailing list and view the meeting notes. TODO: links)
Why should this KEP not be implemented: many bugs in kubectl apply will go away. Users might be depending on the bugs.
It's our belief that all routes to fixing the user pain involve centralizing this functionality in the control plane.