Use server-side diff by default #1556

lblackstone · 2021-04-30T22:57:33Z

Server-side diff has been available since 1.16 (September 2019), so it should be pretty widely supported on deployed clusters by now. Most of the implementation work is done already, so the remaining work should mostly be testing, and then swapping server-side diff to be the default.

As of April 30, 2021, EKS and AKS still support 1.15 clusters, but it appears that this support will be ending soon with the recent upstream release of k8s 1.21.

There is one open issue that also needs investigation: #683

Once we're confident in the test coverage, the enableDryRun flag should flip polarity so that it defaults to True.

Edit: The enableDryRun flag has been deprecated in favor of the enableServerSideApply flag in the v3.20.0 release, and will be removed in the next major release.

The text was updated successfully, but these errors were encountered:

lblackstone · 2021-04-30T22:59:42Z

For some additional context, server-side diff fixes the following issues:
#924
#694
#641

lukehoban · 2021-05-02T06:47:04Z

If we move to server-side diff, does that also enable not emitting last-applied-configuration? I believe that was largely only emitted to help with client-side diffs. If so - that might help with #1048 and #1118.

lblackstone · 2021-05-03T20:52:33Z

If we move to server-side diff, does that also enable not emitting last-applied-configuration?

I believe so. I'm not sure if that has any implications for kubectl integration currently, but my understanding is that the managedFields metadata used by server-side apply provides a more fine-grained alternative to accomplish the same goal.

Edit: Found this article about SSA that explains:

Compared to the last-applied annotation managed by kubectl, Server Side Apply uses a more declarative approach, which tracks a user's field management, rather than a user's last applied state.

kubectl server-side apply keeps the last-applied-configuration annotation up-to-date if you use kubectl apply

stanislav-zaprudskiy · 2021-11-12T08:08:43Z

If we move to server-side diff, does that also enable not emitting last-applied-configuration? I believe that was largely only emitted to help with client-side diffs. If so - that might help with #1048 and #1118.

Based on a simple test - it looks like the annotation is still being set even with SSA; or something else isn't properly configured/set.

I'm trying to deploy kube-prometheus-stack, but prometheuses CRD fails to be deployed with pulumi complaining on

error: resource prometheuses.monitoring.coreos.com was not successfully created by the Kubernetes API server : CustomResourceDefinition.apiextensions.k8s.io "prometheuses.monitoring.coreos.com" is invalid: metadata.annotations: Too long: must have at most 262144 bytes

I tried deploying it with SSA flag (PULUMI_K8S_ENABLE_DRY_RUN=true / enable_dry_run=True), but running into the same error - now also during preview as well. The code to reproduce:

from pulumi_kubernetes.yaml import ConfigGroup

def remove_status_from_crd(obj):
    if obj["kind"] == "CustomResourceDefinition":
        try:
            del obj["status"]
        except KeyError:
            pass

prometheus_crd = ConfigGroup(
    "prometheus-crd",
    files=[
        "https://raw.githubusercontent.com/prometheus-community/helm-charts/main/charts/kube-prometheus-stack/crds/crd-prometheuses.yaml",  # noqa: E501; pylint: disable=C0301
    ],
    transformations=[
        remove_status_from_crd,
    ],
)

It's Docker Desktop kubernetes:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.3", GitCommit:"ca643a4d1f7bfe34773c74f79527be4afd95bf39", GitTreeState:"clean", BuildDate:"2021-07-15T20:58:09Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.3", GitCommit:"ca643a4d1f7bfe34773c74f79527be4afd95bf39", GitTreeState:"clean", BuildDate:"2021-07-15T20:59:07Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"linux/amd64"}

lblackstone · 2021-11-12T22:38:36Z

Based on a simple test - it looks like the annotation is still being set even with SSA

Right, the annotation is still present in both modes. I'm hoping to remove it from the SSA mode in the future.

sathieu · 2021-11-26T09:48:07Z

Have you tried :

kubectl apply --server-side --field-manager=pulumi

Ref: https://kubernetes.io/docs/reference/using-api/server-side-apply/#upgrading-from-client-side-apply-to-server-side-apply

KaoruDev · 2022-06-28T21:21:37Z

Is there documentation on how I would go about migrating to having this flag turned on? When I flip this to true and run pulumi update I get the following error:

 panic: fatal: An assertion has failed: path: [spec nodeSets 0 podTemplate metadata creationTimestamp]  |  v: <nil>  | old: <nil>  |  oldInput: <nil>
    goroutine 657 [running]:
    github.com/pulumi/pulumi/sdk/v3/go/common/util/contract.failfast(...)
        /home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/v3@v3.34.0/go/common/util/contract/failfast.go:23
    github.com/pulumi/pulumi/sdk/v3/go/common/util/contract.Assertf(0x0, {0x22c6cfa, 0x8}, {0xc00124d768, 0x37eff60, 0xc00045eea0})
        /home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/v3@v3.34.0/go/common/util/contract/assert.go:33 +0xf4
    github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter).addPatchValueToDiff(0xc00124f398, {0xc001eea800, 0x0, 0x8}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, ...)
        /home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:2913 +0x188
    github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter).addPatchMapToDiff(0xc001eea800, {0xc001eea800, 0x5, 0x8}, 0x203000, 0x203000, 0xc001eea800, 0x3, 0x1)
        /home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:3024 +0x65a
    github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter).addPatchValueToDiff(0xc00124f398, {0xc001eea800, 0x8, 0x8}, {0x1f39380, 0xc002cc9ad0}, {0x1f39380, 0xc002e7a9f0}, {0x1f39380, 0xc002e68ab0}, ...)
        /home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:2936 +0x2b0
    github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter).addPatchMapToDiff(0xc001ec1d80, {0xc001ec1d80, 0x4, 0x4}, 0x2, 0xffffffffffffffff, 0x40, 0x202b280, 0x1)
        /home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:3015 +0x3f4
    github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter).addPatchValueToDiff(0xc00124f398, {0xc001ec1d80, 0x8, 0x4}, {0x1f39380, 0xc002cc9aa0}, {0x1f39380, 0xc002e7a9c0}, {0x1f39380, 0xc002e68a80}, ...)
        /home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:2936 +0x2b0
    github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter).addPatchMapToDiff(0xc001ec1d80, {0xc001ec1d80, 0x3, 0x4}, 0x203000, 0x203000, 0xc001ec1d80, 0x57780f, 0x1)
        /home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:3015 +0x3f4
    github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter).addPatchValueToDiff(0xc00124f398, {0xc001ec1d80, 0x8, 0x4}, {0x1f39380, 0xc002cc9a40}, {0x1f39380, 0xc002e7a960}, {0x1f39380, 0xc002e68a20}, ...)
        /home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:2936 +0x2b0
    github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter).addPatchArrayToDiff(0x0, {0xc002061b60, 0x2, 0x2}, {0xc002101560, 0x1, 0xc00124e920}, {0xc00024e9c0, 0x1, 0x1}, ...)
        /home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:3046 +0x23e
    github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter).addPatchValueToDiff(0xc00124f398, {0xc002061b60, 0x74b89a4310045424, 0x2}, {0x1e014a0, 0xc002a3a3c0}, {0x1e014a0, 0xc00293c3c0}, {0x1e014a0, 0xc0028b6060}, ...)
        /home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:2943 +0x405
    github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter).addPatchMapToDiff(0xc002101640, {0xc002101640, 0x1, 0x1}, 0xc00124ee80, 0x40d0b4, 0x0, 0x7f9011329ab0, 0x0)
        /home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:3015 +0x3f4
    github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter).addPatchValueToDiff(0xc00124f398, {0xc002101640, 0x1e573c0, 0x1}, {0x1f39380, 0xc002cc9980}, {0x1f39380, 0xc002e690e0}, {0x1f39380, 0xc002e68930}, ...)
        /home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:2936 +0x2b0
    github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*patchConverter).addPatchMapToDiff(0x2, {0x0, 0x0, 0x0}, 0x20, 0x1e573c0, 0x10000c001ef8001, 0xffffffffffffffff, 0x0)
        /home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:3015 +0x3f4
    github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.convertPatchToDiff(0xc001f20d20, 0xc002e69020, 0x1f39380, 0xc002cae060, {0xc002061ae0, 0x2, 0x2})
        /home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:2841 +0x105
    github.com/pulumi/pulumi-kubernetes/provider/v3/pkg/provider.(*kubeProvider).Diff(0xc0004cbe00, {0x26238a8, 0xc002ade330}, 0xc00138af50)
        /home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:1554 +0x1cf9
    github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Diff_Handler.func1({0x26238a8, 0xc002ade330}, {0x2151fe0, 0xc00138af50})
        /home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/v3@v3.34.0/proto/go/provider.pb.go:2604 +0x78
    github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1({0x26238a8, 0xc002906660}, {0x2151fe0, 0xc00138af50}, 0xc0007291a0, 0xc0028da4e0)
        /home/runner/go/pkg/mod/github.com/grpc-ecosystem/grpc-opentracing@v0.0.0-20180507213350-8e809c8a8645/go/otgrpc/server.go:57 +0x406
    github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Diff_Handler({0x21f1ee0, 0xc0004cbe00}, {0x26238a8, 0xc002906660}, 0xc002b94300, 0xc000517000)
        /home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/v3@v3.34.0/proto/go/provider.pb.go:2606 +0x138
    google.golang.org/grpc.(*Server).processUnaryRPC(0xc0004dd340, {0x26643f8, 0xc00058bd40}, 0xc001e1b440, 0xc000a2cc30, 0x3a9ac28, 0x0)
        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.45.0/server.go:1282 +0xccf
    google.golang.org/grpc.(*Server).handleStream(0xc0004dd340, {0x26643f8, 0xc00058bd40}, 0xc001e1b440, 0x0)
        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.45.0/server.go:1619 +0xa2a
    google.golang.org/grpc.(*Server).serveStreams.func1.2()
        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.45.0/server.go:921 +0x98
    created by google.golang.org/grpc.(*Server).serveStreams.func1
        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.45.0/server.go:919 +0x294

RossComputerGuy · 2022-07-03T20:53:01Z

If we move to server-side diff, does that also enable not emitting last-applied-configuration? I believe that was largely only emitted to help with client-side diffs. If so - that might help with #1048 and #1118.

Based on a simple test - it looks like the annotation is still being set even with SSA; or something else isn't properly configured/set.

I'm trying to deploy kube-prometheus-stack, but prometheuses CRD fails to be deployed with pulumi complaining on

error: resource prometheuses.monitoring.coreos.com was not successfully created by the Kubernetes API server : CustomResourceDefinition.apiextensions.k8s.io "prometheuses.monitoring.coreos.com" is invalid: metadata.annotations: Too long: must have at most 262144 bytes

I tried deploying it with SSA flag (PULUMI_K8S_ENABLE_DRY_RUN=true / enable_dry_run=True), but running into the same error - now also during preview as well. The code to reproduce:

from pulumi_kubernetes.yaml import ConfigGroup

def remove_status_from_crd(obj):
    if obj["kind"] == "CustomResourceDefinition":
        try:
            del obj["status"]
        except KeyError:
            pass

prometheus_crd = ConfigGroup(
    "prometheus-crd",
    files=[
        "https://raw.githubusercontent.com/prometheus-community/helm-charts/main/charts/kube-prometheus-stack/crds/crd-prometheuses.yaml",  # noqa: E501; pylint: disable=C0301
    ],
    transformations=[
        remove_status_from_crd,
    ],
)

It's Docker Desktop kubernetes:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.3", GitCommit:"ca643a4d1f7bfe34773c74f79527be4afd95bf39", GitTreeState:"clean", BuildDate:"2021-07-15T20:58:09Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.3", GitCommit:"ca643a4d1f7bfe34773c74f79527be4afd95bf39", GitTreeState:"clean", BuildDate:"2021-07-15T20:59:07Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"linux/amd64"}

I'm getting this exact issue with my local development using Minikube except I don't use Python, I use NodeJS. Have you found a solution because this is preventing my stuff from working?

lblackstone · 2022-07-20T00:27:44Z

Update: Server-side diff/apply can now be enabled with the enableServerSideApply mode available in the v3.20.1 release. This is expected to become the default provider behavior as part of the v4.0.0 release.

lblackstone · 2022-10-26T13:06:14Z

SSA mode was enabled in the v3.22.0 release

mortaelth · 2022-11-01T12:09:44Z

is it to be reopened? The default was reverted: #2206

lblackstone · 2022-11-01T19:52:59Z

We rolled back this default in v3.22.1, and have decided to wait for the v4.0.0 release to enable it by default.

lblackstone added kind/enhancement Improvements or new features dry-run-diff Related to dry run diff behavior labels Apr 30, 2021

leezen added the emergent An issue that was added to the current milestone after planning label May 1, 2021

leezen added this to the 0.56 milestone May 1, 2021

lblackstone mentioned this issue May 3, 2021

Resources with long manifests cannot be created due to too long last-applied-configuration annotation #1048

Closed

ghost mentioned this issue May 3, 2021

Updating a configmap's data with enableDryRun fails with "configsmap already exists" #1560

Closed

leezen assigned lblackstone May 4, 2021

lblackstone mentioned this issue May 5, 2021

Kubernetes 1.18+ with server side apply featuregate causes huge CPU usage and execution time. #1372

Closed

leezen modified the milestones: 0.56, 0.57 May 17, 2021

lblackstone mentioned this issue May 21, 2021

Pulumi preview/up --diff for kubernetes resources does not display diffs of properties in the same level of nesting if one references a fully-encrypted resource by its name #1576

Closed

leezen removed this from the 0.57 milestone Jun 8, 2021

infin8x removed the emergent An issue that was added to the current milestone after planning label Jul 6, 2021

infin8x added this to the 0.59 milestone Jul 6, 2021

infin8x added the size/L Estimated effort to complete (up to 10 days). label Jul 6, 2021

infin8x removed this from the 0.59 milestone Jul 6, 2021

lblackstone mentioned this issue Jul 28, 2021

Remove last-applied-configuration annotation for resources managed with server-side diff #1659

Closed

ncsibra mentioned this issue Nov 26, 2021

[kube-prometheus-stack] Can't upgrade CRD from 0.50.0 to 0.52.0: metadata.annotations: Too long: must have at most 262144 bytes prometheus-community/helm-charts#1500

Closed

viveklak mentioned this issue Jan 7, 2022

Elide last applied configuration annotation when SSA is supported #1863

Merged

AaronFriel mentioned this issue Feb 7, 2022

pulumi config --show-secrets should use batch-decrypt #11850 pulumi/pulumi#8677

Open

lblackstone removed the dry-run-diff Related to dry run diff behavior label Jul 27, 2022

lblackstone added the area/server-side-apply label Jul 27, 2022

lblackstone added the resolution/fixed This issue was fixed label Oct 26, 2022

lblackstone closed this as completed Oct 26, 2022

lblackstone reopened this Nov 1, 2022

lblackstone removed the resolution/fixed This issue was fixed label Nov 1, 2022

lblackstone mentioned this issue Nov 14, 2022

[Epic] pulumi-kubernetes 4.0 breaking changes #1788

Closed

5 tasks

kpitzen mentioned this issue Apr 17, 2023

[Epic] Kubernetes 4.0 #2362

Closed

25 tasks

lblackstone mentioned this issue May 12, 2023

Enable Server-Side Apply mode by default #2398

Merged

lblackstone added the resolution/fixed This issue was fixed label May 18, 2023

lblackstone added this to the 0.89 milestone May 18, 2023

lblackstone closed this as completed May 18, 2023

lblackstone added the last-applied-configuration Issues related to the last-applied-configuration annotation label Jul 18, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use server-side diff by default #1556

Use server-side diff by default #1556

lblackstone commented Apr 30, 2021 •

edited

lblackstone commented Apr 30, 2021

lukehoban commented May 2, 2021

lblackstone commented May 3, 2021 •

edited

stanislav-zaprudskiy commented Nov 12, 2021

lblackstone commented Nov 12, 2021

sathieu commented Nov 26, 2021

KaoruDev commented Jun 28, 2022

RossComputerGuy commented Jul 3, 2022

lblackstone commented Jul 20, 2022

lblackstone commented Oct 26, 2022

mortaelth commented Nov 1, 2022

lblackstone commented Nov 1, 2022

Use server-side diff by default #1556

Use server-side diff by default #1556

Comments

lblackstone commented Apr 30, 2021 • edited

lblackstone commented Apr 30, 2021

lukehoban commented May 2, 2021

lblackstone commented May 3, 2021 • edited

stanislav-zaprudskiy commented Nov 12, 2021

lblackstone commented Nov 12, 2021

sathieu commented Nov 26, 2021

KaoruDev commented Jun 28, 2022

RossComputerGuy commented Jul 3, 2022

lblackstone commented Jul 20, 2022

lblackstone commented Oct 26, 2022

mortaelth commented Nov 1, 2022

lblackstone commented Nov 1, 2022

lblackstone commented Apr 30, 2021 •

edited

lblackstone commented May 3, 2021 •

edited