Slow operations with SSA enabled

[hagaibarel]

What happened?

We've been seeing very slow operation when SSA is enabled (via config kubernetes:enableServerSideApply: true) compared to CSA. For example, a preview operation that usually take a couple of seconds (without any resource changes) now takes 3-4 minutes.

We're using google storage as the backend and stack has ~100 resources

Expected Behavior

Reasonable times when running up/preview

Steps to reproduce

Enable SSA on a stack and run any operation compared to the same stack without it enabled

Output of pulumi about

CLI
Version 3.68.0
Go Version go1.20.4
Go Compiler gc

Plugins
NAME VERSION
go unknown
kubernetes 3.28.0

Host
OS ubuntu
Version 23.04
Arch x86_64

This project is written in go: executable='/usr/local/go/bin/go' version='go version go1.20.4 linux/amd64'

Current Stack: stg2

(stack and backend details redacted)

Dependencies:
NAME VERSION
github.com/bringg/pulib 0.10.0
github.com/creasty/defaults 1.7.0
github.com/onsi/ginkgo/v2 2.9.5
github.com/onsi/gomega 1.27.7
github.com/pulumi/pulumi-kubernetes/sdk/v3 3.28.0
github.com/pulumi/pulumi/sdk/v3 3.68.0

Pulumi locates its logs in /tmp by default

Additional context

We've run a trace and we can see the RegisterResource takes most of the time, see the attached trace for reference
stg2-up.trace.gz

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[lblackstone]

Thanks for the report @hagaibarel. Does this only affect stacks with more resources, or are you noticing a difference across the board when SSA is enabled?

[hagaibarel]

I would say across the board, the behavior describe here was observed in one of our smaller projects (~100 resources), in bigger projects is takes even longer

[lblackstone]

I was able to reproduce the slowness and I think I know why this is happening.

I used the following program to test:

import * as k8s from "@pulumi/kubernetes";

const provider = new k8s.Provider("k8s", {
    enableServerSideApply: true, // Also tried with false
    enableConfigMapMutable: true,
});

for (let i = 0; i < 500; i = i + 1) {
    new k8s.core.v1.ConfigMap(`a${i}`, {
        data: {foo: "baz"}
    }, {
        provider,
    });
}

new k8s.core.v1.ConfigMap("b", {
    data: {foo: "baz"} // Change this value after initial creation
}, {
    provider,
});

The initial creation took approximately the same length of time for both CSA and SSA, but SSA took an order of magnitude longer to update a single resource after that (4 seconds vs 31 seconds).

I'm doing some more testing to confirm, but I think that the SSA diff is doing a networked diff operation per resource in the stack, while the CSA diff is checking against local state instead.

[lblackstone]

Fixed in #2445