Performance degradation with custom endpoints

[lakkeger]

What happened?

The 6.x versions of the package introduced some serious performance degradation to anyone who works with multiple custom endpoints in their stack config.
See a run comparison below in the example for a simple pulumi preview command between the current latest (v6.5.0) vs v5.41. As I've noticed every single custom endpoint increases the run time by 1.5-4s which in case of the usage of pulumi-local, that generates the endpoints for all AWS services (~250) into the config file for Localstack, results in a significant run time up to 500-800s without even start to work on the actual deploy.
To exclude Localstack as the source of fault from the batch, no request arrived during these runs to the container and I couldn't find any indication in Pulumi's log that it would send any (the option I used was -v 3).

Example

Config:

config:
  aws:accessKey: test
  aws:endpoints:
    - accessanalyzer: http://localhost:4566
    - account: http://localhost:4566
    - acm: http://localhost:4566
    - acmpca: http://localhost:4566
    - amg: http://localhost:4566
    - amp: http://localhost:4566
    - amplify: http://localhost:4566
    - apigateway: http://localhost:4566
    - apigatewayv2: http://localhost:4566
    - appautoscaling: http://localhost:4566
    - appconfig: http://localhost:4566
    - appflow: http://localhost:4566
  # Not adding all endpoints here.
  aws:region: us-east-1
  aws:s3UsePathStyle: "true"
  aws:secretKey: test
  aws:skipCredentialsValidation: "true"
  aws:skipRequestingAccountId: "true"

index.ts:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

const bucket = new aws.s3.Bucket("my-bucket");

export const bucketName = bucket.id;

package.json for 6.5.0:

{
    "name": "s3_bucket_example_ts",
    "main": "index.ts",
    "devDependencies": {
        "@types/node": "^16"
    },
    "dependencies": {
        "@pulumi/pulumi": "3.88.1",
        "@pulumi/aws": "6.5.0"
    }
}

Run output:

$ time pulumi preview
Please choose a stack, or create a new one: dev
Previewing update (dev)

View in Browser (Ctrl+O): https://app.pulumi.com/lakkeger/s3_bucket_example_ts/dev/previews/21bf81a0-39d3-4137-9e98-43a87a53a17b

     Type                 Name                      Plan       
 +   pulumi:pulumi:Stack  s3_bucket_example_ts-dev  create     
 +   └─ aws:s3:Bucket     my-bucket                 create     

Outputs:
    bucketName: output<string>

Resources:
    + 2 to create

pulumi preview  17.98s user 0.43s system 113% cpu 16.212 total

package.json for 5.41.0:

{
    "name": "s3_bucket_example_ts",
    "main": "index.ts",
    "devDependencies": {
        "@types/node": "^16"
    },
    "dependencies": {
        "@pulumi/pulumi": "3.88.1",
        "@pulumi/aws": "5.41.0"
    }
}

Run output:

$ time pulumi preview
Please choose a stack, or create a new one: dev
Previewing update (dev)

View in Browser (Ctrl+O): https://app.pulumi.com/lakkeger/s3_bucket_example_ts/dev/previews/06e0de18-702e-488d-9ce4-81b04aaa189a

     Type                 Name                      Plan       
 +   pulumi:pulumi:Stack  s3_bucket_example_ts-dev  create     
 +   └─ aws:s3:Bucket     my-bucket                 create     

Outputs:
    bucketName: output<string>

Resources:
    + 2 to create

pulumi preview  1.35s user 0.36s system 31% cpu 5.534 total

Output of pulumi about

For 6.5.0:

CLI          
Version      3.88.1
Go Version   go1.21.3
Go Compiler  gc

Plugins
NAME    VERSION
aws     6.5.0
nodejs  unknown

Host     
OS       darwin
Version  13.6
Arch     arm64

This project is written in nodejs: executable='/opt/homebrew/bin/node' version='v20.8.0'

Backend        
Name           pulumi.com
URL            https://app.pulumi.com/lakkeger
User           lakkeger
Organizations  lakkeger
Token type     personal

Dependencies:
NAME            VERSION
@pulumi/aws     6.5.0
@pulumi/pulumi  3.88.1
@types/node     16.18.58

Pulumi locates its logs in /var/folders/8x/dyq5jnvj7md_z7km73gn496m0000gn/T/ by default
warning: Failed to get information about the current stack: No current stack

For 5.41.0:

CLI          
Version      3.88.1
Go Version   go1.21.3
Go Compiler  gc

Plugins
NAME    VERSION
aws     5.41.0
nodejs  unknown

Host     
OS       darwin
Version  13.6
Arch     arm64

This project is written in nodejs: executable='/opt/homebrew/bin/node' version='v20.8.0'

Backend        
Name           pulumi.com
URL            https://app.pulumi.com/lakkeger
User           lakkeger
Organizations  lakkeger
Token type     personal

Dependencies:
NAME            VERSION
@types/node     16.18.58
@pulumi/aws     5.41.0
@pulumi/pulumi  3.88.1

Pulumi locates its logs in /var/folders/8x/dyq5jnvj7md_z7km73gn496m0000gn/T/ by default
warning: Failed to get information about the current stack: No current stack

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[mikhailshilkov]

Thank you for reporting this @lakkeger, I can reproduce the issue locally. We'll take a look at why this happens.

[mjeffryes]

Initial tracing shows that we're spending a long time in CheckConfig; looking at the logs, it looks like we're running MakeTerraformInput for the same inputs over and over again. (There's something like 50 copies of the same logs from that method for the same inputs.) Still need more digging to identify why this is happening.

logs.txt

[mjeffryes]

@t0yv0 traced the CheckConfig latency to the terraform plugin itself, possibly in github.com/hashicorp/terraform-plugin-go; we have merged a workaround that just skips the provider framework validation (since we are also validating with sdkv2).

We should be able to cut a release of the aws provider that incorporates this changes before the end of the week.