[codegen/go] pulumiTypes.go is too large to compile

[pierskarsenbarg]

Hello!

• Vote on this issue by adding a 👍 reaction
• To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already)

Issue details

Originally filed on pulumi-aws with title: New WAFv2 rule (in go) causes CPU to spike and doesn't run
When trying to create a new wafv2.rulegroup in Go, the CPU shoots up to nearly max and won't even show a preview.

❯ go version
go version go1.17.5 darwin/amd64

Steps to reproduce

The following code causes this issue to happen:

package main

import (
	"github.com/pulumi/pulumi-aws/sdk/v4/go/aws/wafv2"
	"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
	pulumi.Run(func(ctx *pulumi.Context) error {
		_, err := wafv2.NewRuleGroup(ctx, "rule-group", &wafv2.RuleGroupArgs{
			Capacity: pulumi.Int(713),
			Scope:    pulumi.String("REGIONAL"),
			VisibilityConfig: &wafv2.RuleGroupVisibilityConfigArgs{
				CloudwatchMetricsEnabled: pulumi.Bool(true),
				MetricName:               pulumi.String("waf-rules"),
				SampledRequestsEnabled:   pulumi.Bool(true),
			},
		})

		if err != nil {
			return err
		}

		return nil
	})
}

a similar program in typescript doesn't have the same outcome (the preview shows and and the resource is created):

import * as aws from "@pulumi/aws";

const rule = new aws.wafv2.RuleGroup("rule-group", {
    capacity: 70,
    scope: "REGIONAL",
    visibilityConfig: {
        cloudwatchMetricsEnabled: true,
        metricName: "test-metric",
        sampledRequestsEnabled: true
    }
})

Expected: New rule group is created
Actual: new rule group fails to be created and the program just hangs. It doesn't even show a preview

[pierskarsenbarg]

Output from verbose debugger: out.txt

[Graham-Beer]

Also getting the same issue with Go Version 1.16.12. Just takes all memory and freezes. This is causing an issue building a production system.

[Graham-Beer]

Is there a chance this could be looked at? Causing me a headache for a production build.

[Graham-Beer]

Output from verbose logging (pulumi up --logtostderr -v=9 2> out.txt), from https://www.pulumi.com/docs/troubleshooting/#verbose-logging

This is being run with go version 1.16.

I0128 11:30:21.376753 32552 backend.go:414] found username for access token
I0128 11:30:22.580922 32552 util.go:561] errors detecting git metadata: 1 error occurred:
* detecting Git remote URL: could not read origin information: remote not found

I0128 11:30:22.584718 32552 backend.go:414] found username for access token
I0128 11:30:22.818288 32552 ignore.go:44] Explicitly ignoring and discarding error: could not read origin information: remote not found
I0128 11:30:23.058584 32552 backend.go:414] found username for access token
I0128 11:30:23.058584 32552 backend.go:924] Stack production being updated to version 7
I0128 11:30:23.467584 32552 update.go:184] *** Starting Update(preview=true) ***
I0128 11:30:23.468796 32552 plugins.go:76] gatherPluginsFromProgram(): gathering plugins from language host
I0128 11:30:23.478590 32552 plugins.go:628] GetPluginPath(language, go, ): found on $PATH C:\pulumi\bin\pulumi-language-go.exe
I0128 11:30:23.478590 32552 plugin.go:126] Launching plugin 'go' from 'C:\pulumi\bin\pulumi-language-go.exe' with args: -root=C:\Projects\GoAnywhere_11_waf,127.0.0.1:59320
I0128 11:30:23.718762 32552 langruntime_plugin.go:207] langhost[go].GetPluginInfo() executing
I0128 11:30:23.720483 32552 langruntime_plugin.go:115] langhost[go].GetRequiredPlugins(proj=GoAnywhere_11_waf,pwd=C:\Projects\GoAnywhere_11_waf,program=.) executing
I0128 11:30:24.630371 32552 langruntime_plugin.go:157] langhost[go].GetRequiredPlugins(proj=GoAnywhere_11_waf,pwd=C:\Projects\GoAnywhere_11_waf,program=.) success: #versions=1
I0128 11:30:24.630371 32552 plugins.go:88] gatherPluginsFromProgram(): plugin go () is required by language host
I0128 11:30:24.630371 32552 plugins.go:88] gatherPluginsFromProgram(): plugin aws 4.21.1 () is required by language host
I0128 11:30:24.630371 32552 plugins.go:100] gatherPluginsFromSnapshot(): gathering plugins from snapshot
I0128 11:30:24.630371 32552 plugins.go:137] ensurePluginsAreInstalled(): beginning
I0128 11:30:24.634484 32552 plugins.go:628] GetPluginPath(language, go, ): found on $PATH C:\pulumi\bin\pulumi-language-go.exe
I0128 11:30:24.634484 32552 plugins.go:142] ensurePluginsAreInstalled(): plugin go already installed
I0128 11:30:24.657425 32552 plugins.go:668] GetPluginPath(resource, aws, 4.21.1): enabling new plugin behavior
I0128 11:30:24.657425 32552 plugins.go:750] SelectCompatiblePlugin(..., aws): beginning
I0128 11:30:24.657425 32552 plugins.go:783] SelectCompatiblePlugin(..., aws): skipping plugin aws-1.31.0: semver mismatch
I0128 11:30:24.657425 32552 plugins.go:783] SelectCompatiblePlugin(..., aws): skipping plugin aws-2.13.1: semver mismatch
I0128 11:30:24.657425 32552 plugins.go:783] SelectCompatiblePlugin(..., aws): skipping plugin aws-4.0.0: semver mismatch
I0128 11:30:24.657425 32552 plugins.go:783] SelectCompatiblePlugin(..., aws): skipping plugin aws-4.1.0: semver mismatch
I0128 11:30:24.657425 32552 plugins.go:783] SelectCompatiblePlugin(..., aws): skipping plugin aws-4.3.0: semver mismatch
I0128 11:30:24.657425 32552 plugins.go:783] SelectCompatiblePlugin(..., aws): skipping plugin aws-4.5.1: semver mismatch
I0128 11:30:24.657425 32552 plugins.go:783] SelectCompatiblePlugin(..., aws): skipping plugin aws-4.7.0: semver mismatch
I0128 11:30:24.657425 32552 plugins.go:783] SelectCompatiblePlugin(..., aws): skipping plugin aws-4.8.0: semver mismatch
I0128 11:30:24.657425 32552 plugins.go:783] SelectCompatiblePlugin(..., aws): skipping plugin aws-4.10.0: semver mismatch
I0128 11:30:24.657425 32552 plugins.go:783] SelectCompatiblePlugin(..., aws): skipping plugin aws-4.11.0: semver mismatch
I0128 11:30:24.657425 32552 plugins.go:783] SelectCompatiblePlugin(..., aws): skipping plugin aws-4.12.0: semver mismatch
I0128 11:30:24.657425 32552 plugins.go:783] SelectCompatiblePlugin(..., aws): skipping plugin aws-4.13.0: semver mismatch
I0128 11:30:24.657425 32552 plugins.go:783] SelectCompatiblePlugin(..., aws): skipping plugin aws-4.14.0: semver mismatch
I0128 11:30:24.657425 32552 plugins.go:783] SelectCompatiblePlugin(..., aws): skipping plugin aws-4.15.0: semver mismatch
I0128 11:30:24.657425 32552 plugins.go:783] SelectCompatiblePlugin(..., aws): skipping plugin aws-4.16.0: semver mismatch
I0128 11:30:24.657425 32552 plugins.go:783] SelectCompatiblePlugin(..., aws): skipping plugin aws-4.18.0: semver mismatch
I0128 11:30:24.657425 32552 plugins.go:779] SelectCompatiblePlugin(..., aws): best plugin aws-4.21.1: semver match
I0128 11:30:24.657425 32552 plugins.go:783] SelectCompatiblePlugin(..., aws): skipping plugin aws-4.23.0: semver mismatch
I0128 11:30:24.657425 32552 plugins.go:783] SelectCompatiblePlugin(..., aws): skipping plugin aws-4.24.1: semver mismatch
I0128 11:30:24.657425 32552 plugins.go:783] SelectCompatiblePlugin(..., aws): skipping plugin aws-4.25.0: semver mismatch
I0128 11:30:24.657425 32552 plugins.go:783] SelectCompatiblePlugin(..., aws): skipping plugin aws-4.29.0: semver mismatch
I0128 11:30:24.657425 32552 plugins.go:783] SelectCompatiblePlugin(..., aws): skipping plugin aws-4.30.0: semver mismatch
I0128 11:30:24.657425 32552 plugins.go:783] SelectCompatiblePlugin(..., aws): skipping plugin aws-4.33.0: semver mismatch
I0128 11:30:24.657425 32552 plugins.go:792] SelectCompatiblePlugin(..., aws): selecting plugin 'aws-4.21.1': best match
I0128 11:30:24.657425 32552 plugins.go:715] GetPluginPath(resource, aws, 4.21.1): found in cache at C:\Users\gb73800.pulumi\plugins\resource-aws-v4.21.1\pulumi-resource-aws.exe
I0128 11:30:24.657425 32552 plugins.go:142] ensurePluginsAreInstalled(): plugin aws 4.21.1 already installed
I0128 11:30:24.657425 32552 plugins.go:157] ensurePluginsAreInstalled(): completed
I0128 11:30:24.657425 32552 plugins.go:246] computeDefaultProviderPlugins(): considering go
I0128 11:30:24.657425 32552 plugins.go:249] computeDefaultProviderPlugins(): skipping go, not a resource provider
I0128 11:30:24.657425 32552 plugins.go:246] computeDefaultProviderPlugins(): considering aws-4.21.1
I0128 11:30:24.657425 32552 plugins.go:277] computeDefaultProviderPlugins(): plugin aws-4.21.1 selected for package aws (first seen)
I0128 11:30:24.657425 32552 plugins.go:283] computeDefaultProviderPlugins(): summary of default plugins:
I0128 11:30:24.657425 32552 plugins.go:285] aws = 4.21.1
I0128 11:30:24.663049 32552 step_generator.go:900] stepGenerator.GeneratePendingDeletes(): scanning previous snapshot for pending deletes
I0128 11:30:24.663049 32552 deployment_executor.go:400] deploymentExecutor.retirePendingDeletes(...): no pending deletions
I0128 11:30:24.663049 32552 langruntime_plugin.go:168] langhost[go].Run(pwd=C:\Projects\GoAnywhere_11_waf,program=.,#args=0,proj=GoAnywhere_11_waf,stack=production,#config=3,dryrun=true) executing
I0128 11:30:24.663049 32552 deployment_executor.go:221] deploymentExecutor.Execute(...): waiting for incoming events
I0128 11:30:24.663049 32552 step_executor.go:326] StepExecutor worker(-2): worker coming online
I0128 11:30:24.663049 32552 step_executor.go:326] StepExecutor worker(-2): worker waiting for incoming chains

[stack72]

The issue here is that the schema is highly nested due to the schema that we are ingesting from the TF provider upstream. We would need to rebuild that part of the schema completely and diverge from the upstream provider - this is a major breaking change and not something we can commit to easily

[joeduffy]

Interestingly, if you just try to compile the wafv2 pulumiTypes.go file with Go directly, it hangs indefinitely. It's a giant file (678KLOCs!!!), so maybe that's the simple answer, that Go is just going to chug away indefinitely here, but I have to wonder if there's some kind of workaround at the Go level that would allow for a successful compile.

pulumi/pulumi-aws/sdk/go/aws/wafv2$ go build -x ./pulumiTypes.go
λ go build -x ./pulumiTypes.go
WORK=/var/folders/l2/vv4rcyx972x1mml3ltxfw1fm0000gn/T/go-build179084767
mkdir -p $WORK/b001/
cat >$WORK/b001/importcfg << 'EOF' # internal
# import config
packagefile context=/usr/local/go/pkg/darwin_amd64/context.a
packagefile github.com/pulumi/pulumi/sdk/v3/go/pulumi=/Users/joeduffy/Library/Caches/go-build/21/2190a8526d7221b752f863c5d52a104ea904407ec1c4fd424ddb948511c72a44-d
packagefile reflect=/usr/local/go/pkg/darwin_amd64/reflect.a
EOF
cd /Users/joeduffy/dev/code/src/github.com/pulumi/pulumi-aws/sdk/go/aws/wafv2
/usr/local/go/pkg/tool/darwin_amd64/compile -o $WORK/b001/_pkg_.a -trimpath "$WORK/b001=>" -p command-line-arguments -lang=go1.16 -complete -buildid viOndUgGembTsJ1BBtCR/viOndUgGembTsJ1BBtCR -goversion go1.17.4 -D _/Users/joeduffy/dev/code/src/github.com/pulumi/pulumi-aws/sdk/go/aws/wafv2 -importcfg $WORK/b001/importcfg -pack -c=4 ./pulumiTypes.go
... never returns ...

[joeduffy]

For what it's worth, I ran this for a bit longer, and it did actually return:

real	5m39.656s
user	7m36.562s
sys	3m2.775s

That's just pulumiTypes.go; the entire directory also completes, but takes a bit longer:

real	6m10.177s
user	7m39.509s
sys	3m34.597s

[adomoshe]

Any update on this?

[mikhailshilkov]

This will be fixed by pulumi/pulumi-aws#2589 in 6.0 of the provider