Fix dynamicConfigOverrides defaults getting tied to test's lifecycle

[long-nt-tran]

What changed?

Changed onebox.go to set global dynamicConfigOverrides without binding cleanup to a test, and added unit test to verify behavior works -- reused test gets the same default dynamicConfigOverrides.

Also renamed the existing function overrideDynamicConfig -> overrideTestLevelDynamicConfig to make it clearer that that function's dynamic config override is bounded to the test's lifecycle.

Why?

When onebox.go sets up a new Temporal cluster, it applies some default overrides that is not set by the test.

However, these are set via overrideDynamicConfig(...), which ties the cleanup of these configs to a specific test's lifecycle.

As a result, if we have Test1 that gets a fresh cluster, if some Test2 later reuses this cluster after Test1 ends, Test2 will not have these default dynamicConfigOverrides -- in fact the dynamicConfig would be empty because we cleaned it up after Test1 finished.

These defaults should not be bounded to the lifetime of the test that instantiated the fresh cluster.

Note

Existing tests that instantiate their env with testcore.WithDynamicConfigOverrides(...) options already get a fresh cluster, so this bug impacted tests that don't override any dynamic config options when calling testcore.NewEnv(...).

How did you test it?

• built
• run locally and tested manually
• covered by existing tests
• added new unit test(s)
• added new functional test(s)

I added a test_cluster_pool_test.go which deterministically repros this without the code change in onebox.go:

Expected failure without change in onebox.go:

$ go test ./tests/testcore/ -run TestDedicatedClusterReuseDropsGlobalOverrides -count=1
--- FAIL: TestDedicatedClusterReuseDropsGlobalOverrides (0.05s)
    --- FAIL: TestDedicatedClusterReuseDropsGlobalOverrides/SecondUse (0.00s)
        test_cluster_pool_test.go:42:
                Error Trace:    /Users/longtran/work/forks/temporal-wt/testcore-dedicated-cluster/tests/testcore/test_cluster_pool_test.go:57
                                                        /Users/longtran/work/forks/temporal-wt/testcore-dedicated-cluster/tests/testcore/test_cluster_pool_test.go:42
                Error:          Should NOT be empty, but was []
                Test:           TestDedicatedClusterReuseDropsGlobalOverrides/SecondUse
                Messages:       global override defaults should still be present on reused cluster: key frontend.maxconcurrentbatchoperationpernamespace missing
FAIL
FAIL    go.temporal.io/server/tests/testcore    1.175s
FAIL

With change, it passes:

$ go test ./tests/testcore/ -run TestDedicatedClusterReuseDropsGlobalOverrides -count=1
ok      go.temporal.io/server/tests/testcore    1.086s

Potential risks

Might be breaking changes if some tests rely on the existing bug for some reason (?)

[stephanos]

Awesome - thank you for tracking that down!