cmd/compile: short tests are not short enough

[rsc]

Running all.bash (which does go test -short std cmd), my target is usually that individual tests should run in under 1 second.

These tests take quite a bit longer:

ok  	cmd/compile	17.195s
ok  	cmd/compile/internal/gc	24.281s

Can these be dialed back, either actually shortened or just have parts skipped when testing.Short() is true?

/cc @aclements @griesemer @randall77

[randall77]

cmd/compile has only one test:

--- PASS: TestFormats (3.77s)

we could maybe run this in non-short mode only. @griesemer

Slow compiler tests (>.1 sec):

--- PASS: TestGenFlowGraph (4.02s)
--- PASS: TestArithmeticConst (0.42s)
--- PASS: TestArithmetic (0.41s)
--- PASS: TestTypeAssertion (0.40s)
--- PASS: TestCopy (0.40s)
--- PASS: TestCompound (0.40s)
--- PASS: TestArithmeticBoundary (0.40s)
--- PASS: TestPhi (0.39s)
--- PASS: TestLoadStore (0.39s)
--- PASS: TestComparisonsConst (0.38s)
--- PASS: TestMap (0.36s)
--- PASS: TestAppend (0.35s)
--- PASS: TestZero (0.34s)
--- PASS: TestFPSoftFloat (0.34s)
--- PASS: TestUnsafe (0.33s)
--- PASS: TestFP (0.33s)
--- PASS: TestChan (0.33s)
--- PASS: TestAddressed (0.33s)
--- PASS: TestDuplicateLoad (0.32s)
--- PASS: TestBuiltin (0.32s)
--- PASS: TestNamedReturn (0.31s)
--- PASS: TestClosure (0.31s)
--- PASS: TestSqrt (0.30s)
--- PASS: TestSlice (0.29s)
--- PASS: TestScanfRemoval (0.28s)
--- PASS: TestDashS (0.26s)
--- PASS: TestString (0.25s)
--- PASS: TestCtl (0.25s)
--- PASS: TestShortCircuit (0.22s)
--- PASS: TestBreakContinue (0.21s)
--- PASS: TestArray (0.19s)
--- PASS: TestRegalloc (0.17s)
--- PASS: TestScopeRanges (0.16s)
--- PASS: TestEmptyDwarfRanges (0.14s)

The first one we should fix @dr2chase .
All the ones around 0.30 sec just invoke the compiler and run the result. Some of the files are largeish, but most aren't (map.go for TestMap is 45 lines, for example). Maybe there's some way we could batch all these tests into a single compile to amortize the cost of invoking the compiler (&linker?).

[dr2chase]

TestGenFlowGraph generates 766 functions, compiles that, and runs it. For each function the runner first interprets it up to 4096 steps (it depends on the function, most are 512) and if that terminates, calls the function and compares the results.

One way to speed it up would be to quit testing it for GO_SSA_PHI_LOC_CUTOFF=0, because it turns out that we ignore that parameter now.

	runGenTest(t, "flowgraph_generator1.go", "ssa_fg_tmp1")
	if runtime.GOOS != "windows" {
		runGenTest(t, "flowgraph_generator1.go", "ssa_fg_tmp2", "GO_SSA_PHI_LOC_CUTOFF=0")
	}

That cuts the test time in half with no loss of information....

[gopherbot]

Change https://golang.org/cl/125075 mentions this issue: cmd/compile: don't run gc/ssa_test/TestGenFlowGraph in short mode

[griesemer]

The TestFormats test could be run in non-short mode only as long as we run it from time to time. It does verify that the various format strings used in compiler errors use the correct format strings, etc.

[randall77]

@griesemer: we have a longtest builder which will run this test on every commit.

[gopherbot]

Change https://golang.org/cl/125037 mentions this issue: cmd/compile: run fmt test only in long mode

[ysmolski]

Should we put a general recommendation about wether test falls under short category somewhere. Is threshold on 1 second or 0.1 seconds?

[alexbrainman]

: we have a longtest builder which will run this test on every commit.

@randall77 are you sure? Do you have windows longtest builder? Most changes to Go repo and tools are developed on non-Windows computers. So Go Team developers do not see Windows problems that their changes introduce.

Alex

[randall77]

Yes, we run the longtest builder on every commit. The longtest builder is linux/amd64, there's no windows one (or arm, or ...).
The TestFormats test isn't architecture-dependent (except that it doesn't (or may not? not sure) test files which are excluded on linux/amd64).