proposal: testing: add Testing constant

[FiloSottile]

In cryptographic packages, I would like to perform extra consistency checks that are unsuitable for production either for performance reasons (they might be slow or cause inliner or escape analysis issues) or because they are not safe (they might be non-constant time or expose undesirable attack surface). In other languages, this is usually done with debug mode asserts, but really they are mostly exercised during tests.

I propose we add a constant to testing that is always true if building with go test and false otherwise. I couldn't come up with a better name than testing.Testing.

Then, I could gate all my testing assertions with if testing.Testing and (depending on the order of optimization passes), count on dead code removal to make the effects of that code disappear in production builds. This feels like a technique that would make sense beyond cryptography, too, although for other packages it might be less important for it to be a constant, so it can be emulated with init() in a _test.go file setting a variable.

[seankhliao]

There's this on tip https://pkg.go.dev/testing@master#Testing

[seankhliao]

dup of #52600 ?

[FiloSottile]

Oh, I didn't notice, but unfortunately that API doesn't work for the performance optimizations I'm hoping to get: first, it's implemented as a link-time variable, so the compiler can't act based on it at all; second, as a function it requires inlining to happen before code elimination, and I haven't tested it but it's unlikely that will then inform the inliner weight of the caller.

The first part can be fixed but the second might be a permanent problem if we commit to a function in Go 1.21.

[seankhliao]

release blocker for new api?
cc @ianlancetaylor

[mvdan]

Note that the release-blocker label likely means little unless this is milestoned for 1.21. cc @golang/release

[FiloSottile]

I'm not sure it's worth changing at the last minute, but it might be worth making an explicit decision, yeah.

To test I compiled this program

package main

import "testing"

func main() {
	f()
}

func f() {
	b := make([]byte, 32)
	if testing.Testing() {
		escape(b)
		println("elided")
	}
	println("yes")
}

var sink []byte

func escape(b []byte) {
	sink = b
}

where testing.Testing is

func Testing() bool {
	return false
}

The results were mixed but mostly optimization-averse:

• the escape() and println("elided") calls are not emitted
• the "elided" string is present in RODATA
• b escapes to the heap
• the inliner prices the if body

While if we made it const Testing = false:

• neither the calls nor the "elided" string are emitted
• make([]byte, 32) does not escape
• can inline f with cost 8 as: func() { b := make([]byte, 32); if testing.Testing { }; println("yes") }

[ianlancetaylor]

Making testing.Testing a compile-time constant seems infeasible. That would mean that for go test we would have to recompile everything that imports the "testing" package.

[mvdan]

I initially had the same reaction as @ianlancetaylor - today, if I test package foo via external tests in package foo_test, running go build followed by go test means that I only build foo and its dependencies once. This change could make some of my daily routine noticeably slower to start running the tests.

Worth noting that many people write internal tests at package foo, meaning that go build followed by go test does need to build foo twice. That's partly why #29258 was filed. But at least one can avoid the double-compile by only using external tests.