crypto/x509: TestPlatformVerifier failures on macOS due to mismatched error text

[gopherbot]

#!watchflakes
post <- pkg == "crypto/x509" && test ~ `TestPlatformVerifier` && `x509: “valid.testing.golang.invalid” certificate is not trusted`

Issue created automatically to collect these failures.

Example (log):

=== RUN   TestPlatformVerifier/non-nested_KU
=== PAUSE TestPlatformVerifier/non-nested_KU
=== CONT  TestPlatformVerifier/non-nested_KU
    platform_test.go:245: unexpected verification error: got "x509: “valid.testing.golang.invalid” certificate is not trusted", want "x509: “valid.testing.golang.invalid” certificate is not permitted for this usage"
--- FAIL: TestPlatformVerifier/non-nested_KU (0.00s)

— watchflakes

[gopherbot]

Found new dashboard test flakes for:

#!watchflakes
default <- pkg == "crypto/x509" && test == "TestPlatformVerifier"

2024-02-01 21:42 gotip-darwin-amd64_14 go@c9d88ea2 crypto/x509.TestPlatformVerifier (log)

=== RUN   TestPlatformVerifier/non-nested_KU
=== PAUSE TestPlatformVerifier/non-nested_KU
=== CONT  TestPlatformVerifier/non-nested_KU
    platform_test.go:245: unexpected verification error: got "x509: “valid.testing.golang.invalid” certificate is not trusted", want "x509: “valid.testing.golang.invalid” certificate is not permitted for this usage"
--- FAIL: TestPlatformVerifier/non-nested_KU (0.00s)

2024-02-01 22:57 gotip-darwin-amd64-nocgo go@6d3c1ce8 crypto/x509.TestPlatformVerifier (log)

=== RUN   TestPlatformVerifier/non-nested_KU
=== PAUSE TestPlatformVerifier/non-nested_KU
=== CONT  TestPlatformVerifier/non-nested_KU
    platform_test.go:245: unexpected verification error: got "x509: “valid.testing.golang.invalid” certificate is not trusted", want "x509: “valid.testing.golang.invalid” certificate is not permitted for this usage"
--- FAIL: TestPlatformVerifier/non-nested_KU (0.00s)

2024-02-01 22:59 gotip-darwin-amd64-nocgo go@117164f9 crypto/x509.TestPlatformVerifier (log)

=== RUN   TestPlatformVerifier/non-nested_KU
=== PAUSE TestPlatformVerifier/non-nested_KU
=== CONT  TestPlatformVerifier/non-nested_KU
    platform_test.go:245: unexpected verification error: got "x509: “valid.testing.golang.invalid” certificate is not trusted", want "x509: “valid.testing.golang.invalid” certificate is not permitted for this usage"
--- FAIL: TestPlatformVerifier/non-nested_KU (0.04s)

— watchflakes

[gopherbot]

Found new dashboard test flakes for:

#!watchflakes
default <- pkg == "crypto/x509" && test == "TestPlatformVerifier"

2024-02-02 18:20 gotip-darwin-amd64-nocgo go@6b0309ce crypto/x509.TestPlatformVerifier (log)

=== RUN   TestPlatformVerifier/non-nested_KU
=== PAUSE TestPlatformVerifier/non-nested_KU
=== CONT  TestPlatformVerifier/non-nested_KU
    platform_test.go:245: unexpected verification error: got "x509: “valid.testing.golang.invalid” certificate is not trusted", want "x509: “valid.testing.golang.invalid” certificate is not permitted for this usage"
--- FAIL: TestPlatformVerifier/non-nested_KU (0.01s)

— watchflakes

[bcmills]

(attn @rolandshoemaker)

[rolandshoemaker]

I took a quick look at this, and I have zero clue how this could be flakey. The pattern of failures don't provide any particularly useful insight, the error returned is completely useless and I'm unable to repro locally.

Will think about this a little more, but for now I'm stumped.

[bcmills]

Huh. I wonder if someone was changing the LUCI builder configuration during these runs? (Maybe someone on @golang/release would have some insight?)

[dmitshur]

@prattmic might know if there've been changes to macOS builders on LUCI that would be relevant to this.

Something to check might be whether the builders running into this all have the root from #52108 added. But given the test is running (and not skipped with "test root is not in trust store"), I think it must be.

[prattmic]

Every failing instance linked above was on the same bot, darwin-amd64-14--c1f58526-d1c4-4792-bef1-06d6c16b49e4.golang.cc.macservice.goog. I imagine that is related, but it is still odd because all of the bots use identical macOS images.

[rolandshoemaker]

Bit flip somewhere in the root store? 🤷

[prattmic]

I pulled up the test history on this bot. The bot has been running since 2024-01-09, and the test passed 100% of the time until a run starting 2024-02-01 11:30:06.033 America/Los_Angeles (https://cr-buildbucket.appspot.com/build/8757267625331049905). It has been failing 100% of the time since then.

[prattmic]

I've also verified that this is the only bot that has had any failures.