crypto/x509: TestSystemVerify consistently failing

[bcmills]

#!watchflakes
post <- goos == "windows" && pkg == "crypto/x509" && test == "TestSystemVerify"

https://build.golang.org/log/97b5cac923dc9079392efd9637b9fffa8b9f7592:

--- FAIL: TestSystemVerify (1.59s)
    --- FAIL: TestSystemVerify/Valid (0.10s)
        verify_test.go:479: unexpected error: x509: certificate signed by unknown authority
    --- FAIL: TestSystemVerify/MixedCase (0.00s)
        verify_test.go:479: unexpected error: x509: certificate signed by unknown authority
    --- FAIL: TestSystemVerify/HostnameMismatch (0.00s)
        verify_test.go:378: error was not a HostnameError: x509: certificate signed by unknown authority
    --- FAIL: TestSystemVerify/IPMissing (0.00s)
        verify_test.go:378: error was not a HostnameError: x509: certificate signed by unknown authority
    --- FAIL: TestSystemVerify/Expired (0.00s)
        verify_test.go:388: error was not Expired: x509: certificate signed by unknown authority
    --- FAIL: TestSystemVerify/RootInIntermediates (0.00s)
        verify_test.go:479: unexpected error: x509: certificate signed by unknown authority
    --- FAIL: TestSystemVerify/SHA-384 (0.02s)
        verify_test.go:514: Got 0 matches instead of 1 for expected chain [api.moip.com.br COMODO RSA Extended Validation Secure Server CA COMODO RSA Certification Authority AddTrust External CA Root]
FAIL
FAIL	crypto/x509	2.845s

(attn @golang/security @dmitshur)

[bcmills]

I think this is something to do with an expiration — all of the builders started failing at once, even on TryBots at older base CLs (https://storage.googleapis.com/go-build-log/cd3d6904/windows-386-2016_967987aa.log).

[gopherbot]

Change https://go.dev/cl/472295 mentions this issue: crypto/x509: fix system root tests + darwin intermediate handling

[gopherbot]

Change https://go.dev/cl/472097 mentions this issue: crypto/x509: avoid nil dereference in tests

[gopherbot]

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "windows" && pkg == "crypto/x509" && test == "TestSystemVerify"

2023-02-28 19:35 windows-amd64-2016 go@4a9c84d0 crypto/x509.TestSystemVerify (log)

--- FAIL: TestSystemVerify (0.44s)
    --- FAIL: TestSystemVerify/Valid (0.13s)
        verify_test.go:479: unexpected error: x509: certificate signed by unknown authority
    --- FAIL: TestSystemVerify/MixedCase (0.00s)
        verify_test.go:479: unexpected error: x509: certificate signed by unknown authority
    --- FAIL: TestSystemVerify/HostnameMismatch (0.00s)
        verify_test.go:378: error was not a HostnameError: x509: certificate signed by unknown authority
    --- FAIL: TestSystemVerify/IPMissing (0.00s)
        verify_test.go:378: error was not a HostnameError: x509: certificate signed by unknown authority
    --- FAIL: TestSystemVerify/Expired (0.00s)
        verify_test.go:388: error was not Expired: x509: certificate signed by unknown authority
    --- FAIL: TestSystemVerify/RootInIntermediates (0.00s)
        verify_test.go:479: unexpected error: x509: certificate signed by unknown authority
    --- FAIL: TestSystemVerify/SHA-384 (0.02s)
        verify_test.go:514: Got 0 matches instead of 1 for expected chain [api.moip.com.br COMODO RSA Extended Validation Secure Server CA COMODO RSA Certification Authority AddTrust External CA Root]

2023-02-28 19:35 windows-arm64-11 go@4a9c84d0 crypto/x509.TestSystemVerify (log)

--- FAIL: TestSystemVerify (0.04s)
    --- FAIL: TestSystemVerify/Valid (0.02s)
        verify_test.go:479: unexpected error: x509: certificate signed by unknown authority
    --- FAIL: TestSystemVerify/MixedCase (0.00s)
        verify_test.go:479: unexpected error: x509: certificate signed by unknown authority
    --- FAIL: TestSystemVerify/HostnameMismatch (0.00s)
        verify_test.go:378: error was not a HostnameError: x509: certificate signed by unknown authority
    --- FAIL: TestSystemVerify/IPMissing (0.00s)
        verify_test.go:378: error was not a HostnameError: x509: certificate signed by unknown authority
    --- FAIL: TestSystemVerify/Expired (0.00s)
        verify_test.go:388: error was not Expired: x509: certificate signed by unknown authority
    --- FAIL: TestSystemVerify/RootInIntermediates (0.00s)
        verify_test.go:479: unexpected error: x509: certificate signed by unknown authority
    --- FAIL: TestSystemVerify/SHA-384 (0.00s)
        verify_test.go:514: Got 0 matches instead of 1 for expected chain [api.moip.com.br COMODO RSA Extended Validation Secure Server CA COMODO RSA Certification Authority AddTrust External CA Root]

2023-02-28 21:59 windows-amd64-race go@ec26277a crypto/x509.TestSystemVerify (log)

--- FAIL: TestSystemVerify (0.43s)
    --- FAIL: TestSystemVerify/Valid (0.12s)
        verify_test.go:479: unexpected error: x509: certificate signed by unknown authority
    --- FAIL: TestSystemVerify/MixedCase (0.00s)
        verify_test.go:479: unexpected error: x509: certificate signed by unknown authority
    --- FAIL: TestSystemVerify/HostnameMismatch (0.00s)
        verify_test.go:378: error was not a HostnameError: x509: certificate signed by unknown authority
    --- FAIL: TestSystemVerify/IPMissing (0.00s)
        verify_test.go:378: error was not a HostnameError: x509: certificate signed by unknown authority
    --- FAIL: TestSystemVerify/Expired (0.00s)
        verify_test.go:388: error was not Expired: x509: certificate signed by unknown authority
    --- FAIL: TestSystemVerify/RootInIntermediates (0.00s)
        verify_test.go:479: unexpected error: x509: certificate signed by unknown authority
    --- FAIL: TestSystemVerify/SHA-384 (0.02s)
        verify_test.go:514: Got 0 matches instead of 1 for expected chain [api.moip.com.br COMODO RSA Extended Validation Secure Server CA COMODO RSA Certification Authority AddTrust External CA Root]

2023-02-28 22:18 windows-amd64-2016 go@65fa9ce8 crypto/x509.TestSystemVerify (log)

--- FAIL: TestSystemVerify (0.52s)
    --- FAIL: TestSystemVerify/Valid (0.11s)
        verify_test.go:479: unexpected error: x509: certificate signed by unknown authority
    --- FAIL: TestSystemVerify/MixedCase (0.00s)
        verify_test.go:479: unexpected error: x509: certificate signed by unknown authority
    --- FAIL: TestSystemVerify/HostnameMismatch (0.00s)
        verify_test.go:378: error was not a HostnameError: x509: certificate signed by unknown authority
    --- FAIL: TestSystemVerify/IPMissing (0.00s)
        verify_test.go:378: error was not a HostnameError: x509: certificate signed by unknown authority
    --- FAIL: TestSystemVerify/Expired (0.00s)
        verify_test.go:388: error was not Expired: x509: certificate signed by unknown authority
    --- FAIL: TestSystemVerify/RootInIntermediates (0.00s)
        verify_test.go:479: unexpected error: x509: certificate signed by unknown authority
    --- FAIL: TestSystemVerify/SHA-384 (0.01s)
        verify_test.go:514: Got 0 matches instead of 1 for expected chain [api.moip.com.br COMODO RSA Extended Validation Secure Server CA COMODO RSA Certification Authority AddTrust External CA Root]

2023-02-28 22:18 windows-amd64-race go@65fa9ce8 crypto/x509.TestSystemVerify (log)

--- FAIL: TestSystemVerify (0.66s)
    --- FAIL: TestSystemVerify/Valid (0.13s)
        verify_test.go:479: unexpected error: x509: certificate signed by unknown authority
    --- FAIL: TestSystemVerify/MixedCase (0.01s)
        verify_test.go:479: unexpected error: x509: certificate signed by unknown authority
    --- FAIL: TestSystemVerify/HostnameMismatch (0.00s)
        verify_test.go:378: error was not a HostnameError: x509: certificate signed by unknown authority
    --- FAIL: TestSystemVerify/IPMissing (0.00s)
        verify_test.go:378: error was not a HostnameError: x509: certificate signed by unknown authority
    --- FAIL: TestSystemVerify/Expired (0.00s)
        verify_test.go:388: error was not Expired: x509: certificate signed by unknown authority
    --- FAIL: TestSystemVerify/RootInIntermediates (0.00s)
        verify_test.go:479: unexpected error: x509: certificate signed by unknown authority
    --- FAIL: TestSystemVerify/SHA-384 (0.02s)
        verify_test.go:514: Got 0 matches instead of 1 for expected chain [api.moip.com.br COMODO RSA Extended Validation Secure Server CA COMODO RSA Certification Authority AddTrust External CA Root]

2023-02-28 22:18 windows-arm64-11 go@65fa9ce8 crypto/x509.TestSystemVerify (log)

--- FAIL: TestSystemVerify (0.04s)
    --- FAIL: TestSystemVerify/Valid (0.02s)
        verify_test.go:479: unexpected error: x509: certificate signed by unknown authority
    --- FAIL: TestSystemVerify/MixedCase (0.00s)
        verify_test.go:479: unexpected error: x509: certificate signed by unknown authority
    --- FAIL: TestSystemVerify/HostnameMismatch (0.00s)
        verify_test.go:378: error was not a HostnameError: x509: certificate signed by unknown authority
    --- FAIL: TestSystemVerify/IPMissing (0.00s)
        verify_test.go:378: error was not a HostnameError: x509: certificate signed by unknown authority
    --- FAIL: TestSystemVerify/Expired (0.00s)
        verify_test.go:388: error was not Expired: x509: certificate signed by unknown authority
    --- FAIL: TestSystemVerify/RootInIntermediates (0.00s)
        verify_test.go:479: unexpected error: x509: certificate signed by unknown authority
    --- FAIL: TestSystemVerify/SHA-384 (0.00s)
        verify_test.go:514: Got 0 matches instead of 1 for expected chain [api.moip.com.br COMODO RSA Extended Validation Secure Server CA COMODO RSA Certification Authority AddTrust External CA Root]

2023-02-28 23:42 windows-amd64-2016 go@0384235a crypto/x509.TestSystemVerify (log)

--- FAIL: TestSystemVerify (0.74s)
    --- FAIL: TestSystemVerify/Valid (0.13s)
        verify_test.go:479: unexpected error: x509: certificate signed by unknown authority
    --- FAIL: TestSystemVerify/MixedCase (0.01s)
        verify_test.go:479: unexpected error: x509: certificate signed by unknown authority
    --- FAIL: TestSystemVerify/HostnameMismatch (0.00s)
        verify_test.go:378: error was not a HostnameError: x509: certificate signed by unknown authority
    --- FAIL: TestSystemVerify/IPMissing (0.00s)
        verify_test.go:378: error was not a HostnameError: x509: certificate signed by unknown authority
    --- FAIL: TestSystemVerify/Expired (0.00s)
        verify_test.go:388: error was not Expired: x509: certificate signed by unknown authority
    --- FAIL: TestSystemVerify/RootInIntermediates (0.00s)
        verify_test.go:479: unexpected error: x509: certificate signed by unknown authority
    --- FAIL: TestSystemVerify/SHA-384 (0.02s)
        verify_test.go:514: Got 0 matches instead of 1 for expected chain [api.moip.com.br COMODO RSA Extended Validation Secure Server CA COMODO RSA Certification Authority AddTrust External CA Root]

2023-02-28 23:42 windows-amd64-race go@0384235a crypto/x509.TestSystemVerify (log)

--- FAIL: TestSystemVerify (0.62s)
    --- FAIL: TestSystemVerify/Valid (0.13s)
        verify_test.go:479: unexpected error: x509: certificate signed by unknown authority
    --- FAIL: TestSystemVerify/MixedCase (0.00s)
        verify_test.go:479: unexpected error: x509: certificate signed by unknown authority
    --- FAIL: TestSystemVerify/HostnameMismatch (0.00s)
        verify_test.go:378: error was not a HostnameError: x509: certificate signed by unknown authority
    --- FAIL: TestSystemVerify/IPMissing (0.00s)
        verify_test.go:378: error was not a HostnameError: x509: certificate signed by unknown authority
    --- FAIL: TestSystemVerify/Expired (0.00s)
        verify_test.go:388: error was not Expired: x509: certificate signed by unknown authority
    --- FAIL: TestSystemVerify/RootInIntermediates (0.00s)
        verify_test.go:479: unexpected error: x509: certificate signed by unknown authority
    --- FAIL: TestSystemVerify/SHA-384 (0.02s)
        verify_test.go:514: Got 0 matches instead of 1 for expected chain [api.moip.com.br COMODO RSA Extended Validation Secure Server CA COMODO RSA Certification Authority AddTrust External CA Root]

2023-02-28 23:42 windows-arm64-11 go@0384235a crypto/x509.TestSystemVerify (log)

--- FAIL: TestSystemVerify (0.04s)
    --- FAIL: TestSystemVerify/Valid (0.03s)
        verify_test.go:479: unexpected error: x509: certificate signed by unknown authority
    --- FAIL: TestSystemVerify/MixedCase (0.00s)
        verify_test.go:479: unexpected error: x509: certificate signed by unknown authority
    --- FAIL: TestSystemVerify/HostnameMismatch (0.00s)
        verify_test.go:378: error was not a HostnameError: x509: certificate signed by unknown authority
    --- FAIL: TestSystemVerify/IPMissing (0.00s)
        verify_test.go:378: error was not a HostnameError: x509: certificate signed by unknown authority
    --- FAIL: TestSystemVerify/Expired (0.00s)
        verify_test.go:388: error was not Expired: x509: certificate signed by unknown authority
    --- FAIL: TestSystemVerify/RootInIntermediates (0.00s)
        verify_test.go:479: unexpected error: x509: certificate signed by unknown authority
    --- FAIL: TestSystemVerify/SHA-384 (0.00s)
        verify_test.go:514: Got 0 matches instead of 1 for expected chain [api.moip.com.br COMODO RSA Extended Validation Secure Server CA COMODO RSA Certification Authority AddTrust External CA Root]

— watchflakes