cmd/go, crypto/x509: go mod tidy reports certificate expired on a non-expired certificate

[ysmilda]

What version of Go are you using (go version)?

$ go version
go version go1.20.5 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/yorick/.cache/go-build"
GOENV="/home/yorick/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/yorick/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/yorick/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.20.5"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/yorick/Development/dss/udptogrpc/go.mod"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build1398362822=/tmp/go-build -gno-record-gcc-switches"

What did you do?

When running go mod tidy with an import from a local GitLab instance with a non-expired certificate it is reported as expired.

package main

import (
	"localgitlabinstance/package"
)

func main() {
	package.Function()
}

The snippet above illustrates the basic setup used. When running go mod tidy it returns the following output:

go mod tidy Output

server response: not found: localgitlabinstance/package@v0.0.0-20230606093113-3ea3ca83ba60: unrecognized import path "localgitlabinstance/package": https fetch: Get "localgitlabinstance/package?go-get=1": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2023-06-07T08:48:08Z is after 2013-07-26T11:03:57Z

When checking the certificate via Chrome and by using echo | openssl s_client -showcerts -servername localgitlabinstance -connect localgitlabinstance:443 2>/dev/null | openssl x509 -inform pem -noout -text the certificate is valid from:

Validity
  Not Before: Jan 16 00:00:00 2023 GMT
  Not After : Jan 18 23:59:59 2024 GMT

When changing my system time to before 2013-07-26T11:03:57Z I get the following output:

go mod tidy Output

localgitlabinstance/package: cannot find module providing package localgitlabinstance/package: unrecognized import path "localgitlabinstance/package": https fetch: Get "localgitlabinstance/package?go-get=1": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2013-06-01T00:01:16+02:00 is before 2023-01-16T00:00:00Z

It then successfully retrieves the Not Before timestamp.

The path taken from retrieving the certificate retrieval in the TLS handshake to verifying it seems pretty clear, so I'm uncertain where this discrepancy between the crypto/x509 parsing and the other tooling comes from.

[seankhliao]

can you show the entire certificate?

[ysmilda]

Ofcourse, here is the (slightly redacted) certificate.

echo | openssl s_client -showcerts -servername localgitlabinstance -connect localgitlabinstance:443 2>/dev/null | openssl x509 -inform pem -noout -text Output

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            0e:f4:9d:77:31:77:92:42:60:eb:55:1b:2a:ad:79:d2
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = NL, O = Trust Provider B.V., OU = Domain Validated SSL, CN = Trust Provider B.V. TLS RSA CA G1
        Validity
            Not Before: Jan 16 00:00:00 2023 GMT
            Not After : Jan 18 23:59:59 2024 GMT
        Subject: CN = localgitlabinstance
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:a9:84:6a:b8:ff:92:0f:32:59:1b:d3:82:9b:61:
                    ea:4d:b6:4e:3e:7f:be:06:2f:2f:e5:66:96:9a:6d:
                    ce:1c:49:4c:91:68:40:e8:38:92:b4:3d:0a:e4:57:
                    60:e9:06:12:e1:b2:70:f4:3f:e6:8e:8a:14:63:9a:
                    ec:ef:d5:c1:d8:8c:53:9b:cb:42:b4:4a:84:2b:bb:
                    8e:6e:87:6a:73:6d:78:e3:e2:67:be:74:8c:57:59:
                    bc:c1:bb:9e:b8:af:93:e0:29:f9:55:51:27:5e:b9:
                    04:ff:a2:0f:2b:0d:dd:f7:c4:df:11:2c:80:ac:d0:
                    70:6e:eb:79:87:d5:0e:da:0f:bc:88:80:42:a5:2c:
                    98:cd:0c:f4:79:00:37:64:fc:44:cb:12:70:1a:0d:
                    db:6e:e6:67:42:f7:45:c5:8c:77:5e:7d:46:02:40:
                    5b:3e:61:6d:a1:5b:48:43:08:95:e6:13:81:29:fc:
                    68:d5:bf:59:34:e0:b7:f6:bc:9c:ed:18:73:a7:34:
                    9a:d5:f6:d3:b3:fb:3f:fb:64:bc:1e:ff:92:c9:69:
                    9c:12:ae:e5:87:7a:be:56:1c:61:fe:9c:8c:f7:b6:
                    01:2e:30:c0:98:0a:cf:2a:fc:b1:d0:a3:2e:d5:aa:
                    9d:7d:f0:64:70:54:73:6b:da:c6:e5:1c:90:ef:6e:
                    12:cf
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Authority Key Identifier: 
                F5:56:22:1F:D9:BF:6B:59:24:52:B0:E1:6A:CD:C0:E1:57:67:E9:E8
            X509v3 Subject Key Identifier: 
                FC:D9:62:E2:98:5A:B3:8B:77:EE:A2:F5:FE:1F:F6:A7:BF:BB:EA:3D
            X509v3 Subject Alternative Name: 
                DNS:localgitlabinstance
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 CRL Distribution Points: 
                Full Name:
                  URI:http://cdpd.digitalcertvalidation.com/TrustProviderBVTLSRSACAG1.crl
            X509v3 Certificate Policies: 
                Policy: 2.23.140.1.2.1
                  CPS: http://www.digicert.com/CPS
            Authority Information Access: 
                OCSP - URI:http://statusd.digitalcertvalidation.com
                CA Issuers - URI:http://cacerts.digitalcertvalidation.com/TrustProviderBVTLSRSACAG1.crt
            X509v3 Basic Constraints: 
                CA:FALSE
            CT Precertificate SCTs: 
                Signed Certificate Timestamp:
                    Version   : v1 (0x0)
                    Log ID    : 76:FF:88:3F:0A:B6:FB:95:51:C2:61:CC:F5:87:BA:34:
                                B4:A4:CD:BB:29:DC:68:42:0A:9F:E6:67:4C:5A:3A:74
                    Timestamp : Jan 16 12:41:04.883 2023 GMT
                    Extensions: none
                    Signature : ecdsa-with-SHA256
                                30:44:02:20:6B:AD:8D:F8:8A:57:F7:39:1C:42:BD:31:
                                7F:23:0F:31:84:64:1B:8A:7D:AD:A2:B5:D0:7E:E8:38:
                                A9:D5:A6:A5:02:20:15:44:7C:DB:2B:45:E1:06:A6:44:
                                49:7E:74:55:EC:4E:64:14:82:C0:FA:4F:DD:89:21:2D:
                                71:8C:43:40:18:E6
                Signed Certificate Timestamp:
                    Version   : v1 (0x0)
                    Log ID    : 73:D9:9E:89:1B:4C:96:78:A0:20:7D:47:9D:E6:B2:C6:
                                1C:D0:51:5E:71:19:2A:8C:6B:80:10:7A:C1:77:72:B5
                    Timestamp : Jan 16 12:41:04.888 2023 GMT
                    Extensions: none
                    Signature : ecdsa-with-SHA256
                                30:45:02:21:00:AC:F4:9B:1E:78:29:E0:EA:72:98:18:
                                CF:60:60:94:BE:64:19:1B:83:A3:4E:9C:81:31:CB:21:
                                5D:BC:E8:73:2B:02:20:12:61:91:80:65:C8:9E:31:D7:
                                26:33:2D:B2:B7:3A:4D:25:F1:73:35:6A:E6:69:8F:7E:
                                AC:98:F6:4B:AA:97:08
                Signed Certificate Timestamp:
                    Version   : v1 (0x0)
                    Log ID    : 48:B0:E3:6B:DA:A6:47:34:0F:E5:6A:02:FA:9D:30:EB:
                                1C:52:01:CB:56:DD:2C:81:D9:BB:BF:AB:39:D8:84:73
                    Timestamp : Jan 16 12:41:04.839 2023 GMT
                    Extensions: none
                    Signature : ecdsa-with-SHA256
                                30:45:02:21:00:97:71:93:C5:43:BB:EA:7B:37:D3:D3:
                                76:FE:5A:45:9C:DA:B6:41:8D:99:2F:67:71:1B:96:BC:
                                60:A9:49:17:8E:02:20:2A:72:C7:DB:41:69:D2:B4:EA:
                                C0:45:BA:95:B7:33:FA:45:80:D0:D5:FA:75:03:F2:A4:
                                7A:A8:2E:38:0C:49:1E
    Signature Algorithm: sha256WithRSAEncryption
    Signature Value:
        4d:5a:ac:c8:b4:b1:15:b6:ec:fe:91:6b:af:c3:c5:ad:36:b7:
        29:25:b4:8c:db:08:4f:c7:ea:e2:66:b4:c8:f4:d3:d7:45:31:
        4a:11:74:f7:c3:d3:49:07:29:d6:ee:62:f7:11:15:b5:fc:89:
        12:b6:89:e6:0e:4f:aa:62:3a:43:e3:9f:0d:72:8a:8d:99:c5:
        dd:e3:40:34:ec:6a:75:f6:2f:d2:a9:b6:d7:8e:d5:e3:34:86:
        33:28:e2:bf:85:e2:06:bd:a9:7b:81:5a:1e:61:1b:fd:7e:44:
        e4:7f:37:b6:eb:0a:41:61:08:39:18:b7:43:6b:d7:af:27:9c:
        a3:8e:24:5d:86:26:dd:9a:0f:16:7c:3b:fa:4c:ab:2b:a5:dc:
        f6:69:81:04:b3:9a:6e:be:f9:f5:70:36:c6:df:3a:eb:d2:7c:
        2e:17:2d:3d:5a:ab:51:97:05:f5:e2:d0:22:15:fe:3f:5e:37:
        ae:e1:c2:5c:f3:21:96:fd:19:a5:a8:27:48:b7:b2:cb:c0:33:
        3b:f9:dc:82:0a:3c:1d:b6:76:f8:69:c9:d4:2b:b9:82:8d:e2:
        e9:6b:f3:7a:1b:33:c1:5b:52:d4:12:67:45:90:1f:8e:f1:eb:
        31:cb:52:20:76:ab:b3:50:47:22:6a:22:1b:af:a7:d3:65:7b:
        4f:3d:b3:58

[gopherbot]

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

[seankhliao]

I don't think we've been given enough information to identify the issue.
The given certificate is a precertificate, with no real leaf certificate issued: https://crt.sh/?serial=0e%3Af4%3A9d%3A77%3A31%3A77%3A92%3A42%3A60%3Aeb%3A55%3A1b%3A2a%3Aad%3A79%3Ad2 . that doesn't sound like something we would be trusting

[heschi]

cc @golang/security

[FiloSottile]

The given certificate is a precertificate, with no real leaf certificate issued: https://crt.sh/?serial=0e%3Af4%3A9d%3A77%3A31%3A77%3A92%3A42%3A60%3Aeb%3A55%3A1b%3A2a%3Aad%3A79%3Ad2 . that doesn't sound like something we would be trusting

No, the certificate in #60653 (comment) is the certificate corresponding to the precertificate you found. It's pretty normal for CT logs to observe the precert and not the cert, especially for internal services.

The question is where the 2013-07-26T11:03:57Z timestamp came from.

@ysmilda are you sure the server is not serving different certificates to different connections? Is the issue intermittent or does it always reproduce? If you try to make a simple net/http connection without go mod tidy, does it reproduce?

Thank you.

[ysmilda]

@FiloSottile With go mod tidy the issue always reproduces, however with a simple http.Get() the request is succesfull.

[rolandshoemaker]

@ysmilda is the timestamp returned always 2013-07-26T11:03:57Z?

[ysmilda]

Yes, tested over multiple machines it always returns the same timestamp.

The http.Get() which was succesfull before now returns the same certificate issue.

[rolandshoemaker]

If you are on an enterprise network, it seems plausible that this could be a TLS middlebox trying to do something funky and failing, or some other class of TLS interception failure.

Running the following program should give you an impression of the certificate chain being returned, I expect this is a local configuration issue, rather than a problem on our end, but if it turns up something strange I'd be interested in seeing it.

package main

import (
	"crypto/sha256"
	"fmt"
	"log"
	"net/http"
)

func main() {
	r, err := http.Get("<insert the url you are using for your local instance>")
	if err != nil {
		log.Fatal(err)
	}
	for i, chain := range r.TLS.VerifiedChains {
		fmt.Println("chain", i)
		for _, c := range chain {
			fmt.Printf("%x %s\n", sha256.Sum256(c.Raw), c.Subject)
		}
	}
}

[ysmilda]

I would expect it to be an issue on our end, as it is a enterprise network where the gitlab instance is not accessible from outside. However there doesn't seem to be anything in our network that does non standard TLS handling, but I must admit my knowledge on the TLS subject is limited.

outcome test script Output

chain 0
bd1a100977dfa0ef4379c507635c3bfd885d57ba51624b114705b7c1c59cb1c2 CN=*.domain.nl
009871c3a4c607311e5ae92f01095f9bf76100b8794ab0a9a5210e6794c8607c CN=Trust Provider B.V. TLS RSA CA G1,OU=Domain Validated SSL,O=Trust Provider B.V.,C=NL
cb3ccbb76031e5e0138f8dd39a23f9de47ffc35e43c1144cea27d46a5ab1cb5f CN=DigiCert Global Root G2,OU=www.digicert.com,O=DigiCert Inc,C=US

[ysmilda]

When setting the GOPRIVATE variable via go env to recognize the local instance as a private instance I've got it to work.

go env -w GOPRIVATE='localinstance'

I now see that go mod uses a proxy to get the modules instead of contacting the source directly. Our local instance is not reachable from the outside world and will be redirected to our main website. This is what serves the "invalid" certificate.

The interesting part here is that here Chrome still reports a valid certificate while the above script reports the aforementioned x509: certificate has expired or is not yet valid: current time 2013-06-01T00:01:16+02:00 is before 2023-01-16T00:00:00Z error.