New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x509 signed by unknown authority with Let's Encrypt certificate #2533

Closed
cehoffman opened this Issue Aug 28, 2017 · 5 comments

Comments

Projects
None yet
2 participants
@cehoffman

cehoffman commented Aug 28, 2017

Other go built tools hitting the same service do not express this issue. I believe the problem stems from git-lfs not using SNI.

$ git push
Remote "origin" does not support the LFS locking API. Consider disabling it with:
  $ git config lfs.https://git.vertiv.life/omni/vps.git/info/lfs.locksverify false
Git LFS: (0 of 29 files) 0 B / 97.76 MB
batch response: Post https://git.vertiv.life/omni/vps.git/info/lfs/objects/batch: x509: certificate signed by unknown authority
error: failed to push some refs to 'git@git.vertiv.life:omni/vps.git'

With SNI request

openssl s_client -servername git.vertiv.life -connect git.vertiv.life:443/omni/vps.git/info/lfs/objects/batch
CONNECTED(00000003)
depth=1 /C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
verify error:num=20:unable to get local issuer certificate
verify return:0
---
Certificate chain
 0 s:/CN=git.vertiv.life
   i:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
 1 s:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
   i:/O=Digital Signature Trust Co./CN=DST Root CA X3
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFATCCA+mgAwIBAgISBFGWoRShKU27A++OqatTZDSqMA0GCSqGSIb3DQEBCwUA
MEoxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MSMwIQYDVQQD
ExpMZXQncyBFbmNyeXB0IEF1dGhvcml0eSBYMzAeFw0xNzA4MjIwMjEwMDBaFw0x
NzExMjAwMjEwMDBaMBoxGDAWBgNVBAMTD2dpdC52ZXJ0aXYubGlmZTCCASIwDQYJ
KoZIhvcNAQEBBQADggEPADCCAQoCggEBANGDoVoOV/zsU/VHWk4fXyvkcqiOI2Ec
YAxASOSXuWGK3QonO0M7iJBI/C4Ekhwo9tfRkgBCY2fAoiSVTDbY75LyIDxZVzfw
ZAMAuFjkGMQ+fQ7RTOi4GNChgee9yOsbHZLWjPO9jZJrlNIOPuHQOtEynzk1FBPs
gSkgBdRtJfCTze26Z/4lfTD5ttn6GDjE5tZQ06jUVCJnBAIeRll8tbIEvQJrqCty
1ZZNOyK67Pb89SGg3qxOrg8apRt2dEh+vqWXIbnkIbyrrk0Wc4ARupmQ+FKJatq6
mWEjev7j0ReGFGKudlPw50hBw2gdPKxCfPJJS9dk8DG4kvxKQtYBOG0CAwEAAaOC
Ag8wggILMA4GA1UdDwEB/wQEAwIFoDAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYB
BQUHAwIwDAYDVR0TAQH/BAIwADAdBgNVHQ4EFgQUaeBfkahTFBcrAsVFtho8la6r
S+0wHwYDVR0jBBgwFoAUqEpqYwR93brm0Tm3pkVl7/Oo7KEwbwYIKwYBBQUHAQEE
YzBhMC4GCCsGAQUFBzABhiJodHRwOi8vb2NzcC5pbnQteDMubGV0c2VuY3J5cHQu
b3JnMC8GCCsGAQUFBzAChiNodHRwOi8vY2VydC5pbnQteDMubGV0c2VuY3J5cHQu
b3JnLzAaBgNVHREEEzARgg9naXQudmVydGl2LmxpZmUwgf4GA1UdIASB9jCB8zAI
BgZngQwBAgEwgeYGCysGAQQBgt8TAQEBMIHWMCYGCCsGAQUFBwIBFhpodHRwOi8v
Y3BzLmxldHNlbmNyeXB0Lm9yZzCBqwYIKwYBBQUHAgIwgZ4MgZtUaGlzIENlcnRp
ZmljYXRlIG1heSBvbmx5IGJlIHJlbGllZCB1cG9uIGJ5IFJlbHlpbmcgUGFydGll
cyBhbmQgb25seSBpbiBhY2NvcmRhbmNlIHdpdGggdGhlIENlcnRpZmljYXRlIFBv
bGljeSBmb3VuZCBhdCBodHRwczovL2xldHNlbmNyeXB0Lm9yZy9yZXBvc2l0b3J5
LzANBgkqhkiG9w0BAQsFAAOCAQEAbz5RcZICOgRMZa6LpYCJ//rfi1L/21be1jOb
RZjoK83zRUII8Be25b+xfz26gZpBUnjBaPPDE+ndH0KcN3dZ9wKLLS6YaMBUdT5d
blnXVEciVCT/0j4mIkrw/cpYq3TLZyAhVJtAy4XZFg6ROBfbeGiqPY9UsVyEyniL
BMyLd5W/2MXMmNHCjHqj42YFk0Df9Lvlmc+IePLVRDnTY482mH78+uq+CqCjnL30
FyXZPxS4y1S6whPfDzx5kTtx2Yjug7UmckUXgGssMT/Q2cqR4rNrXqRvJ3o3ml8u
6PvnLusbKOaGmm3QAEFxJyFcu1MjYPzCSMVDauTNWwV2WLCLOg==
-----END CERTIFICATE-----
subject=/CN=git.vertiv.life
issuer=/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
---
No client certificate CA names sent
---
SSL handshake has read 2814 bytes and written 450 bytes
---
New, TLSv1/SSLv3, Cipher is AES128-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : AES128-SHA
    Session-ID: 3261F456BDF208639F7585FD1AA249C410FC3DF44A5D6DA079E9AD05DDC78D9E
    Session-ID-ctx:
    Master-Key: 80B530A2420FEA1851D9AA52355FD9B16F72FFFD6C6E1B19524816C643A7E00A7B89BD1638AC30F9314199985113E024
    Key-Arg   : None
    TLS session ticket lifetime hint: 600 (seconds)
    TLS session ticket:
    0000 - b8 19 d6 63 f5 f9 62 3d-c9 5a 37 3b cd a1 32 0c   ...c..b=.Z7;..2.
    0010 - a8 b1 15 88 38 61 16 ef-9d bf 7a 7b 0f 75 8a 41   ....8a....z{.u.A
    0020 - 08 7e 6b c5 fa ad 55 e4-0e a6 a1 46 b3 2d 87 79   .~k...U....F.-.y
    0030 - fe f1 69 d3 94 68 8e 40-af 56 d0 f3 dd bc 09 36   ..i..h.@.V.....6
    0040 - d3 1d 3e 02 f2 ad b1 35-3e 23 38 0c e3 64 04 ff   ..>....5>#8..d..
    0050 - 67 41 ac 2c 6d 9b c9 61-98 60 22 33 8a 21 54 a0   gA.,m..a.`"3.!T.
    0060 - 5b 5c 12 75 8d 62 3e 28-fd 7c 75 5d 04 f1 95 f6   [\.u.b>(.|u]....
    0070 - 04 56 a2 5c 12 50 6e 3c-2e 43 5d 3a 84 49 ad 23   .V.\.Pn<.C]:.I.#
    0080 - 07 09 d0 2d 26 2d b8 bf-1f 6d 69 ac 04 46 1b 9c   ...-&-...mi..F..
    0090 - 78 6b 3b 94 53 3d cf 9d-ef 8e 2e 4f 3d 63 7f c2   xk;.S=.....O=c..
    00a0 - 82 57 ad 04 be 72 c8 59-a8 0b bc 9e 69 19 c9 f8   .W...r.Y....i...
    00b0 - 4d a3 c7 fd fd db 62 ea-44 2a 41 d9 31 35 0c f0   M.....b.D*A.15..

    Start Time: 1503955382
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---

Without SNI request

openssl s_client -connect git.vertiv.life:443/omni/vps.git/info/lfs/objects/batch
CONNECTED(00000003)
depth=0 /O=Acme Co/CN=Kubernetes Ingress Controller Fake Certificate
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 /O=Acme Co/CN=Kubernetes Ingress Controller Fake Certificate
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
 0 s:/O=Acme Co/CN=Kubernetes Ingress Controller Fake Certificate
   i:/O=Acme Co/CN=Kubernetes Ingress Controller Fake Certificate
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIDbzCCAlegAwIBAgIQM5n5LFlILSWJZVXQ3urLUDANBgkqhkiG9w0BAQsFADBL
MRAwDgYDVQQKEwdBY21lIENvMTcwNQYDVQQDEy5LdWJlcm5ldGVzIEluZ3Jlc3Mg
Q29udHJvbGxlciBGYWtlIENlcnRpZmljYXRlMB4XDTE3MDgxODA2NTk1NloXDTE4
MDgxODA2NTk1NlowSzEQMA4GA1UEChMHQWNtZSBDbzE3MDUGA1UEAxMuS3ViZXJu
ZXRlcyBJbmdyZXNzIENvbnRyb2xsZXIgRmFrZSBDZXJ0aWZpY2F0ZTCCASIwDQYJ
KoZIhvcNAQEBBQADggEPADCCAQoCggEBALHhsL9p57bOsJdIlE+sznvwEj0gPfBK
N167+yj2Uc208pVw75qZ+px9B0Ms2KFAZQgdduS3jRCtTtR3Hpd9RXYoW0I1s2fV
DjgReekUp0arEPSpaa9SD5k+m8B2NR78bmt6VKr8ABZ6PDxjMCaOjdlxy8Nl5p2e
9UAx0c75T6GKAT2DbqMK5v7v5+x0Rfm2Tt+japrzFntJ6K1DcQV8hbB4dNiVbxjb
7kiCXZubGpsjwSrUcU/iMh9QzOr5z8xjA18DQrQ9+EFsKIQb5f479j0hpAh6CqEo
7ZUrk8O0etz5nX++7EaEHdSRX4P+Hmmb6L1rO32npwpxfigkZ+8fb7ECAwEAAaNP
ME0wDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQMMAoGCCsGAQUFBwMBMAwGA1UdEwEB
/wQCMAAwGAYDVR0RBBEwD4INaW5ncmVzcy5sb2NhbDANBgkqhkiG9w0BAQsFAAOC
AQEAn2p1ZuPWR7s89KuXBMdZq4u8iJ5ESVW+ukRYPnbkTjQk4rxp2Rjgk1TlctWv
Wy3jXlQAKWksw8HTfq3Ut7zJd9FR9L/wIl33B09/f9Oiao4SDXRbmtjdrWzQI49K
rspD+CWLBGkgRNnit5VFZCfnrzmF81NkIE2FzEp5zup5SvnKNw0hGTYPh3Y/hbjj
VJT72/p/Ulsf2dizqWDM9+rC4HkjIL7/vn3crG5gb9/FF5EqF0rn7uoLQVxs7qMD
hKZzqqFaV4qL8ufg5oZAyHsJClpCqEySPZQoNygbnu6EgfvOCMvnis9ja47GSb05
uxi1YNsWedJY6vXhakaz12e10A==
-----END CERTIFICATE-----
subject=/O=Acme Co/CN=Kubernetes Ingress Controller Fake Certificate
issuer=/O=Acme Co/CN=Kubernetes Ingress Controller Fake Certificate
---
No client certificate CA names sent
---
SSL handshake has read 1052 bytes and written 456 bytes
---
New, TLSv1/SSLv3, Cipher is AES128-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : AES128-SHA
    Session-ID: 91D70FFA4272F9E21CE00B8315A62CFD2E83E2CA5E34340C62C53886B3B125A6
    Session-ID-ctx:
    Master-Key: CCA3DB209C7C63A3EFEBBA4E00AE808BA10AB6A0F8E6B608C6B2ED6D37062B1049905987A97E15086E0A856F70D0E030
    Key-Arg   : None
    Start Time: 1503955494
    Timeout   : 300 (sec)
    Verify return code: 21 (unable to verify the first certificate)
---
@technoweenie

This comment has been minimized.

Member

technoweenie commented Aug 31, 2017

Those SSL dumps show:

verify error:num=20:unable to get local issuer certificate
verify return:0

So, it looks like it's failing verification. Are there other root certs that your computer needs to trust?
You can disable SSL verification with one of the two commands:

# disable ssl verification everywhere
$ git config http.sslVerify false

# disable ssl verification on the specific site
$ git config http.https://git.vertiv.life.sslVerify false
@cehoffman

This comment has been minimized.

cehoffman commented Aug 31, 2017

This is a dump from my development machine where every tool but git-lfs is fine verifying the SSL certificate. There seems to be a problem with how git-lfs is integrating with the host to find certificates. If there is a problem with root certs on the computer, shouldn't things like an API tool using https://github.com/xanzy/go-gitlab, gitlab-ci-multi-runner, and git itself have problems verifying the certificate? It is strange that if I switch to using a different openssl version, e.g. a more recent version compiled through homebrew, it gets

CONNECTED(00000003)
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
verify return:1
depth=0 CN = git.vertiv.life
verify return:1
---
Certificate chain
 0 s:/CN=git.vertiv.life
   i:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
 1 s:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
   i:/O=Digital Signature Trust Co./CN=DST Root CA X3
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFATCCA+mgAwIBAgISBFGWoRShKU27A++OqatTZDSqMA0GCSqGSIb3DQEBCwUA
MEoxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MSMwIQYDVQQD
ExpMZXQncyBFbmNyeXB0IEF1dGhvcml0eSBYMzAeFw0xNzA4MjIwMjEwMDBaFw0x
NzExMjAwMjEwMDBaMBoxGDAWBgNVBAMTD2dpdC52ZXJ0aXYubGlmZTCCASIwDQYJ
KoZIhvcNAQEBBQADggEPADCCAQoCggEBANGDoVoOV/zsU/VHWk4fXyvkcqiOI2Ec
YAxASOSXuWGK3QonO0M7iJBI/C4Ekhwo9tfRkgBCY2fAoiSVTDbY75LyIDxZVzfw
ZAMAuFjkGMQ+fQ7RTOi4GNChgee9yOsbHZLWjPO9jZJrlNIOPuHQOtEynzk1FBPs
gSkgBdRtJfCTze26Z/4lfTD5ttn6GDjE5tZQ06jUVCJnBAIeRll8tbIEvQJrqCty
1ZZNOyK67Pb89SGg3qxOrg8apRt2dEh+vqWXIbnkIbyrrk0Wc4ARupmQ+FKJatq6
mWEjev7j0ReGFGKudlPw50hBw2gdPKxCfPJJS9dk8DG4kvxKQtYBOG0CAwEAAaOC
Ag8wggILMA4GA1UdDwEB/wQEAwIFoDAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYB
BQUHAwIwDAYDVR0TAQH/BAIwADAdBgNVHQ4EFgQUaeBfkahTFBcrAsVFtho8la6r
S+0wHwYDVR0jBBgwFoAUqEpqYwR93brm0Tm3pkVl7/Oo7KEwbwYIKwYBBQUHAQEE
YzBhMC4GCCsGAQUFBzABhiJodHRwOi8vb2NzcC5pbnQteDMubGV0c2VuY3J5cHQu
b3JnMC8GCCsGAQUFBzAChiNodHRwOi8vY2VydC5pbnQteDMubGV0c2VuY3J5cHQu
b3JnLzAaBgNVHREEEzARgg9naXQudmVydGl2LmxpZmUwgf4GA1UdIASB9jCB8zAI
BgZngQwBAgEwgeYGCysGAQQBgt8TAQEBMIHWMCYGCCsGAQUFBwIBFhpodHRwOi8v
Y3BzLmxldHNlbmNyeXB0Lm9yZzCBqwYIKwYBBQUHAgIwgZ4MgZtUaGlzIENlcnRp
ZmljYXRlIG1heSBvbmx5IGJlIHJlbGllZCB1cG9uIGJ5IFJlbHlpbmcgUGFydGll
cyBhbmQgb25seSBpbiBhY2NvcmRhbmNlIHdpdGggdGhlIENlcnRpZmljYXRlIFBv
bGljeSBmb3VuZCBhdCBodHRwczovL2xldHNlbmNyeXB0Lm9yZy9yZXBvc2l0b3J5
LzANBgkqhkiG9w0BAQsFAAOCAQEAbz5RcZICOgRMZa6LpYCJ//rfi1L/21be1jOb
RZjoK83zRUII8Be25b+xfz26gZpBUnjBaPPDE+ndH0KcN3dZ9wKLLS6YaMBUdT5d
blnXVEciVCT/0j4mIkrw/cpYq3TLZyAhVJtAy4XZFg6ROBfbeGiqPY9UsVyEyniL
BMyLd5W/2MXMmNHCjHqj42YFk0Df9Lvlmc+IePLVRDnTY482mH78+uq+CqCjnL30
FyXZPxS4y1S6whPfDzx5kTtx2Yjug7UmckUXgGssMT/Q2cqR4rNrXqRvJ3o3ml8u
6PvnLusbKOaGmm3QAEFxJyFcu1MjYPzCSMVDauTNWwV2WLCLOg==
-----END CERTIFICATE-----
subject=/CN=git.vertiv.life
issuer=/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-384, 384 bits
---
SSL handshake has read 3189 bytes and written 490 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-GCM-SHA256
    Session-ID: FE8DBEEEC646A1E3B164C2D8AD8FD56D06079E9A6BFAE4A4C47468C9639853BA
    Session-ID-ctx:
    Master-Key: 88C8E02C0362ADC24CE2E22FE6E7B5754E168B93127E92D7E4B0E95162206E29571A5521F4F57E0DA8CC948DBF79295D
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 600 (seconds)
    TLS session ticket:
    0000 - 82 49 cf 67 bb e6 68 99-f0 51 f7 39 85 59 01 18   .I.g..h..Q.9.Y..
    0010 - f3 79 d4 6b 7b 86 02 61-9b a9 c9 aa 70 78 b2 10   .y.k{..a....px..
    0020 - 40 91 3f 15 cb 3b 39 91-b4 f8 ee 1e fa 71 0d e7   @.?..;9......q..
    0030 - a0 4c e5 70 48 13 60 1d-ac 2a 06 f0 34 fd 89 87   .L.pH.`..*..4...
    0040 - 65 1b 9c 4f ae 9e 11 2f-58 9b cc 42 22 0f 92 5e   e..O.../X..B"..^
    0050 - 87 ca d0 0b fb 63 d0 cf-df 8b 13 15 2a a9 d5 a7   .....c......*...
    0060 - 11 ee 1c 00 06 0e 79 a8-21 ee 04 eb 88 31 f5 9e   ......y.!....1..
    0070 - c0 b8 65 b1 eb d9 dc ec-93 ad b4 3c 65 08 48 1e   ..e........<e.H.
    0080 - 3a 7c 7a c0 ea 9d 9b c1-bc 2d 6f cf 21 18 20 01   :|z......-o.!. .
    0090 - 67 a8 8e f1 b8 9c bb 0b-d2 de 0c d4 95 94 99 15   g...............
    00a0 - 90 03 3b 0c 7a 06 46 e5-2e 0a fd 15 ba 81 36 8e   ..;.z.F.......6.
    00b0 - 32 f4 54 7a 5a f2 c4 0d-12 cd 9d a9 0f f3 69      2.TzZ.........i
    00c0 - <SPACES/NULS>

    Start Time: 1504191632
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---

The root certificate DST Root CA X3 is in the Keychain under System Roots. I have tried compiling git-lfs through homebrew without success at resolving this problem.

@technoweenie

This comment has been minimized.

Member

technoweenie commented Sep 1, 2017

Ah, that dump does look like it verifies, while the other dumps you provided don't. It looks like your certs are in a location that your other tools recognize, but not Git LFS.

Git LFS relies on Go's crypto/x509 package to find certs, and extends it with support for some of Git's CA config values, specifically http.sslCAInfo/GIT_SSL_CAINFO and http.sslCAPath/GIT_SSL_CAPATH

https://git-scm.com/docs/git-config#git-config-httpsslCAInfo

Can you try configuring those values and seeing if you can get it to work?

@technoweenie

This comment has been minimized.

Member

technoweenie commented Sep 1, 2017

BTW, the crypto/x509 package source lists the files and paths it checks on linux:

https://golang.org/src/crypto/x509/root_linux.go
https://golang.org/src/crypto/x509/root_unix.go

@cehoffman

This comment has been minimized.

cehoffman commented Oct 6, 2017

The problem was I had git specific CA directory specified and that directory did not contain the Let's Encrypt CA. This had been setup a long time ago, and I had completely forgotten. Thanks for the pointer.

@cehoffman cehoffman closed this Oct 6, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment