docker pull failing with failed to parse certificate: x509: unhandled critical extension (based on #31949)

[GarageDeveloper]

Dear all,

I'm opening this one but it definitely follows #31949.
I add here informations and diagnostics we made while facing the problem.

Description

We get a "failed to parse certificate: x509: unhandled critical extension" error message when trying to pull images on public regitries.

The environment is RHEL 7 with all docker releases we tested (EE and CE) and in particular with: docker-ce-17.09.0. This is due to Name Constraints on the certificate generated by the local PKI (see further details below)

Steps to reproduce the issue:

1. Configure HTTPS_PROXY and HTTP_PROXY correctly in the system and in the systemd unit file for docker
2. docker pull centos

Describe the results you received:

... failed to parse certificate: x509: unhandled critical extension

And no image pulled at all ;)

Describe the results you expected:

We should have pulled the image without error. (curl works fine in the same situation).

Additional information you deem important (e.g. issue happens only occasionally):

We are in a situation where security team put in place SSL interception with a local PKI. Each https request is intercepted and certificates are generated on the fly for the sites we browse. The local PKI as a CA and a sub CA. The TLS certificate of the sub CA which appears in the TLS exchange includes "Name Constraints" as critical with something like this:

X509v3 Name Constraints: critical
                Excluded:
                  DNS:.localdom

After research, it seems that docker for RHEL7 is built with golang 1.8 and that this version of golang does not support Name Constraints of type Excluded.

You can reproduce the exact same behavior by running the simple example of upstream golang and using any generated certificate with Name Constraints critical+excluded.

For example, this one to put in the certPEM in the golang example:

-----BEGIN CERTIFICATE-----
MIIDejCCAmKgAwIBAgIJAIc9PgrS4S4JMA0GCSqGSIb3DQEBCwUAMEIxCzAJBgNV
BAYTAkZSMQ0wCwYDVQQIDARQQUNBMRIwEAYDVQQHDAlNYXJzZWlsbGUxEDAOBgNV
BAoMB1Jvb3QgNDIwHhcNMTcxMDEwMTExMDQ4WhcNMjcxMDA4MTExMDQ4WjBCMQsw
CQYDVQQGEwJGUjENMAsGA1UECAwEUEFDQTESMBAGA1UEBwwJTWFyc2VpbGxlMRAw
DgYDVQQKDAdSb290IDQyMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA
7N9WtdUiLvN3a2InayZe2HGQKQCKVGsOM3NKMO2GnoWmZN5iA0OSNa4VDW8Lx5pH
nN+CIo87wD+boA1QyfkjJfkKzU6Dg1yoE5jFEPs3UJbkn/JQEmFlBI6rMio7Y7FW
feMXgQiKenqX44fDDgdLaEwFvg+B98mj2Md/sXywHw+k6cSIEZkRMK+/P+PTU0BY
B8P4wDsWmgpukN61KGnZg5FT7G+0rxAedf8Ix4JpGn/bE+86YL4sS5raI5yCjVom
TMLz2ZN6DzyAAmfxszQHz2faxA0Xlgcxd8cS0TB8wgAh4KrSe7rnZGGcnJt2hGLr
S+zIo25TJVKSFpe6qkPzdwIDAQABo3MwcTAdBgNVHQ4EFgQUsdPw4t8nn3dKWMbV
qrgepSKO9f4wHwYDVR0jBBgwFoAUsdPw4t8nn3dKWMbVqrgepSKO9f4wDwYDVR0T
AQH/BAUwAwEB/zAeBgNVHR4BAf8EFDASoRAwDoIMLmxvY2FsZG9tYWluMA0GCSqG
SIb3DQEBCwUAA4IBAQAsD1uiJCGb35NcuZupAo/5+LDFbt9R39HhMogHk1d2x1YW
0p+04M8MApS6Ki6j98IjgKsAZrRxO4k38Yj6cMUHMnE3DGE7myYimtYsB5lzhLPC
ubTfp83bJQvd66t8GArgyvrnEiSYV7zQVKchxGM25a6nrU9xn4DloP5fVYElanVD
xTevqzYpfRxglUHDQwyHalPHQinGQWelPZzQrzH7HXccyPl6a/bGTtwXSzrb8Z8N
UI1oqcW3u2VZuc65+6y41RLzEyD++lVudkSU2OVNluatQeMSj6/MYPhffD7Kj6Hr
JuSSB5Tg47pwyBdYZ1cOGZ9uq5eZfO93QIfDVoVM
-----END CERTIFICATE-----

This has been fixed in golang 1.9 commit here

Output of docker version:

Client:
 Version:      17.09.0-ce
 API version:  1.32
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:41:23 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.09.0-ce
 API version:  1.32 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:42:49 2017
 OS/Arch:      linux/amd64
 Experimental: false

Output of docker info:

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 2
Server Version: 17.09.0-ce
Storage Driver: vfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0
runc version: 3f2f8b84a77f73d38244dd690525642a72156c64
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.10.15-1-pve
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1GiB
Name: docker-test-centos
ID: TKJ7:JIPX:WZAX:2DQG:7X5R:FKU2:XIPO:JPWS:37KK:NTV6:FGAJ:3LWL
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.):

It's a golang issue and so we'd like to know if you plan to build docker on RHEL7 with golang 1.9 which solves the problem or if you could ask RH to backport commit d1211b9 to golang 1.8 as included in RH so that you rebuild docker with this particular version.
Until that we can't see any other option than bypassing SSL interception which is not always possible in companies with security in mind.

Best and thank you for the good work!

[thaJeztah]

ping @riyazdf @n4ss PTAL

[thaJeztah]

Thanks for the detailed information and analysis @GarageDeveloper

[riyazdf]

let's link this to the tracking PR for golang 1.9: #33892

[mgjadoul]

Hello,
I work for a large company and we have the same error when trying to do a docker pull from the LAN through an Internet Proxy implementing SSL decryption. And I also found that the CA Certificate in the chain that the proxy is using to "reencrypt" the ssl connection has the same extension.
Unfortunatelly, we cant disable reencryption easily as it is another team.
Is there any workaround and Do you have any idea when will this issue be solved?
Regards,
Marc

[GarageDeveloper]

Hi Marc (@mgjadoul),
as mentioned earlier, you'll have to wait for #33892 to be closed and a release of moby built with golang 1.9.x.
Until that you may use another TLS interception between you and the security equipment of your company (here your blucoat) and make so that this one does not add the unwanted extension.
Something like this:
docker -> your proxy doing ssl interception and configured without unwanted ext -> bluecoat with unwanted extensions -> internet.
I did not do it myself because we were allowed to bypass the equipment for a while but that's what I would have ended up testing.
You can create transparent interception with some iptables rules and a proxy like squid for example.
If you need help concerning this, contact me in private, I'll try to manage to find time to put something in place you could use on your side.

HTH.
Raph

[mgjadoul]

Hello, Thanks for your message. We effectivelly already solved this as you proposed, using mitmdump. So for now docker --> mitmproxy --> cntlm --> bluecoat --> Internet.... Ideally I would have liked to used squid and replace cntlm + mitmdump.... but couldn't make it working. I guess I need more time than what I have. Thanks, Marc Le dim. 19 nov. 2017 à 20:58, Raphaël Enrici <notifications@github.com> a écrit :

…

Hi Marc ***@***.*** <https://github.com/mgjadoul>), as mentioned earlier, you'll have to wait for #33892 <#33892> to be closed and a release of moby built with golang 1.9.x. Until that you may use another TLS interception between you and the security equipment of your company (here your blucoat) and make so that this one does not add the unwanted extension. Something like this: docker -> your proxy doing ssl interception and configured without unwanted ext -> bluecoat with unwanted extensions -> internet. I did not do it myself because we were allowed to bypass the equipment for a while but that's what I would have ended up testing. You can create transparent interception with some iptables rules and a proxy like squid for example. If you need help concerning this, contact me in private, I'll try to manage to find time to put something in place you could use on your side. HTH. Raph — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#35152 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ARLbSCyArxzHo034yXV94Je5M-szTtn5ks5s4IiDgaJpZM4Pzyer> .