Skip to content
This repository has been archived by the owner on Jan 25, 2022. It is now read-only.

Garden should work with a private registry that has a CA signed cert #31

Closed
onsi opened this issue Apr 4, 2015 · 10 comments
Closed

Garden should work with a private registry that has a CA signed cert #31

onsi opened this issue Apr 4, 2015 · 10 comments

Comments

@onsi
Copy link
Contributor

onsi commented Apr 4, 2015

It currently explodes. Someone running a private registry tried to fetch the Docker image using Lattice and got:

29 Mar 13:51 [garden-linux|lattice-cell-01] 2015/03/29 12:51:18 http: panic serving 127.0.0.1:41233: runtime error: invalid memory address or nil pointer dereference
29 Mar 13:51 [garden-linux|lattice-cell-01] goroutine 2028 [running]:
29 Mar 13:51 [garden-linux|lattice-cell-01] net/http.func·011()
29 Mar 13:51 [garden-linux|lattice-cell-01]     /usr/local/go/src/net/http/server.go:1130 +0xbb
29 Mar 13:51 [garden-linux|lattice-cell-01] github.com/docker/docker/registry.NewSession(0x0, 0x0, 0xc20821e240, 0x1, 0xc2081bb3e0, 0x0, 0x0)
29 Mar 13:51 [garden-linux|lattice-cell-01]     /workspace/diego-release/src/github.com/cloudfoundry-incubator/garden-linux/Godeps/_workspace/src/github.com/docker/docker/registry/session.go:58 +0x75a
29 Mar 13:51 [garden-linux|lattice-cell-01] github.com/cloudfoundry-incubator/garden-linux/old/linux_backend/repository_fetcher.registryProvider.ProvideRegistry(0xa72110, 0x1b, 0xc208160960, 0x1, 0x1, 0xc20820ae19, 0x23, 0x0, 0x0, 0x0, ...)
29 Mar 13:51 [garden-linux|lattice-cell-01]     /workspace/diego-release/src/github.com/cloudfoundry-incubator/garden-linux/old/linux_backend/repository_fetcher/repository_provider.go:49 +0x2ff
29 Mar 13:51 [garden-linux|lattice-cell-01] github.com/cloudfoundry-incubator/garden-linux/old/linux_backend/repository_fetcher.(*registryProvider).ProvideRegistry(0xc2081688a0, 0xc20820ae19, 0x23, 0x0, 0x0, 0x0, 0x0)
29 Mar 13:51 [garden-linux|lattice-cell-01]     <autogenerated>:14 +0xe1
29 Mar 13:51 [garden-linux|lattice-cell-01] github.com/cloudfoundry-incubator/garden-linux/old/linux_backend/repository_fetcher.(*DockerRepositoryFetcher).Fetch(0xc2081688d0, 0x7f0f92528ad8, 0xc208310840, 0xc20815ef50, 0xc20820ae4d, 0x6, 0x0, 0x0, 0xc208205660, 0x0, ...)
29 Mar 13:51 [garden-linux|lattice-cell-01]     /workspace/diego-release/src/github.com/cloudfoundry-incubator/garden-linux/old/linux_backend/repository_fetcher/repository_fetcher.go:101 +0x324
29 Mar 13:51 [garden-linux|lattice-cell-01] github.com/cloudfoundry-incubator/garden-linux/old/linux_backend/repository_fetcher.Retryable.Fetch(0x7f0f9252a260, 0xc2081688d0, 0x7f0f92528ad8, 0xc208310840, 0xc20815ef50, 0xc20820ae4d, 0x6, 0x0, 0x0, 0x0, ...)
29 Mar 13:51 [garden-linux|lattice-cell-01]     /workspace/diego-release/src/github.com/cloudfoundry-incubator/garden-linux/old/linux_backend/repository_fetcher/retryable.go:21 +0x15b
29 Mar 13:51 [garden-linux|lattice-cell-01] github.com/cloudfoundry-incubator/garden-linux/old/linux_backend/repository_fetcher.(*Retryable).Fetch(0xc208160970, 0x7f0f92528ad8, 0xc208310840, 0xc20815ef50, 0xc20820ae4d, 0x6, 0x0, 0x0, 0x7f0f92516000, 0x0, ...)
29 Mar 13:51 [garden-linux|lattice-cell-01]     <autogenerated>:15 +0x141
29 Mar 13:51 [garden-linux|lattice-cell-01] github.com/cloudfoundry-incubator/garden-linux/old/linux_backend/rootfs_provider.(*dockerRootFSProvider).ProvideRootFS(0xc2080314a0, 0x7f0f92528ad8, 0xc208310840, 0xc2082a8ed0, 0xb, 0xc20815ef50, 0x0, 0x0, 0xc20808e660, 0x0, ...)
29 Mar 13:51 [garden-linux|lattice-cell-01]     /workspace/diego-release/src/github.com/cloudfoundry-incubator/garden-linux/old/linux_backend/rootfs_provider/docker_rootfs_provider.go:56 +0x13e
29 Mar 13:51 [garden-linux|lattice-cell-01] github.com/cloudfoundry-incubator/garden-linux/container_pool.(*LinuxContainerPool).acquireSystemResources(0xc20816e000, 0xc2082a8ed0, 0xb, 0xc2083af040, 0x34, 0xc208326a20, 0x25, 0xc20820ae10, 0x43, 0xc2083107e0, ...)
29 Mar 13:51 [garden-linux|lattice-cell-01]     /workspace/diego-release/src/github.com/cloudfoundry-incubator/garden-linux/container_pool/container_pool.go:567 +0x6c9
29 Mar 13:51 [garden-linux|lattice-cell-01] github.com/cloudfoundry-incubator/garden-linux/container_pool.(*LinuxContainerPool).Create(0xc20816e000, 0xc2083af040, 0x34, 0x34630b8a000, 0xc20820ae10, 0x43, 0x0, 0x0, 0x0, 0x0, ...)
29 Mar 13:51 [garden-linux|lattice-cell-01]     /workspace/diego-release/src/github.com/cloudfoundry-incubator/garden-linux/container_pool/container_pool.go:252 +0x4b6
29 Mar 13:51 [garden-linux|lattice-cell-01] github.com/cloudfoundry-incubator/garden-linux/old/linux_backend.(*LinuxBackend).Create(0xc208031540, 0xc2083af040, 0x34, 0x34630b8a000, 0xc20820ae10, 0x43, 0x0, 0x0, 0x0, 0x0, ...)
29 Mar 13:51 [garden-linux|lattice-cell-01]     /workspace/diego-release/src/github.com/cloudfoundry-incubator/garden-linux/old/linux_backend/linux_backend.go:151 +0x1a3
29 Mar 13:51 [garden-linux|lattice-cell-01] github.com/cloudfoundry-incubator/garden/server.(*GardenServer).handleCreate(0xc208066800, 0x7f0f9252ab28, 0xc20812a1e0, 0xc2081c5d40)
29 Mar 13:51 [garden-linux|lattice-cell-01]     /workspace/diego-release/src/github.com/cloudfoundry-incubator/garden-linux/Godeps/_workspace/src/github.com/cloudfoundry-incubator/garden/server/request_handling.go:61 +0x309
29 Mar 13:51 [garden-linux|lattice-cell-01] github.com/cloudfoundry-incubator/garden/server.*GardenServer.(github.com/cloudfoundry-incubator/garden/server.handleCreate)·fm(0x7f0f9252ab28, 0xc20812a1e0, 0xc2081c5d40)
29 Mar 13:51 [garden-linux|lattice-cell-01]     /workspace/diego-release/src/github.com/cloudfoundry-incubator/garden-linux/Godeps/_workspace/src/github.com/cloudfoundry-incubator/garden/server/server.go:79 +0x45
29 Mar 13:51 [garden-linux|lattice-cell-01] net/http.HandlerFunc.ServeHTTP(0xc208160d60, 0x7f0f9252ab28, 0xc20812a1e0, 0xc2081c5d40)
29 Mar 13:51 [garden-linux|lattice-cell-01]     /usr/local/go/src/net/http/server.go:1265 +0x41
29 Mar 13:51 [garden-linux|lattice-cell-01] github.com/bmizerany/pat.(*PatternServeMux).ServeHTTP(0xc20802e160, 0x7f0f9252ab28, 0xc20812a1e0, 0xc2081c5d40)
29 Mar 13:51 [garden-linux|lattice-cell-01]     /workspace/diego-release/src/github.com/cloudfoundry-incubator/garden-linux/Godeps/_workspace/src/github.com/bmizerany/pat/mux.go:109 +0x21c
29 Mar 13:51 [garden-linux|lattice-cell-01] github.com/cloudfoundry-incubator/garden/server.func·002(0x7f0f9252ab28, 0xc20812a1e0, 0xc2081c5d40)
29 Mar 13:51 [garden-linux|lattice-cell-01]     /workspace/diego-release/src/github.com/cloudfoundry-incubator/garden-linux/Godeps/_workspace/src/github.com/cloudfoundry-incubator/garden/server/server.go:115 +0x57
29 Mar 13:51 [garden-linux|lattice-cell-01] net/http.HandlerFunc.ServeHTTP(0xc208160f90, 0x7f0f9252ab28, 0xc20812a1e0, 0xc2081c5d40)
29 Mar 13:51 [garden-linux|lattice-cell-01]     /usr/local/go/src/net/http/server.go:1265 +0x41
29 Mar 13:51 [garden-linux|lattice-cell-01] net/http.serverHandler.ServeHTTP(0xc208066810, 0x7f0f9252ab28, 0xc20812a1e0, 0xc2081c5d40)
29 Mar 13:51 [garden-linux|lattice-cell-01]     /usr/local/go/src/net/http/server.go:1703 +0x19a
29 Mar 13:51 [garden-linux|lattice-cell-01] net/http.(*conn).serve(0xc20812a140)
29 Mar 13:51 [garden-linux|lattice-cell-01]     /usr/local/go/src/net/http/server.go:1204 +0xb57
29 Mar 13:51 [garden-linux|lattice-cell-01] created by net/http.(*Server).Serve
29 Mar 13:51 [garden-linux|lattice-cell-01]     /usr/local/go/src/net/http/server.go:1751 +0x35e
29 Mar 13:51 [executor|lattice-cell-01] {"timestamp":"1427633478.838733673","source":"executor","message":"executor.request.depot-client.run-container.create-in-garden.failed-creating-garden-container","log_level":2,"data":{"container-guid":"affiliate-sales-0d26596a-08fe-40e3-715e-5b30d5517fe1","error":"Posthttp://api/containers: EOF","guid":"affiliate-sales-0d26596a-08fe-40e3-715e-5b30d5517fe1","method":"POST","request":"/containers/affiliate-sales-0d26596a-08fe-40e3-715e-5b30d5517fe1/run?%3Aguid=affiliate-sales-0d26596a-08fe-40e3-715e-5b30d5517fe1\u0026","session":"405.1.1.1"}}
29 Mar 13:51 [executor|lattice-cell-01] {"timestamp":"1427633478.838890076","source":"executor","message":"executor.request.depot-client.run-container.failed-creating-container-in-garden","log_level":2,"data":{"error":"Post http://api/containers: EOF","guid":"affiliate-sales-0d26596a-08fe-40e3-715e-5b30d5517fe1","method":"POST","request":"/containers/affiliate-sales-0d26596a-08fe-40e3-715e-5b30d5517fe1/run?%3Aguid=affiliate-sales-0d26596a-08fe-40e3-715e-5b30d5517fe1\u0026","session":"405.1.1"}}
@cf-gitbot
Copy link

We have created an issue in Pivotal Tracker to manage this. You can view the current status of your issue at: https://www.pivotaltracker.com/story/show/91802212.

@BugRoger
Copy link

I ran into a similar problem using Concourse. We have a self-signed certificate on our registry. I would need to be able to add our root CA to garden's certificate chain.

Here is my stacktrace:

2015/04/27 13:40:53 http: panic serving 10.0.2.15:40133: runtime error: invalid memory address or nil pointer dereference
goroutine 365 [running]:
net/http.func·011()
    /usr/local/go/src/pkg/net/http/server.go:1100 +0xb7
runtime.panic(0x921000, 0xc9cbf3)
    /usr/local/go/src/pkg/runtime/panic.c:248 +0x18d
github.com/docker/docker/registry.NewSession(0x0, 0x0, 0xc208421480, 0x1, 0xc20816ee70, 0x0, 0x0)
    /var/vcap/packages/garden-linux/src/github.com/cloudfoundry-incubator/garden-linux/Godeps/_workspace/src/github.com/docker/docker/registry/session.go:58 +0x5b5
github.com/cloudfoundry-incubator/garden-linux/old/repository_fetcher.registryProvider.ProvideRegistry(0xa1e390, 0x1b, 0xc20807af80, 0x2, 0x2, 0xc20837bb09, 0x12, 0x0, 0x0, 0x0, ...)
    /var/vcap/packages/garden-linux/src/github.com/cloudfoundry-incubator/garden-linux/old/repository_fetcher/repository_provider.go:49 +0x27f
github.com/cloudfoundry-incubator/garden-linux/old/repository_fetcher.(*registryProvider).ProvideRegistry(0xc20814d350, 0xc20837bb09, 0x12, 0x0, 0x0, 0x0, 0x0)
    <autogenerated>:14 +0xd7
github.com/cloudfoundry-incubator/garden-linux/old/repository_fetcher.(*DockerRepositoryFetcher).Fetch(0xc20814d380, 0x7f9e80d510b8, 0xc2081bcf00, 0xc20807d030, 0x99d6d0, 0x6, 0x0, 0x0, 0x0, 0x0, ...)
    /var/vcap/packages/garden-linux/src/github.com/cloudfoundry-incubator/garden-linux/old/repository_fetcher/repository_fetcher.go:101 +0x2e5
github.com/cloudfoundry-incubator/garden-linux/old/repository_fetcher.Retryable.Fetch(0x7f9e80d576d8, 0xc20814d380, 0x7f9e80d510b8, 0xc2081bcf00, 0xc20807d030, 0x99d6d0, 0x6, 0x0, 0x0, 0x7, ...)
    /var/vcap/packages/garden-linux/src/github.com/cloudfoundry-incubator/garden-linux/old/repository_fetcher/retryable.go:21 +0x160
github.com/cloudfoundry-incubator/garden-linux/old/repository_fetcher.(*Retryable).Fetch(0xc20809af40, 0x7f9e80d510b8, 0xc2081bcf00, 0xc20807d030, 0x99d6d0, 0x6, 0x0, 0x0, 0xc2082338c0, 0x0, ...)
    <autogenerated>:15 +0x137
github.com/cloudfoundry-incubator/garden-linux/old/rootfs_provider.(*dockerRootFSProvider).ProvideRootFS(0xc208154730, 0x7f9e80d510b8, 0xc2081bcf00, 0xc208075500, 0xb, 0xc20807d030, 0x0, 0x0, 0x1, 0x0, ...)
    /var/vcap/packages/garden-linux/src/github.com/cloudfoundry-incubator/garden-linux/old/rootfs_provider/docker_rootfs_provider.go:56 +0x14c
github.com/cloudfoundry-incubator/garden-linux/container_pool.(*LinuxContainerPool).acquireSystemResources(0xc208164000, 0xc208075500, 0xb, 0xc208075500, 0xb, 0xc20840cf30, 0x27, 0xc20837bb00, 0x20, 0xc2081bcde0, ...)
    /var/vcap/packages/garden-linux/src/github.com/cloudfoundry-incubator/garden-linux/container_pool/container_pool.go:569 +0x622
github.com/cloudfoundry-incubator/garden-linux/container_pool.(*LinuxContainerPool).Create(0xc208164000, 0x0, 0x0, 0x45d964b800, 0xc20837bb00, 0x20, 0x0, 0x0, 0x0, 0x0, ...)
    /var/vcap/packages/garden-linux/src/github.com/cloudfoundry-incubator/garden-linux/container_pool/container_pool.go:254 +0x493
github.com/cloudfoundry-incubator/garden-linux/linux_backend.(*LinuxBackend).Create(0xc208154820, 0x0, 0x0, 0x45d964b800, 0xc20837bb00, 0x20, 0x0, 0x0, 0x0, 0x0, ...)
    /var/vcap/packages/garden-linux/src/github.com/cloudfoundry-incubator/garden-linux/linux_backend/linux_backend.go:147 +0x1b7
github.com/cloudfoundry-incubator/garden/server.(*GardenServer).handleCreate(0xc208068200, 0x7f9e80d58630, 0xc208240500, 0xc208001d40)
    /var/vcap/packages/garden-linux/src/github.com/cloudfoundry-incubator/garden-linux/Godeps/_workspace/src/github.com/cloudfoundry-incubator/garden/server/request_handling.go:61 +0x2c6
github.com/cloudfoundry-incubator/garden/server.*GardenServer.(github.com/cloudfoundry-incubator/garden/server.handleCreate)·fm(0x7f9e80d58630, 0xc208240500, 0xc208001d40)
    /var/vcap/packages/garden-linux/src/github.com/cloudfoundry-incubator/garden-linux/Godeps/_workspace/src/github.com/cloudfoundry-incubator/garden/server/server.go:74 +0x44
net/http.HandlerFunc.ServeHTTP(0xc20809b2e0, 0x7f9e80d58630, 0xc208240500, 0xc208001d40)
    /usr/local/go/src/pkg/net/http/server.go:1235 +0x40
github.com/bmizerany/pat.(*PatternServeMux).ServeHTTP(0xc20813a060, 0x7f9e80d58630, 0xc208240500, 0xc208001d40)
    /var/vcap/packages/garden-linux/src/github.com/cloudfoundry-incubator/garden-linux/Godeps/_workspace/src/github.com/bmizerany/pat/mux.go:109 +0x20b
github.com/cloudfoundry-incubator/garden/server.func·002(0x7f9e80d58630, 0xc208240500, 0xc208001d40)
    /var/vcap/packages/garden-linux/src/github.com/cloudfoundry-incubator/garden-linux/Godeps/_workspace/src/github.com/cloudfoundry-incubator/garden/server/server.go:113 +0x56
net/http.HandlerFunc.ServeHTTP(0xc20809b530, 0x7f9e80d58630, 0xc208240500, 0xc208001d40)
    /usr/local/go/src/pkg/net/http/server.go:1235 +0x40
net/http.serverHandler.ServeHTTP(0xc208068210, 0x7f9e80d58630, 0xc208240500, 0xc208001d40)
    /usr/local/go/src/pkg/net/http/server.go:1673 +0x19f
net/http.(*conn).serve(0xc208162700)
    /usr/local/go/src/pkg/net/http/server.go:1174 +0xa7e
created by net/http.(*Server).Serve
    /usr/local/go/src/pkg/net/http/server.go:1721 +0x313

@hashneo
Copy link

hashneo commented May 18, 2015

@onsi
I fixed this in the docker code (session.go) but you haven't synced the project from github.

@hashneo
Copy link

hashneo commented May 18, 2015

@BugRoger update
./src/github.com/cloudfoundry-incubator/docker_app_lifecycle/Godeps/_workspace/src/github.com/docker/docker/registry/session.go (line 56) with

if info.Standalone && authConfig != nil && factory != nil {

and recreate release and deploy via bosh.

@goonzoid
Copy link
Contributor

@hashneo @BugRoger yeah, we're quite out of date with the version of docker we're using at this point. Unfortunately it's not super straightforward to upgrade at the moment either. We'll try to look into this soon. Apologies, and thanks for your patience.

@julz
Copy link
Contributor

julz commented May 19, 2015

@goonzoid @hashneo @BugRoger fwiw I'm fairly sure we can work around the issue without updating the docker dependency by simply passing an empty (rather than nil) AuthConfig struct to NewSession() on this line. Perhaps someone might be interested in submitting a PR with that change?

@caseyhadden
Copy link
Contributor

Upfront caveat: complete Go and garden-linux newbie here.

Our internal docker registry is setup in this manner, so we couldn't readily use it and Lattice as a gateway towards the larger platform. The last comment from @julz seemed fairly straightforward, so I figured I could possibly pattern match code my way to a solution and move things forward a little bit.

Good news is that the suggestion by @julz is in the right direction. The only addition is that the HTTPRequestFactory argument also must be non-nil. With that change, I'm able to successfully build and use the garden-linux binary against our private docker repository. I've made that change (see newbie caveat).

I did make an attempt to add a test, but am completely out of my depth (see newbie caveat). I couldn't determine how to pattern match my way into that solution. I'm a little hesitant to turn my change into a full fledged pull request, but can do so or am happy to get a little poke in the right direction with the test or another aspect (or be told that catching up with the docker dependency is the way to ultimately complete this).

The other caveat: my testing was of the integration variety and consisted of copying the garden-linux binary into my existing Lattice VM. That uncovered another change,
54fd514, that was on the surface incompatible with the current Lattice version (0.2.4). I reverted to just always using the vcap user locally for my test, but didn't push that change out. I think they are orthogonal things, but that is only an
educated guess.

Thanks.

@julz
Copy link
Contributor

julz commented May 22, 2015

hi @caseyhadden I added a comment on the linked commit, but it looks pretty close to me. You're right that failures related to changes to remove the special vcap user are unrelated (lattice just hasn't updated to the latest garden-linux yet).

If you want to try writing an integration test, we have some tests that try to connect to a registry here

@caseyhadden
Copy link
Contributor

Thanks, @julz. Those were just the pointers I needed. I've made those changes and was able to add the test. I've taken a look at the integration tests. I see usage of the docker-registry bits and how it can be configured with a cert. Will have to give some thought as to how to manage the certs themselves in the integration test environment.

I'm currently trying to track down the status of our corporate CLA. Indications are that we should have signed and submitted it with our CFF membership, but we can't seem to put our hands on it. I'll create a PR once that is sorted out, assuming this isn't fixed in the meantime.

@goonzoid
Copy link
Contributor

Fixed by #47 🎉 🎈

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

7 participants