Support typical enterprise proxy and internal mirrors setup

[rmohta]

Description

It would be nice (and increase adoption) if pack cli can support typical enterprise proxy and mirror setup.

Proposed solution

Without changing the toml files or re-building images, pack cli should be able to update the URL to download docker images used in buildpacks and stack settings.
For example, http_proxy environment variables are inherited during the build process - but not everything would have to go via http_proxy. It's common for an enterprise to have internal mirrors using Artifactory or Nexus, so that engineers don't have to fiddle with proxy settings for different types of applications used across different tech stacks.

Similar to --env, if we have --docker-mirror-registry or --docker-proxy-registry, which would allow us to provide the prefix that can be used when downloading images.

Describe alternatives you've considered

Re-build sample applications by adding our docker registry.

Additional context

• This feature should be documented somewhere

[jromero]

I thought I had replied to this...

Images are typically declared including the hostname. A format such as <hostname>/<repo>[(:<tag>|@<hash>)], for example gcr.io/my-org/my-image:latest.

If I'm understanding of how this would work with something like Artifactory's virtual repos or Nexus's proxy...

If a --docker-mirror-registry of artifactory.my-company.org/docker-repo was configured and a request for gcr.io/my-org/my-image:latest was made, the actual request for the image should be artifactory.my-company.org/docker-repo/my-org/my-image:latest. The virtual registry would in turn search one or more registries based on how it's configured.

Ultimately we'd be removing the original <hostname> and replacing it with the --docker-mirror-registry value.

Does that sound right?

Additional references:

• https://help.sonatype.com/repomanager3/formats/docker-registry/pulling-images

[jromero]

Another relevant conversation about being able to push to these proxies/mirrors. I suspect this may or may not work depending on the service (artifactory, nexus, harbor).

• I use Harbor as docker hub mirror，when I log in harbor and push mirror into harbor it not work goharbor/harbor#220
• registry:2.2 - proxy: option in config.yml breaks push & proxy/cache doesn't work distribution/distribution#1218

[rmohta]

@jromero Yes. I was hoping this to be independent of the Container Runtime. I was hoping for pack to mutate the image name, before it's sent down to "docker pull" command. If my builder.tom file has cnbs/sample-stack-run:alpine, then based on the flag, pack can have docker pull as as myregistry.com/cnbs/sample-stack-run:alpine and the might have to tag it as original name for rest of the steps to work.

If a --docker-mirror-registry of artifactory.my-company.org/docker-repo was configured and a request for gcr.io/my-org/my-image:latest was made, the actual request for the image should be artifactory.my-company.org/docker-repo/my-org/my-image:latest. The virtual registry would in turn search one or more registries based on how it's configured.

[jromero]

This issue was brought up during our working group meeting today. The question as to whether a lower level support already existing came up. After another quick search, it doesn't appear like there is a pre-existing solution for this specific use case within the docker daemon or within GGCR.

• Slack discussion: https://buildpacks.slack.com/archives/CJ6B92ZSB/p1600287646017900

[rmohta]

Thank You @jromero. I am not sure if I followed the Slack discussion on namespace and additional complexity. Was it related to this ticket?

[jromero]

@dfreilich let's sync up on this

[samj1912]

https://github.com/buildpacks/pack/pull/1088/files might help with this

[abdennour]

in addition, I think there is two strategies of registry proxy:

• pub.example.com/a-repo/a-image:a-tag becomes after proxy like: private.local/a-repo/a-image:a-tag
• pub.example.com/a-repo/a-image:a-tag becomes after proxy like : private.local/pub.example.com/a-repo/a-image:a-tag
  i think we need to support the first strategy as per how nexus sonatype works.

I am feeling that we will need a config file called .packconf which looks like :

registries-proxy:  
    docker.io: nexus.mycompany.local
    gcr.io: nexus.mycompany.local
mounts:
     ~/.m2:~/.m2
# ....

[dfreilich]

@abdennour Interesting point! For a config file, we would probably just use the existing pack config, if need be, but that's definitely food for thought.