Pull registry is not getting authenticated

[balbusm]

Describe the bug

<sever> authentication in settings.xml doesn't work for pullregistry.
Also using docker login is not respected with pullregistry (settings.xml without server section).

I have two separate docker registries. One for pull (dockerhub proxy) and one for push.
Case 1. jkube doesn't respect settings.xml configuration

settings.xml

<settings>
  <servers>
    <server>
      <id>my.proxy</id>
      <username>user</username>
      <password>pass</password>
    </server>
  </servers>
</settings>

Dockerfile

FROM my.proxy/openjdk:17
# rest of the file

I noticed that everything works as expected when I set server id to docker.io
Case 2. After logging via docker login to my.proxy I'm getting unauthorized exception.

Eclipse JKube version

1.10.0

Component

Kubernetes Maven Plugin

Apache Maven version

3.8.1

Gradle version

No response

Steps to reproduce

1. Set settings.xml

<settings>
  <servers>
    <server>
      <id>my.proxy</id>
      <username>user</username>
      <password>pass</password>
    </server>
  </servers>
</settings>

2. Set docker file
   Dockerfile

FROM my.proxy/openjdk:17

3. Set pull registry in pom:

 <plugin>
        <groupId>org.eclipse.jkube</groupId>
        <artifactId>kubernetes-maven-plugin</artifactId>
        <version>1.10.1</version>
        <configuration>
          <pullregistry>my.proxy</pullregistry>
        </configuration>

Expected behavior

Image is being downloaded from registry.

Runtime

Kubernetes (vanilla)

Kubernetes API Server version

other (please specify in additional context)

Environment

macOS

Eclipse JKube Logs

Error while trying to build the image: Unable to pull 'my.proxy/openjdk:17' : {"message":"Head \"https://my.proxy/v2/openjdk/manifests/17\": unknown: Authentication is required"} (Internal Server Error: 500) -> [Help 1]

Sample Reproducer Project

No response

Additional context

No response

[rohanKanojia]

@balbusm : Hi, Did you provide Docker Build arg mvn k8s:build -Ddocker.buildArg.https_proxy=my.proxy (It's for setting container proxy)?

[rohanKanojia]

@balbusm : Sorry, I have another question. In your settings.xml configuration, is the username password for your proxy authentication or for your DockerHub authentication?

Dockerfile
FROM my.proxy/openjdk:17

Why do you have proxy URL in image name?

[balbusm]

-Djkube.docker.pull.username/-Djkube.docker.pull.password works for me but according to documentation settings.xml or docker login should also work, but it doesn't.
I rather meant docker registry mirror, which adds registry name before image name.

[rohanKanojia]

Does it work if you set the server id to index.docker.io or docker.io in settings.xml?

[balbusm]

Yes, it works but it's not obvious and not consistent when you compare it to push config.

[rohanKanojia]

not consistent when you compare it to push config.

@balbusm : Umm, I would appreciate it if you could elaborate your point a little. By push config, are you referring to config in settings.xml or something else.

[rohanKanojia]

I tried configuring my docker daemon to use a local squid proxy and tried building from a private base image from dockerhub but it seems to be working for me. Is proxy using authentication in your case?

While doing docker login into my.proxy, did you specify https://my.proxy or just my.proxy? Just wondering whether this could be a misconfiguration issue.

[balbusm]

Let's simplify a scenario.
I have one custom repository to pull and push images.
Push config works as one would expect:

• it's possible to push image my.repo/my-image to registry after docker login my.repo is executed
• OR in settings.xml one needs to create server with id my.repo and push image my.repo/my-image

<settings>
  <servers>
    <server>
      <id>my.repo</id>
      <username>user</username>
      <password>pass</password>
    </server>
  </servers>
</settings>

In both cases push works. So I would expect if I try to pull image during a docker build above cases would work.
However:
If I try to build docker file with FROM my.repo/my-image jkube is not able to pull image.
I have to add second entry to the settings.xml:

<settings>
  <servers>
    <server>
      <id>my.repo</id> # this works only for push to my.repo
      <username>user</username>
      <password>pass</password>
    </server>
    <server>
      <id>docker.io</id> # passing here again same credentials to my.repo docker registry for pull command, 
      <username>user</username>
      <password>pass</password>
    </server>
  </servers>
</settings>

I login with docker login my.repo no https://my.repo.

[rohanKanojia]

@balbusm : Thanks a lot for your detailed explanation. I was trying to reproduce your issue by pulling a private dockerhub image as a base image and pushing it to the same dockerhub username; I had provided credentials in my settings.xml. In this case, both push and pull and working.

Let me try to reproduce your issue by setting up a local registry.

[balbusm]

The thing is you need a prefix, dockerhub registry doesn't have a prefix.

[rohanKanojia]

The thing is you need a prefix, dockerhub registry doesn't have a prefix.

By prefix, you mean a domain name?

I just tried again by setting up a local docker registry and using a base image based on that. However, I'm struggling to setup a docker registry with TLS.

I also tried the abovementioned scenario using a private image on Quay as base image and using k8s:push to push image image build (credentials specified in settings.xml) . But in this case as well, pull and push are working for me.

I would really appreciate it if you could provide some reproducer as I'm struggling in reproducing this issue.

[rohanKanojia]

@balbusm : May I ask which registry service are you using? (GitLab, Nexus ..?) I can try to create an account there for testing to replicate your scenario.

[balbusm]

I'm pulling from jfrog and pushing to local nexus.

[rohanKanojia]

Okay, so I tried this set up but still couldn't reproduce.

I'll explain the steps I followed:

1. Create an account on JFrog and set up a docker registry
2. Do a docker login myusername.jfrog.io and tag+push openjdk:17 image to myusername.jfrog.io/default-docker/openjdk:17
3. Do docker logout (Making sure there are no saved logins in ~/.docker/config.json
4. Download Nexus and set up a private docker hosted registry at localhost:8083
5. Provide credentials for both registries in ~/.m2/settings.xml

  <servers>
    <server>
      <id>myusername.jfrog.io</id>
      <username>myusername@example.com</username>
      <password>secret</password>  
    </server>
    <server>
      <id>localhost:8083</id>
      <username>admin</username>
      <password>secret</password>  
    </server>
  </servers>

6. Configure push image name in pom.xml

<jkube.image.name>localhost:8083/%a:%l</jkube.image.name>

7. Reference base image from JFrog registry in Dockerfile:

FROM myusername.jfrog.io/default-docker/openjdk:17

8. Do mvn k8s:build . Image Pulled from JFrog and builds okay.
9. Do mvn k8s:push. Image Pushed to local Nexus

Could you please review these and check if I missed anything?

[balbusm]

I cannot reproduce it on free version of JFrog.
Maybe adding extra logs might help?
This is what I get currently:

[INFO] --- kubernetes-maven-plugin:1.10.1:build (default-cli) @ ---
[INFO] k8s: Building Docker image in Kubernetes mode
[INFO] k8s: Using Dockerfile: sample/src/main/docker/Dockerfile
[INFO] k8s: Using Docker Context Directory: sample
[ERROR] k8s: Failed to execute the build [Error while trying to build the image: Unable to pull 'sample.jfrog.io/java:jre-17' : {"message":"Head \"https://sample.jfrog.io/v2/java/manifests/jre-17\": unknown: Authentication is required"} (Internal Server Error: 500)]

[manusa]

Hi @balbusm
Maybe we can have a sync session you debug in your environment so we can try to figure out what might be going on.

[balbusm]

Sure, let's do a debugging session.
How about 13th Jan (Friday)?
What time suits you?

[balbusm]

@manusa
I'm usually have some spare time on Friday. Let me know if there is any Friday that fits you.

[rohanKanojia]

I'm sorry I completely missed this. I don't have many meetings on Friday as well. What timezone are you based? What time would be appropriate for you? We usually use Google Meet for video calls.

[balbusm]

I'm based in CET Tz (Central European Time).
What Tz are you based?
I can make 9.30 CET or 15+ CET.

[rohanKanojia]

I'm based in IST and @manusa and @sunix are based in CET .

[rohanKanojia]

@balbusm : Could you please send me an email (rohaan@redhat.com) so that we can go ahead and schedule a call?

[rohanKanojia]

Debug Session Findings:

1. User seems to be using the image in form of foo-bar-example-registry.jfrog.io/java:jre-17
2. Our ImageName class tries to parse registry from image name but fails to do so. It assumes the first part is a user and expects format $registry/$user/$imageName:$imageTag

Outcome:

We need to change ImageName behavior to correctly parse the image name of pattern $registry/$imageName:$imageTag pattern. Check Docker documentation for more details.

[sunix]

maybe should be parsed with a regexp like here: https://github.com/eclipse-che/che-plugin-registry/blob/main/build/dockerfiles/entrypoint.sh#L39

[manusa]

Related code in Docker's code-base:

• https://github.com/distribution/distribution/blob/0c958010ace2e0c2a87f1bf9915b7c74157dfb62/reference/normalize.go
• https://github.com/distribution/distribution/blob/0c958010ace2e0c2a87f1bf9915b7c74157dfb62/reference/normalize.go#L126-L140

[rohanKanojia]

@balbusm : Upon revisiting this issue and trying to reproduce it locally using a local registry I wasn't able to reproduce the issue. I can see that ImageName still parses it as a user but somehow docker daemon seems to pull the image correctly.

Could you please check my steps and see if you see similar behavior?

1. Run a local docker registry: docker run -d -p 80:5000 --name registry_test registry:2
2. Add an entry for foo-bar-registry.jfrog.io domain name mapping to localhost
3. Upload an image openjdk:17 to this local registry using docker push
4. Configure project to pull base image by adding property <jkube.generator.from>foo-bar-registry.jfrog.io/openjdk:17</jkube.generator.from>
5. Run mvn k8s:build (For me image seems to be getting pulled rather than complaining about registry host)
6. mvn k8s:push (For me image seems to be getting pushed okay)

[rohanKanojia]

@balbusm : I'm going to revert my changes from your branch.

[rohanKanojia]

@balbusm : Sorry, it took quite longer than I expected for this issue.

Could you please try out whether the fix is working for your pipeline on 1.14-SNAPSHOT? You would need to add this to your pom.xml in order to enable SNAPSHOTs:

  <pluginRepositories>
    <pluginRepository>
      <id>oss.sonatype.org</id>
      <url>https://oss.sonatype.org/content/repositories/snapshots</url>
      <snapshots>
        <enabled>true</enabled>
        <updatePolicy>always</updatePolicy>
      </snapshots>
    </pluginRepository>
  </pluginRepositories>