Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issues with private Docker Cloud repos and the 1.9.0 CLI #57427

Closed
pluttrell opened this issue Dec 20, 2017 · 14 comments

Comments

Projects
None yet
@pluttrell
Copy link

commented Dec 20, 2017

/kind bug

We're having trouble retrieving images from a private registry on DockerCloud. For nearly a year the following has worked flawlessly. Last Friday I pulled down the 1.9.0 CLI (via brew) and since then have not been able to access any image in a private registry on DockerCloud. I've tried this on 10+ brand new clusters created with kops v1.8.0. The Kubernetes cluster version have been 1.8.0, 1.8.4, 1.8.5 and 1.9.0 (which is not supported by kops yet) and all exhibit the same problem.

We create the secret as such:

kubectl create secret docker-registry dockerclouduser \
    --docker-server=https://index.docker.io/v1/ \
    --docker-username={my-username} \
    --docker-password={my-password} \
    --docker-email={my-email}

And verify it using:

kubectl get secret dockerclouduser --output=json | jq --raw-output '.data | .".dockercfg"' | base64 -D | python -m json.tool

Which yields:

{
    "auths": {
        "https://index.docker.io/v1/": {
            "auth": "{what-appears-to-be-a-token}",
            "email": "{my-email}",
            "password": "{my-password}",
            "username": "{my-username}"
        }
    }
}

And create a test service with kubectl create -f with these contents:

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: test-service
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: test-service
    spec:
      containers:
      - name: test-service
        image: {my-org}/{my-repo-name}:{my-tag}
        imagePullPolicy: Always
      imagePullSecrets:
        - name: dockerclouduser

The pod fails immediatly with:

Failed to pull image "{my-org}/{mmy-repo-name}:{my-tag}": rpc error: code = Unknown desc = Error response from daemon: repository {my-org}/{mmy-repo-name} not found: does not exist or no pull access

I've reviewed the 1.9.0 release notes and nothing jumps out as the root cause.

For reference I've tried putting quotes around the {my-password} value when creating the secret to no avail.

I've also tried prefixing the spec.template.spec.containers.[0].image value with index.docker.io, so the full value would read: image: index.docker.io/{my-org}/{my-repo-name}:{my-tag} also to no avail.

Environment:
Kubernetes version:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.0", GitCommit:"925c127ec6b946659ad0fd596fa959be43f0cc05", GitTreeState:"clean", BuildDate:"2017-12-16T03:16:50Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.0", GitCommit:"925c127ec6b946659ad0fd596fa959be43f0cc05", GitTreeState:"clean", BuildDate:"2017-12-15T20:55:30Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}

Cloud provider and installation info:

  • AWS cluster built with Kops 1.8.0, using (to isolate this issue) nearly the default settings for everything.
@pluttrell

This comment has been minimized.

Copy link
Author

commented Dec 20, 2017

@kubernetes/sig-cli-bugs
@kubernetes/sig-cli-maintainers

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

commented Dec 20, 2017

@pluttrell: Reiterating the mentions to trigger a notification:
@kubernetes/sig-cli-bugs

In response to this:

@kubernetes/sig-cli-bugs
@kubernetes/sig-cli-maintainers

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@drazul

This comment has been minimized.

Copy link

commented Dec 20, 2017

I have same problem, and I "solved" it creating a docker-registry secret in an old kubernetes version and pasting it on new kubernetes

@ITler

This comment has been minimized.

Copy link

commented Dec 20, 2017

I'm recently (I know it worked perfectly about last month) facing issues with kubectl create secret docker-registry ... as well. For me it ended in a similar manner saying: Failed to pull image "{my-org}/{my-repo-name}:{my-tag}": rpc error: code = Unknown desc = unauthorized: authentication required

I was able to confirm, that this is a kubernetes CLI related issue and not end user. Based on this guide and this article I was able to create properly formatted auth information looking like this: { "https://my.repo/v1/": { "auth": "SOMEnAUTHnTOKENnGENERATEDnBYnDOCKERnLOGIN" } }. When encoding this one-liner using base64 (MacOS workstation) and apply like kubectl apply -f /tmp/manually_encoded_secret.yaml this works fine and images can be pulled from private repository again.
However this breaks all automation and can't be considered as solution

Confirmed working with kubectl 1.8.5 and server 1.8.4 (guess 1.7.9 as well)

@dims

This comment has been minimized.

Copy link
Member

commented Dec 20, 2017

looks like there's an extra "auths" element when adding the secret using 1.9 kubectl

[dims@bigbox 11:34] ~/go/src/k8s.io/kubernetes ⟩ kubectl get secret dockerclouduser --output=json | jq --raw-output '.data | .".dockercfg"' | base64 -d | python -m json.tool
{
    "auths": {
        "https://index.docker.io/v1/": {
            "auth": "ZGltczpPcGVuU3RhY2s=",
            "email": "dims@yahoo.com",
            "password": "OpenStack",
            "username": "dims"
        }
    }
}
[dims@bigbox 11:34] ~/go/src/k8s.io/kubernetes ⟩ kubectl get secret dockerclouduser-old --output=json | jq --raw-output '.data | .".dockercfg"' | base64 -d | python -m json.tool
{
    "https://index.docker.io/v1/": {
        "auth": "ZGltczpPcGVuU3RhY2s=",
        "email": "dims@yahoo.com",
        "password": "OpenStack",
        "username": "dims"
    }
}
@liggitt

This comment has been minimized.

Copy link
Member

commented Dec 20, 2017

Added in #53916

/assign @juanvallejo

k8s-github-robot pushed a commit that referenced this issue Dec 20, 2017

Kubernetes Submit Queue
Merge pull request #57463 from dims/fix-accessing-private-docker-regi…
…stries

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Fix problem accessing private docker registries

**What this PR does / why we need it**:
In 027c8b9, we added code to
move from .dockercfg to config.json file. But we forgot to use
the right secret type and the key to store the base64'ed creds

**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #57427 #57273

**Special notes for your reviewer**:

**Release note**:

```release-note
Fixes issue creating docker secrets with kubectl 1.9 for accessing docker private registries.
```
@pluttrell

This comment has been minimized.

Copy link
Author

commented Dec 21, 2017

@dims Many thanks for the quick fix on this.

Any idea when this might get rolled out?

Also, it might be a good idea if this was added to the known issue list for v1.9.0. It would have saved us a bunch of time trying to track down the root cause. Just trying to make sure no one else needs to do the same.

@dims

This comment has been minimized.

Copy link
Member

commented Dec 21, 2017

linking to the "v1.9.0 known issues" list #57159

@dims

This comment has been minimized.

Copy link
Member

commented Dec 21, 2017

@pluttrell ack, asking Mehdy since he is the patch manager for v1.9 branch.

@mbohlool Mehdy, any idea when we can ship 1.9.1 with this fix?

@liggitt

This comment has been minimized.

Copy link
Member

commented Dec 24, 2017

workarounds for folks until 1.9.1 is released:

if you have a .dockerconfigjson for your private registry already, you can manually specify the type and data key (e.g. kubectl create secret generic my-secret-name --type=kubernetes.io/dockerconfigjson --from-file .dockerconfigjson=/path/to/.dockerconfigjson)

if you don't have a .dockerconfigjson file already, you can fix up the secret produced by kubectl create secret docker-registry manually:

  1. add --dry-run -o yaml > secret.yaml
  2. change the type from kubernetes.io/dockercfg to kubernetes.io/dockerconfigjson
  3. change the data key from .dockercfg to .dockerconfigjson
  4. create the modified secret with kubectl create -f secret.yaml
@mbohlool

This comment has been minimized.

Copy link
Member

commented Jan 4, 2018

1.9.1 will be released tomorrow

@aknuds1

This comment has been minimized.

Copy link
Contributor

commented Jan 6, 2018

I can confirm @liggitt's solution works, thanks so much!

@WillPlatnick

This comment has been minimized.

Copy link

commented Jan 17, 2018

I am still having issues in 1.9.1. This worked in the 1.8.x branches:

  docker-registry gitlab-registry \
  --docker-server="$CI_REGISTRY" \
  --docker-username="$GL_USERNAME" \
  --docker-password="$GL_PASSWORD" \
  --docker-email="$GITLAB_USER_EMAIL" \
  -o yaml --dry-run | kubectl apply -n $KUBE_NAMESPACE -f -```

results in the following error:
```The Secret "gitlab-registry" is invalid: type: Invalid value: "kubernetes.io/dockerconfigjson": field is immutable```
@liggitt

This comment has been minimized.

Copy link
Member

commented Jan 17, 2018

you have to remove the gitlab-registry secret before you can apply an update... type is immutable, and the fix in 1.9.1 required changing the type of the secret

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.