Unable to authenticate private repository

[ddebrecenijr]

What happened:
Attempting to use secrets to authenticate for my deployments on a local cluster does not work correctly.
Using imagePullSecrets does not appear to do anything, whether creating it directly from docker/config.json or inputting credentials manually.

What you expected to happen:
Secrets should authenticate private repositories as described in documentation.

How to reproduce it (as minimally and precisely as possible):
Setup minimal cluster via kubeadm running Ubuntu Server master and Windows Server node.

Anything else we need to know?:
Nodes are all Windows based.

Environment:

• Kubernetes version (use kubectl version):
  Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.3", GitCommit:"2bba0127d85d5a46ab4b778548be28623b32d0b0", GitTreeState:"clean", BuildDate:"2018-05-21T09:17:39Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"windows/amd64"}
  Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.4", GitCommit:"c27b913fddd1a6c480c229191a087698aa92f0b1", GitTreeState:"clean", BuildDate:"2019-02-28T13:30:26Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
• Cloud provider or hardware configuration: Virtual Machines
• OS (e.g: cat /etc/os-release): Master: Ubuntu Server 18.04, Nodes: Windows Server 1809
• Kernel (e.g. uname -a): Linux 4.15.0-46-generic
• Install tools: kubeadm
• Others:
  /sig node
  /sig windows

[neolit123]

Install tools: kubeadm

kubeadm is not supported on Windows ATM.
can you provide feedback of how are you setting this up using kubeadm?

Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.3", GitCommit:"2bba0127d85d5a46ab4b778548be28623b32d0b0", GitTreeState:"clean", BuildDate:"2018-05-21T09:17:39Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"windows/amd64"}

the client is way outdated compared to the server. try upgrading to 1.13.

[ddebrecenijr]

The versioning is my bad, the master and all nodes are on v1.13.4. The client was my kubectl version that I was using for cli commands.
As for kubeadm, I followed the guidelines outlined here by Microsoft Docs, I am using a host-gw flannel network, for creating the secrets I followed both ways outlined and my private repository is the standard docker registry image, i think like registry:2.7 or something like that.
Out of curiosity, where are you finding that kubeadm is not supported for Windows? From my understanding (still new to this), kubeadm is the only tool available right now for supporting windows nodes to join clusters via #53553

[neolit123]

thanks for the details,

it was mentioned to me recently that kubeadm no longer works correctly on Windows nodes, but i have no means to test this ATM.

...it used to work a couple of years ago.

here is the tracking issue for kubeadm to get this working correctly again in 1.15 (hopefully):
kubernetes/kubeadm#1393

if you find kubeadm / windows problems, please report them in that ticket.

Attempting to use secrets to authenticate for my deployments on a local cluster does not work correctly. Using imagePullSecrets does not appear to do anything, whether creating it directly from docker/config.json or inputting credentials manually.

seems like SIG Windows will have to respond to the original issue about the registry and imagePullSecrets.

[andyzhangx]

/assign
I will take a look

[andyzhangx]

I tried on following k8s windows cluster set up by aks-engine, imagePullSecrets works well:

 "agentWindowsSku": {
      "value": "Datacenter-Core-1809-with-Containers-smalldisk"
    },

kubectl create secret docker-registry regsecret --docker-server=docker.io --docker-username=<> --docker-password=<> --docker-email=<>

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: aspnet
  name: deployment-dockerhub
spec:
  replicas: 1
  selector:
    matchLabels:
      app: aspnet  
  template:
    metadata:
      labels:
        app: aspnet
      name: deployment
    spec:
      containers:
      - name: deployment
        image: docker.io/xxx/aspnet:4.7.2-windowsservercore-ltsc2019
      imagePullSecrets:
      - name: regsecret

[andyzhangx]

could you paste the error logs by kubectl describe po POD-NAME
could you also check your --docker-server address, whether it matches the image address in k8s deplyment?

[ddebrecenijr]

I removed the cluster and created a full Linux cluster instead to move forward with what I had to do. That being said I'm still interested in figuring this out.
I think my original issue may have not been descriptive enough, but it looks like you are using a docker hosted registry? I was hosting the registry myself with resolvable DNS (I could access it from local machines).
The error I remember was an ImagePullError I believe, upon further inspection it was saying that it was missing authentication despite being passed the secret to authenticate. Remoting into each machine I was able to manually authenticate each one using the same setup.
I'm unfamiliar with AKS-Engine and did not realize it was supporting Windows nodes and if thats the case, then I will try looking into them.

[andyzhangx]

@ddebrecenijr K8s on Windows should support private registry with imagePullSecrets, you could follow the same way as I do. The key point is set docker-server as your private registry domain name:

kubectl create secret docker-registry regsecret --docker-server=...

aks-engine is a tool to deploy k8s on Azure, it could provision both Linux and Windows nodes.

[ddebrecenijr]

I will try it out and see how it goes. Thank you.

[ddebrecenijr]

With the drop of v1.14.0 offering full support for windows nodes, I am going to try and upgrade my cluster and add the windows node. Closing unless the issue persists.

[ddebrecenijr]

Confirmed to work. It was an oversight on my part with nesting imagePullSecrets inside the containers array.