Local docker registry with self-signed certificate

[ltetrel]

Hello,

We are trying to set-up binderhub with our own docker registry in a local network as stated here.
We first tried with a http registry, the problem is that docker forces us to use https connection so it is not reliable.
So we tried to use https with a self-signed certificate, the issue is that binderhub just allows pulling from registry with a CA certificate.
There are two solutions :

1. Change this line to allow
   for untrusted certificate with validate_cert=False => Fast and easy but maybe more insecure
2. Adding the self-signed certificate to /etc/docker/certs.d like here but should be inside the building pod, with proper mounting options => lot more "complicated" but secure

Do you have some ideas about this ?

ps: Using a public registry is not ideal for us since we sould need to buy a floating ip, a domain name, and it is far less optimal in term of pushing/pulling time (pull binderhub -> registry on local network VS pull binderhub -> DNS -> registry on public network)

Thanks,

[betatim]

Did you mean validate_cert=True in option 1 or validate_cert=False? My guess is that it is the latter.

I don't think we should continue to validate certificates. If we stop doing that we open up a whole can of worms and accidental security problems.

I don't know enough about kubernetes but I thought that each k8s cluster has its own CA that is used to sign certificates used by components of kubernetes itself, authentication towards the API server from pods, etc. Maybe it is possible to use that CA to issue a cert for the docker registry. My guess would be that this would solve the trust issue.

Otherwise, yeah it is tedious to provide the cert to all the pods but doable. Potentially even possible without having to modify BinderHub itself via https://kubernetes.io/docs/concepts/workloads/pods/podpreset/?

[ltetrel]

*Yes sorry I meant validate_cert = False

So your suggestion is to use CA certificate from k8s ? Do you know how I could get it ?
For your second suggestion, would this be done when we spawn the binderhub instance for the first time ? Or can i do it now ?

Thanks for your quick reply :)

[betatim]

I have no idea how the k8s internal CA works and if you could get it to issue you certificates or not. Was just an idea because that CA is trusted by pods already.

The pod preset is something you create once when you setup the BinderHub and it then gets applied to every pod that matches the selector you define. So you could make it match all build pods. You'd have to come up with the right configuration for you preset so that it would mount something that contains the certificate. My thinking was "store the self-signed cert in a kubernetes Secret, then mount that secret in the right place of every build pod via the preset". I've not done this myself so I can't give you more detailed instructions unfortunately.

[ltetrel]

I found the k8s certificates, there are here but seems to be self-signed /etc/kubernetes/pki/ca.crt.
Your second suggestion seems to be feasible, I will try it and give feedbacks

[ltetrel]

So I struggled all day with this. I first configured the server to use admission controller PodPreset, I had to :

1. add - --runtime-config=settings.k8s.io/v1alpha1=true to /etc/kubernetes/manifests/kube-apiserver.yaml
2. change --enable-admission-plugins=NodeRestriction,NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,PodPreset
3. delete the pod running kube-api-server to rebuild it

After that I created and applied a podpreset with :

apiVersion: settings.k8s.io/v1alpha1
kind: PodPreset
metadata:
  name: certificates
spec:
  selector:
    matchLabels:
      component: binder
  volumeMounts:
    - mountPath: /etc/docker/certs.d/IP/
      name: docker-certs
    - mountPath: /usr/local/share/ca-certificates/
      name: ubuntu-certs
  volumes:
    - name: docker-certs
      hostPath:
        path: /certs
    - name: ubuntu-certs
      hostPath:
        path: /certs

And it is working with a test pod (with both apply or create), I see the volumes *-certs mounted:

apiVersion: v1
kind: Pod
metadata:
  name: website
  labels:
    component: binder 
spec:
  containers:
    - name: website
      image: nginx
      ports:
        - containerPort: 80

However, it is not working with the binder pod. Even after deleting it..

metadata:
  annotations:
    checksum/config-map: 
    checksum/secret: 
  creationTimestamp: 
  generateName: 
  labels:
    app: binder
    component: binder
    heritage: Tiller
    name: binder
    pod-template-hash: 7bfc9b7484
    release: binderhub

Any ideas ?

[betatim]

(I've never used Pod presets myself, so this is a bit "the blind leading the blind")

One idea could be that the preset only works if the pod doesn't already have volumes/mounts defined. It would be weird but who knows. I'd define a volume and mount in your test pod to see if it still works. The other thing to double check is if the selector works correctly and that the preset is in the right namespace. Other than that I am a bit stumped too :-/

[umberto10]

Did you, guys solve the issue yet?

[ltetrel]

Hi, I don't remember exactly how I resolved this. I think it was a matter of properly initialize it after our k8s installation:

su ${admin_user} -c "sudo docker login ${docker_registry} --username ${docker_id} --password ${docker_password}"
while [ ! -d /var/lib/kubelet/ ]; do sleep 1; done; 
cp /home/${admin_user}/.docker/config.json /var/lib/kubelet/

I can close the issue since it is resolved ?

[umberto10]

emm, maybe you know how should I do it on k8s multinode cluster, with containerd as container runtime? It cannot connect to my dockerhub instance, because of remote error: tls: unknown certificate authority :/

[manics]

Hi 👋 This is an old issue which doesn't sound like a BinderHub bug so I'm closing it. If you have more questions please use the Jupyter community forum https://discourse.jupyter.org/.