Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document how to add corporate root SSL certificates #1408

Open
isuftin opened this issue Apr 25, 2017 · 31 comments

Comments

Projects
None yet
@isuftin
Copy link

commented Apr 25, 2017

This is a question/feature request.

I am currently operating minikube on a network that requires all SSL traffic to be intercepted using a root SSL certificate. Minikube is using the Virtualbox driver.

Using docker-machine and boot2docker, I was able to add my root certificate to I need to be able to append my SSL root cert to the /var/lib/boot2docker/certs directory. This does not work with the minikube ISO. What does work is appending my certificate to /etc/ssl/certs/ca-certificates.crt

I create a bootlocal.sh for this. This is the command I run from my host:

read -d '' String <<"EOF"
#!/bin/bash
grep -q '<single line from root cert>' /etc/ssl/certs/ca-certificates.crt
EXIT_CODE=\\$?
if [ \\$EXIT_CODE -ne 0 ]; then
  curl http://<rootcert location>/root.cer | sudo tee -a /etc/ssl/certs/ca-certificates.crt
  sudo systemctl restart docker.service
fi
EOF
echo "echo "\""${String}"\"" | sudo tee -a /var/lib/boot2docker/bootlocal.sh && sudo chmod +x /var/lib/boot2docker/bootlocal.sh && /var/lib/boot2docker/bootlocal.sh" | minikube ssh 

This works fine until I restart the VM. It looks like what's happening is that bootlocal may be running first or not at all because when the machine restarts, ca-certificates.crt is back to the original state without my certificate in it.

How should I add my root SSL cert into the VM so Docker can pull from registries?

minikube version: v0.18.0
OS: OSX 10.10.5
Drivername: Virtualbox
ISO: minikube-v0.18.0.iso
Install Tools: homebrew

What happened:

Corporate root SSL certificate does not exist on the VM and is not picked up by the Docker daemon

What you expected to happen:

I expect to be able to add my root SSL certificate to the VM and have Docker be able to pick it up to validate TLS communication

How to reproduce it (as minimally and precisely as possible):

$ minikube start

< run above commandset to pull down any SSL cert >

$ minikube stop && minikube start

< verify that SSL cert does not exist in ca-certificates.crt >
@r2d4

This comment has been minimized.

Copy link
Member

commented Apr 27, 2017

I'm not exactly sure why this method isn't working, since it seems like it should. Either way, we should expose something similar to /var/lib/boot2docker/certs to auto load self-signed certs.

@r2d4 r2d4 added the kind/bug label Apr 27, 2017

@isuftin

This comment has been minimized.

Copy link
Author

commented Apr 28, 2017

@r2d4 Are you able to tell whether bootlocal.sh runs before or after the certificate configuration happens? If before, there's the issue.

I'd love to be able to just dump my cert into /var/lib/boot2docker/certs and call it a day like I do with docker-machine

@isuftin

This comment has been minimized.

Copy link
Author

commented Apr 28, 2017

@r2d4 The other issue would be that /var/lib/boot2docker/certs gets wiped on VM reboot, so that would need to be persisted or maybe copied in from /data/certs or /Users/[...]. Having it persist would be preferable.

@calaniz

This comment has been minimized.

Copy link

commented Jun 6, 2017

The latest minikube VM isn't actually based off of boot2docker any more. You can find the buildroot documentation online.

What I did was add the appropriate crt to both the /etc/ssl/certs directory and the /etc/ssl/certs/ca-certificates.crt by adding them to the overlay in the current repo. You can find it at deploy/iso/minikube-iso/board/coreos/minikube/rootfs-overlay/etc/. Create a ssl directory and drop your crts in.

@isuftin

This comment has been minimized.

Copy link
Author

commented Jun 7, 2017

@calaniz I will give this a shot.

@jwflory

This comment has been minimized.

Copy link
Contributor

commented Jun 28, 2017

For what it's worth, I tried following the advice of @calaniz above, but it didn't work for me on boot. I dropped my certs into /etc/ssl/certs/ and appended them to ca-certificates.crt, but the system isn't picking them up by default, e.g. running curl https://www.google.com. If I specify the path manually (--cacert /etc/ssl/certs/ca-certificates.crt), then it works as expected.

Trying to figure out a way to get this to work in Buildroot.

@jwflory

This comment has been minimized.

Copy link
Contributor

commented Jun 29, 2017

Hmm, so I tried doing this again today and I think there might have been an issue with the certs I had on my local system. Reinstalling the ca-certificates package on my own system, rebuilding the ISO with the Makefiles, and running it in Minikube solved this issue. So it seems like @calaniz's solution also works for me after all.

Thanks for your help in the Slack yesterday, @r2d4!

@chino

This comment has been minimized.

Copy link

commented Aug 22, 2017

Is there going to be an easy way to do this at install/runtime without the need to rebuild ISOs and other steps?

@ztripez

This comment has been minimized.

Copy link

commented Sep 12, 2017

We are unable to run our docker repo without a cert and it's not exposed to the internet, iow we must use a self signed cert.
If there is no way to add the cert to minikube without building a new ISO makes it very hard for us to using minikube/kubernetes.

@dkirrane

This comment has been minimized.

Copy link

commented Oct 19, 2017

Could someone provide the steps to create the minikube ISO with my certs and how to pass it to minikube start?

@jwflory

This comment has been minimized.

Copy link
Contributor

commented Oct 19, 2017

@dkirrane I actually forgot I was supposed to write documentation for this. A few months ago, I added a flag to Minikube that lets you specify a remote volume for generating the ISO (if you're making it from scratch).

See: #1657

Unfortunately, I no longer have access to the workstation / network where I was doing this development work. But if I remember right, you should be able to use the ISO_DOCKER_EXTRA_ARGS environment variable to specify a path to a volume, like…

$ ISO_DOCKER_EXTRA_ARGS='-v /path/to/certs/:/etc/ssl/certs/

Hopefully this is helpful for you. Sorry I can't test this further anymore.

@ChristofferNicklassonLenswayGroup

This comment has been minimized.

Copy link

commented Dec 12, 2017

Hi,

Docker wants to have the certs for repository's in /etc/docker/certs.d/site/ca.crt
Would like to see that path being persistent.

@fejta-bot

This comment has been minimized.

Copy link

commented Mar 12, 2018

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@erip

This comment has been minimized.

Copy link

commented Apr 9, 2018

Any update on this? I'm not sure if I can do this, but /remove-lifecycle stale

@fejta-bot

This comment has been minimized.

Copy link

commented May 9, 2018

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

@dkirrane

This comment has been minimized.

Copy link

commented May 9, 2018

Still an issue /remove-lifecycle rotten

@certifiedwaif

This comment has been minimized.

Copy link

commented May 15, 2018

This is still an issue for me. /remove-lifecycle rotten

@ghandim

This comment has been minimized.

Copy link

commented Jun 6, 2018

This is also still an issue for me. I found no real sophisticated solution for this problem at the moment.

@nrupendesai

This comment has been minimized.

Copy link

commented Jun 26, 2018

This is still an issue for me. If we have any solution where I can inject my SSL would be very helpful.

@mynameisD

This comment has been minimized.

Copy link

commented Jul 18, 2018

still an issue for me

@sean-dooher

This comment has been minimized.

Copy link

commented Aug 1, 2018

I tried all of the above to no avail, it's still an issue for me

@rhetzler

This comment has been minimized.

Copy link

commented Aug 13, 2018

this is an issue with me as well

@edwinpjacques

This comment has been minimized.

Copy link

commented Aug 16, 2018

This is still an issue for me. /remove-lifecycle rotten

@LupusUmbrae

This comment has been minimized.

Copy link

commented Aug 24, 2018

You can get the SSL into minikube by adding it to the files directory (at least on linux)

If you go to ~/.minikube/files these are copied onto the VM, so to copy a self signed SSL into docker you can just create the folder .minikube/files/etc/docker/certs.d/<domain>/ and drop in the certificate as a ca.crt

I did have this all working awhile ago, wrote some docs on it and then it all stopped working :(, so I have a feeling that I also need a way to add the self signed cert into the ca-certificates or something

@nrupendesai

This comment has been minimized.

Copy link

commented Aug 24, 2018

This steps worked for me.

  1. Download files related to ingress from github at https://github.com/nrupendesai/minikube/tree/master/deploy/addons/ingress
  2. Disable minikube ingress addon.
  3. Create new ingress deployment from downloaded files.
  4. Create secret for ssl certificate using file. This has to be created in kube-system name space.
    kubectl create secret tls tls-certificate --key key.pem --cert cert.pem -n kube-system
  5. Specify secret information in ingress file.
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: colortokens-colormaster
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.org/ssl-services: "<name of services separated by comma>"
    nginx.ingress.kubernetes.io/proxy-body-size: "100m"
spec:
  tls:
    - hosts:
      secretName: tls-certificate
@smiron

This comment has been minimized.

Copy link

commented Sep 18, 2018

/remove-lifecycle rotten

@mvaldesdeleon

This comment has been minimized.

Copy link

commented Sep 26, 2018

The following just worked for me, so I'm sharing it in case anybody else can benefit from it.

Following on @LupusUmbrae recommendation (which I tried but did not work for me), I settled by providing the PEM-encoded certificate file for the private registry in the following path:

~/.minikube/files/etc/ssl/certs/private-registry.example.com.pem

I set this file with 750 permissions just in case, recreated the VM and was able to connect to the registry without any further issues.

minikube version: v0.28.2
OSX 10.11.6 (15G19009)

@fejta-bot

This comment has been minimized.

Copy link

commented Dec 25, 2018

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@LupusUmbrae

This comment has been minimized.

Copy link

commented Dec 26, 2018

As @mvaldesdeleon commented if you add the SSL to the files it works, for me it turns out I needed the FQDN not just the domain root.

A short document on this would help I think

@ghandim

This comment has been minimized.

Copy link

commented Dec 27, 2018

Seems also to work for me with the hint from @mvaldesdeleon.

@tstromberg tstromberg changed the title Unable to add corporate root SSL certificate to minikube VM for VirtualBox Document how to add corporate root SSL certificates Jan 23, 2019

@fejta-bot

This comment has been minimized.

Copy link

commented Apr 29, 2019

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.