Allow for full local development with minikube

[rodcloutier]

When using minikube, we can re-use the docker daemon that is running on the VM without having to use an external registry, as stated in the documentation:

When using a single VM of Kubernetes, it’s really handy to reuse the minikube’s built-in Docker daemon; as this means you don’t have to build a docker registry on your host machine and push the image into it - you can just build inside the same docker daemon as minikube which speeds up local experiments.

Would it be possible to replicate this experience with draft and document how to do it?

[rodcloutier]

As I understand it right now, the images are built on the draftd service running in Kubernertes.
If we are using the docker daemon provided by Kubernetes, as I imagine, maybe we could just prevent the push?

If so, what would be the best way to control this? An argument?

I will look into this.

[bacongobbler]

As long as you're pointing the draft client at the minikube cluster, you are running "locally". Asides from the configured registry (which you'd need to set up in your local network) and any deps your docker image requires to build, you can be completely offline. Just point your kubectl client to your minikube cluster (done automatically when you call minikube create) and everything is all set up.

[bacongobbler]

In other words, if you're following the getting started guide you should already be offline. Perhaps some more clarity on your question would better shed some light on the problem you're having.

[rodcloutier]

Actually, if you do not setup a docker registry, you cannot use draft. It will always try to push and fail. We might not want to always push. I think the first experience should not rely on the fact that you must have an account in an external registry. We are also reluctant to have each team member push all their images all the time to the registry while developing. Maybe we shouldn't be.

On a broader topic, come to think of it, do we really need to push to a registry if this is a developer tool? The documentation seems to suggest that we should probably not use draft in the CI pipeline. As such it would make sense to build the image in the CI and push it "officially" there.

Edit: Ok, after some thoughts, this only applies to minikube which has only one node. Any other cluster must push to a registry.

[bacongobbler]

At the present moment, there are only two ways to distribute images to every kubelet in a cluster (single-node clusters are still clusters in my mind):

1. Push an image to a central Docker registry that is available to every kubelet
2. After building, distribute the image to every kubelet node.

Option 2 is problematic for 2 reasons:

1. It is not The Docker Way to distribute images
2. Setting up glusterfs, ceph, etc. and then maintaining a distributed file system is too difficult for most users (ask me how I know)

So given those issues there is really only one option: push to a registry. Now we originally toyed with deploying a local docker registry when installing draft but that also turned out to be difficult to install and configure. See #82 for more background on that.

If you happen to have a solution to fix #82 then that would probably alleviate most of your concerns. Is that correct?

[rodcloutier]

Yes it would be a welcome addition and address most of our concerns.

But since minikube already introduces the notion of sharing the docker daemon, I thought something similar might be acceptable when using draft with minikube. We fell that this would make it really easy to work locally and see our images, manage them, etc. That's a common workflow for us with minikube.

I did a proof of concept by introducing a skip-image-push option to the up command. It works fine since we have only one kubelet using one daemon.

[bacongobbler]

See Azure/draft#104 (comment) for my reasoning on why I don't enjoy the idea of skipping the image push option flag.

[rodcloutier]

Thanks for taking time to discuss this. I would like to keep discussing about this since this is somewhat shaking the foundation of our current workflow.

Currently, we have leveraged the functionalities offered by Minikube in our workflow. We didn't have Draft, nor did we want to write it a this time. We were doing everything locally on Minikube and once ready we are pushing to the registry.

One big advantage is that we do not need to manage the image registry URL since we can reuse the same URL that we built locally, by using IfNotPresent image pull policy.

Maybe this is not a common case and we are going down the wrong road?

Question I have for the Draft team:

• What is the best strategy to deal with all the versions that will be pushed to the registry?
• If the recommendation is to host a local registry (RFC: adding built-in docker registry , do not dependent thirdparty docker registry #82), we probably have to manage multiple registry URL. Do you plan to help support registry URL management (documentation, strategy, best practices)?

If anyone as thought on this I would be more than welcome to discuss. Are there any slack channel that I could move this discussion?

Thanks!

[bacongobbler]

What is the best strategy to deal with all the versions that will be pushed to the registry?

For myself, I just use Azure Container Registry so space concerns are not a problem. It's more-or-less a "fire and forget".

If the recommendation is to host a local registry (#82), ...

To clarify, I am arguing against hosting a local registry unless the user understands how to administer a docker registry. I am personally in favour of providing documentation on how to connect to an in-house registry, but not bundle it with draft out of the box similar to how we don't bundle kubernetes/tiller either. I don't want to provide support for kubernetes, helm/tiller, docker registries and draft all at the same time. We've gone down that path before, and it isn't feasible unless you have an entire org to do community support. So far it's just me and a few helping hands from other Azure/Deis-related projects.

If anyone as thought on this I would be more than welcome to discuss. Are there any slack channel that I could move this discussion?

See #85. I just got back from a 2-week vacation so I'm just catching up on things now. We're still ironing out the kinks :)

[bacongobbler]

following up on this: there is now a slack channel available at the kubernetes slack org. Join up in #draft-users :)

[bacongobbler]

You might want to check out #154 as it allows for full local development using minikube in the sense that

• it uses a built-in docker registry add-on
• there are no external dependencies (save for fetching images)

That should solve your use case, but let me know if I missed a particular point.