Better documentation for running skaffold in docker

[nkubala]

we should put together a small quickstart and best practice guide for running skaffold through the provided docker image. we could give quick examples for running locally in the docker daemon, running on a cluster building with kaniko (locally or remote), and running in a cloud build pipeline.

[dalbani]

I'm very much interested in having such documentation. Because at that point I have no idea where to start from. Where's this "provided Docker image" by the way?

[dalbani]

Oh, I found it, nevermind: https://skaffold.dev/docs/install/ and then the Docker tab.

[amitkarpe]

Hi @nkubala ,

Where I can add example/docs for "running skaffold in docker"?
Following location for examples will be right for this?
https://github.com/GoogleContainerTools/skaffold/tree/master/examples

[nkubala]

@amitkarpe I think this particular example would be better as a docs page, maybe either https://skaffold.dev/docs/quickstart/ or https://skaffold.dev/docs/tutorials/. the content for these pages lives at https://github.com/GoogleContainerTools/skaffold/tree/master/docs/content/en/docs

[dalbani]

Speaking of using the Docker image of skaffold, am I seeing this correctly or the image is huge?
A virtual size of 934.72 MB is for example reported for v1.12.1 on https://console.cloud.google.com/gcr/images/k8s-skaffold/GLOBAL/skaffold@sha256:0a475a9b6dead1f176af3fc30ff09e03d427f453fbf845504f0f5d971fc948f2/details?tab=info.
Where's the catch?

[nkubala]

@dalbani unfortunately our Docker images are bigger than we'd like, because we bundle in all required dependencies for the different build/deploy combinations we natively support. we've considered offering "slim" versions of our images (which would only bundle "common" dependencies e.g. docker/kubectl), but haven't had a chance to do that yet.

[waaghals]

As I was struggling quite a bit with running skaffold from docker. I will share my findings as I'm sure it will be helpful for others.
My example uses minikube but I expect configuration for other clusters to be similar.

This all gets rather complicated real quick. But my goal is to create a bash alias which will be sourced in bash. This way I can be sure that all the developers run the exact same skaffold version and that it is compatible with the project. (as the aliases are checked in)

# Preconfigure some environment variables based on defaults. This allows the developer to override environment variables to configure the configuration and cache directories.
SKAFFOLD_CACHE=${SKAFFOLD_CACHE:-$HOME/.skaffold/cache} && \
SKAFFOLD_CONFIG=${SKAFFOLD_CONFIG:-$HOME/.skaffold/config} && \
KUBECONFIG=${KUBECONFIG:-$HOME/.kube/config} && \
MINIKUBE_HOME=${MINIKUBE_HOME:-$HOME/.minikube} && \
MINIKUBE_PROFILE=${MINIKUBE_PROFILE:-minikube} && \

# For minikube it is important to configure the docker client to communicate with the docker engine in minikube.
eval $(minikube -p ${MINIKUBE_PROFILE} docker-env) && \

docker run --rm -it \
# Mount all the directories which skaffold will read from
--volume ${PWD}:/data \
--volume ${SKAFFOLD_CACHE}:${SKAFFOLD_CACHE} \
--volume ${SKAFFOLD_CONFIG}:${SKAFFOLD_CONFIG} \
--volume ${KUBECONFIG}:${KUBECONFIG}:ro \
--volume ${MINIKUBE_HOME}:${MINIKUBE_HOME}:ro \
--workdir /data \
# Use the environment as the developer configured it
--env SKAFFOLD_CONFIG=${SKAFFOLD_CONFIG} \
--env KUBECONFIG=${KUBECONFIG} \
--env MINIKUBE_HOME=${MINIKUBE_HOME} \
# Manually configure the docker-env file variables from minikube docker-env, these are not inherited from the host
--env DOCKER_TLS_VERIFY=${DOCKER_TLS_VERIFY} \
--env DOCKER_HOST=${DOCKER_HOST} \
--env DOCKER_CERT_PATH=${DOCKER_CERT_PATH} \
--env MINIKUBE_ACTIVE_DOCKERD=${MINIKUBE_ACTIVE_DOCKERD} \
gcr.io/k8s-skaffold/skaffold:latest skaffold

Another important step when using minikube is to consider that minikube vm does not have access to the host's files.
Because eval $(minikube docker-env) is ran just before docker run ... will mean that the container will be started in minikube's docker engine. For the files to be available in minikube vm, the files need to be mounted in minikube when starting.
minikube start --mount --mount-string "${HOME}:${HOME}"
In my example all files are within the developer's home directory so minikube only needs that directory mounted.

A more simpler example (but makes a lot of assumptions), also I did not test this.

eval $(minikube docker-env) && \

docker run --rm -it \
--volume ${PWD}:/data \
--volume ${HOME}:${HOME} \
--workdir /data \
--env DOCKER_TLS_VERIFY=${DOCKER_TLS_VERIFY} \
--env DOCKER_HOST=${DOCKER_HOST} \
--env DOCKER_CERT_PATH=${DOCKER_CERT_PATH} \
--env MINIKUBE_ACTIVE_DOCKERD=${MINIKUBE_ACTIVE_DOCKERD} \
gcr.io/k8s-skaffold/skaffold:latest skaffold

---

I think the following changes to the image might make it easier to use and document.

1. Make it easier to configure docker-env so docker run ... is not ran within minikube, making volume mounting easier.
2. Create an SKAFFOLD_HOME environment variable to ~/.skaffold. No more need to configure cache and config separately.
3. Configure entrypoint so the image can be used as it is a binary.
4. Configure workdir

[MarlonGamez]

commenting here to make triage party happy

[ipv1337]

Has anyone gotten this approach to work with Bazel?

I've tried but have been unsuccessful at any attempts so I was curious if anyone else has done so or at least attempted it.