Problems running Kubernetes in Docker for Windows

[jseldess]

Jesse Seldess (jseldess) commented:

Moved from #3184.

From @jazoom:

I have been using a CockroachDB cluster in Swarm Mode in Docker for Windows for a few months now and it has worked well (was using volumes). I'd never tried direct path mapping to Windows file system. What's actually the difference that makes one work and not the other? It seems docker volumes are pretty much the same thing except docker keeps the data in a special directory on the host that you don't need to know the path of.

Now I'm moving everything over to Kubernetes in Docker for Windows and it seems hostPath isn't working for CockroachDB. I haven't found an equivalent of the docker volumes here that work around this issue. What can we do in Kubernetes?

...

Circling back to answer my own question in case it helps someone else in future.

This is how I got around this issue in Kubernetes in Docker for Windows.

apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: cockroachdb
spec:
  replicas: 1
  template:
    spec:
      volumes:
      - name: data
        $patch: delete
      - name: data
        emptyDir: {}
      - name: dump
        hostPath:
          path: /E/kubernetes-volumes/cockroachdb-dump
          type: DirectoryOrCreate
      initContainers:
      - name: restore-data
        image: busybox:1.29.2
        command: ['sh', '-c', 'if [ -e /cockroach/dump/data/ ]; then echo "restoring data..." && cp -rf /cockroach/dump/data/. /cockroach/cockroach-data/ && echo "data restored"; fi']
        volumeMounts:
        - name: data
          mountPath: /cockroach/cockroach-data
        - name: dump
          mountPath: /cockroach/dump
      containers:
      - name: cockroachdb
        volumeMounts:
        - name: dump
          mountPath: /cockroach/dump
        command:
        - "/bin/bash"
        - "-ecx"
        - "exec /cockroach/cockroach start --insecure --logtostderr --cache 25% --max-sql-memory 25% --http-host 0.0.0.0"
        lifecycle:
          preStop:
            exec:
              command:  ['sh', '-c', 'echo "dumping data..." && cp -r /cockroach/cockroach-data/ /cockroach/dump/data-temp/ && rm -rf /cockroach/dump/data/ && mv /cockroach/dump/data-temp/ /cockroach/dump/data/ && echo "data dumped"']

The above is actually the file I use to patch the production cockroachdb.yaml with Kustomize (which is where $patch: delete comes from, which isn't valid Kubernetes config YAML). It essentially shows the changes that need to be made to a normal cockroachdb config.

It's a little hacky, but it seems to work well. After the container has received the signal to terminate it saves CockroachDB's data to a hostPath. Before the pod starts next time an initContainer runs and restores that data to CockroachDB's data directory. I used an initContainer instead of a postStart hook because postStart runs in parallel to the container's entry command.

This gets around the issue because CockroachDB doesn't ever have to do its operations on a hostPath, yet the data is still persisted to a hostPath between pod restarts.

Jira Issue: DOC-235

[jseldess]

Follow-up from @dlouwers:

Is there a proposed workaround like a certain modification to the cockroachdb-statefulset-secure.yaml file to make this work on WIndows?

Follow-up from @xambas84:

sorry for the bump. but are there any plans to fix this?

[jseldess]

Adding @rolandcrosby for PM support.

[jseldess]

Moving this to later in favor of other Kubernetes-centric docs projects for 20.1. We can reassess/reprioritize as needed.

[github-actions]

We have marked this issue as stale because it has been inactive for
18 months. If this issue is still relevant, removing the stale label
or adding a comment will keep it active. Otherwise, we'll close it in
10 days to keep the issue queue tidy. Thank you for your contribution
to CockroachDB docs!