Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

windows: Linux builds don't work on windows filesystems (via docker/WSL) #3184

Open
ssmun opened this Issue May 22, 2018 · 15 comments

Comments

Projects
None yet
10 participants
@ssmun
Copy link

ssmun commented May 22, 2018

I tried to test CockroachDB and followed instructions on this link: https://www.cockroachlabs.com/docs/stable/start-a-local-cluster-in-docker.html#os-windows

ERROR: failed to start server: could not create temp storage: could not open rocksdb instance: IO error: While fsync: a directory: Invalid argument

Log file:

I180522 20:22:08.011632 1 util/log/clog.go:1134 [config] file created at: 2018/05/22 20:22:08
I180522 20:22:08.011632 1 util/log/clog.go:1134 [config] running on machine: roach1
I180522 20:22:08.011632 1 util/log/clog.go:1134 [config] binary: CockroachDB CCL v2.0.2 (x86_64-unknown-linux-gnu, built 2018/05/21 14:55:20, go1.10)
I180522 20:22:08.011632 1 util/log/clog.go:1134 [config] arguments: [/cockroach/cockroach start --insecure]
I180522 20:22:08.011632 1 util/log/clog.go:1134 line format: [IWEF]yymmdd hh:mm:ss.uuuuuu goid file:line msg utf8=✓
W180522 20:22:08.011625 1 cli/start.go:909 RUNNING IN INSECURE MODE!

  • Your cluster is open for any client that can access .
  • Any user, even root, can log in without providing a password.
  • Any user, connecting as root, can read or write any data in your cluster.
  • There is no network encryption nor authentication, and thus no confidentiality.

Check out how to secure your cluster: https://www.cockroachlabs.com/docs/v2.0/secure-a-cluster.html
I180522 20:22:08.046850 1 server/config.go:330 available memory from cgroups (8.0 EiB) exceeds system memory 1.9 GiB, using system memory
W180522 20:22:08.066892 1 cli/start.go:812 Using the default setting for --cache (128 MiB).
A significantly larger value is usually needed for good performance.
If you have a dedicated server a reasonable setting is --cache=.25 (495 MiB).
I180522 20:22:08.067224 1 server/config.go:330 available memory from cgroups (8.0 EiB) exceeds system memory 1.9 GiB, using system memory
W180522 20:22:08.067282 1 cli/start.go:825 Using the default setting for --max-sql-memory (128 MiB).
A significantly larger value is usually needed in production.
If you have a dedicated server a reasonable setting is --max-sql-memory=.25 (495 MiB).
I180522 20:22:08.067330 1 cli/start.go:923 CockroachDB CCL v2.0.2 (x86_64-unknown-linux-gnu, built 2018/05/21 14:55:20, go1.10)
I180522 20:22:08.296553 1 server/config.go:330 available memory from cgroups (8.0 EiB) exceeds system memory 1.9 GiB, using system memory
I180522 20:22:08.296800 1 server/config.go:430 system total memory: 1.9 GiB
I180522 20:22:08.297016 1 server/config.go:432 server configuration:
max offset 500000000
cache size 128 MiB
SQL memory pool size 128 MiB
scan interval 10m0s
scan max idle time 200ms
event log enabled true
I180522 20:22:08.297153 1 cli/start.go:789 using local environment variables: COCKROACH_CHANNEL=official-docker
I180522 20:22:08.297233 1 cli/start.go:796 process identity: uid 0 euid 0 gid 0 egid 0
I180522 20:22:08.297314 1 cli/start.go:461 starting cockroach node
I180522 20:22:08.317992 21 storage/engine/rocksdb.go:552 opening rocksdb instance at "/cockroach/cockroach-data/cockroach-temp451412792"
E180522 20:22:08.454632 1 cli/error.go:109 failed to start server: could not create temp storage: could not open rocksdb instance: IO error: While fsync: a directory: Invalid argument
Error: failed to start server: could not create temp storage: could not open rocksdb instance: IO error: While fsync: a directory: Invalid argument

@quartex

This comment has been minimized.

Copy link

quartex commented May 30, 2018

Hi,
the same for me on Windows 8 with Docker 18.03.01

docker run -d --name=roach1 --hostname=roach1 --net=roachnet -p 26257:26257 -p 8080:8080 -v "D:\docker\cockroach\data1:/cockroach/cockroach-data" cockroachdb/cockroach:v2.0.2 start --insecure

report me

  • ERROR: failed to start server: could not create temp storage: could not open rocksdb instance: IO error: While fsync: a directory: Invalid argument

Failed running "start"

@jseldess

This comment has been minimized.

Copy link
Contributor

jseldess commented May 30, 2018

Thanks for the report, @ssmun and @quartex.

@BramGruneir, @bdarnell, @a-robinson, based on the provided logs, do you have ideas about what's going on?

@jseldess jseldess added this to the 2.1 milestone May 30, 2018

@bdarnell

This comment has been minimized.

Copy link
Member

bdarnell commented May 30, 2018

Our linux builds don't currently work very well on windows filesystems (this affects both the windows linux subsystem and docker). We recommend using the native windows binaries on windows. Or if you want to run linux on windows for some reason, create and mount a linux filesystem instead of having them write directly to your windows filesystem.

@bdarnell bdarnell changed the title CockroachDB exits as soon as it starts on Docker on Windows windows: Linux builds don't work on windows filesystems (via docker/WSL) May 30, 2018

@quartex

This comment has been minimized.

Copy link

quartex commented May 31, 2018

Hi,
I had the same problem with Mysql with Direct_io settings that use fsync.
Using a docker-compose with all 3 nodes settings inside it and using VOLUMES instead directory remapping for storage all works on Windows 10 with last version of Docker!

I'm using a docker.compose.yaml like this

`
version: '3'

services:

roach1:
volumes:
#- D:\docker\cockroach\data1:/cockroach/cockroach-data
- data-volume1:/cockroach/cockroach-data
image: cockroachdb/cockroach:v2.0.2
command: start --insecure
ports:
- 26257:26257
- 8080:8080
networks:
- roachnet
roach2:
volumes:
#- D:\docker\cockroach\data2:/cockroach/cockroach-data
- data-volume2:/cockroach/cockroach-data
image: cockroachdb/cockroach:v2.0.2
depends_on:
- roach1
command: start --insecure --join=roach1
networks:
- roachnet
roach3:
volumes:
#- D:\docker\cockroach\data3:/cockroach/cockroach-data
- data-volume3:/cockroach/cockroach-data
image: cockroachdb/cockroach:v2.0.2
depends_on:
- roach1
command: start --insecure --join=roach1
networks:
- roachnet
networks:
roachnet:
volumes:
data-volume1:
data-volume2:
data-volume3:
`
Thank you

@BramGruneir

This comment has been minimized.

Copy link
Member

BramGruneir commented May 31, 2018

@quartex Wonderful!

We'll try that out and update our docs accordingly.

@jazoom

This comment has been minimized.

Copy link

jazoom commented Aug 17, 2018

I have been using a CockroachDB cluster in Swarm Mode in Docker for Windows for a few months now and it has worked well (was using volumes). I'd never tried direct path mapping to Windows file system. What's actually the difference that makes one work and not the other? It seems docker volumes are pretty much the same thing except docker keeps the data in a special directory on the host that you don't need to know the path of.

Now I'm moving everything over to Kubernetes in Docker for Windows and it seems hostPath isn't working for CockroachDB. I haven't found an equivalent of the docker volumes here that work around this issue. What can we do in Kubernetes?

@jazoom

This comment has been minimized.

Copy link

jazoom commented Aug 18, 2018

Circling back to answer my own question in case it helps someone else in future.

This is how I got around this issue in Kubernetes in Docker for Windows.


apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: cockroachdb
spec:
  replicas: 1
  template:
    spec:
      volumes:
      - name: data
        $patch: delete
      - name: data
        emptyDir: {}
      - name: dump
        hostPath:
          path: /E/kubernetes-volumes/cockroachdb-dump
          type: DirectoryOrCreate
      initContainers:
      - name: restore-data
        image: busybox:1.29.2
        command: ['sh', '-c', 'if [ -e /cockroach/dump/data/ ]; then echo "restoring data..." && cp -rf /cockroach/dump/data/. /cockroach/cockroach-data/ && echo "data restored"; fi']
        volumeMounts:
        - name: data
          mountPath: /cockroach/cockroach-data
        - name: dump
          mountPath: /cockroach/dump
      containers:
      - name: cockroachdb
        volumeMounts:
        - name: dump
          mountPath: /cockroach/dump
        command:
        - "/bin/bash"
        - "-ecx"
        - "exec /cockroach/cockroach start --insecure --logtostderr --cache 25% --max-sql-memory 25% --http-host 0.0.0.0"
        lifecycle:
          preStop:
            exec:
              command:  ['sh', '-c', 'echo "dumping data..." && cp -r /cockroach/cockroach-data/ /cockroach/dump/data-temp/ && rm -rf /cockroach/dump/data/ && mv /cockroach/dump/data-temp/ /cockroach/dump/data/ && echo "data dumped"']

The above is actually the file I use to patch the production cockroachdb.yaml with Kustomize (which is where $patch: delete comes from, which isn't valid Kubernetes config YAML). It essentially shows the changes that need to be made to a normal cockroachdb config.

It's a little hacky, but it seems to work well. After the container has received the signal to terminate it saves CockroachDB's data to a hostPath. Before the pod starts next time an initContainer runs and restores that data to CockroachDB's data directory. I used an initContainer instead of a postStart hook because postStart runs in parallel to the container's entry command.

This gets around the issue because CockroachDB doesn't ever have to do its operations on a hostPath, yet the data is still persisted to a hostPath between pod restarts.

@BramGruneir

This comment has been minimized.

Copy link
Member

BramGruneir commented Oct 15, 2018

@rmloveland This needs to be tested and our docs need to be updated, we're leading users astray.

@jazoom

This comment has been minimized.

Copy link

jazoom commented Oct 15, 2018

Also I should add that my solution above does have a problem, so there still isn't a good solution to this. The problem is that the preStop hook doesn't seem to run when shutting down Docker or Windows, which means I need to manually delete the CockroachDB pod if I want the current state persisted to disk. It's pretty annoying.

@jseldess jseldess added the A-ops&tools label Nov 9, 2018

@jseldess

This comment has been minimized.

Copy link
Contributor

jseldess commented Nov 9, 2018

@lhirata, @rolandcrosby, this problem has come up several time, so we may want to prioritize a docs update for 2.2.

@jseldess jseldess added the O-community label Nov 9, 2018

@jseldess jseldess assigned lhirata and unassigned rmloveland Nov 9, 2018

@jseldess jseldess modified the milestones: 2.1, 2.2 Nov 9, 2018

@lhirata lhirata added the P-1 label Nov 15, 2018

@dlouwers

This comment has been minimized.

Copy link

dlouwers commented Jan 2, 2019

Is there a proposed workaround like a certain modification to the cockroachdb-statefulset-secure.yaml file to make this work on WIndows?

@xambas84

This comment has been minimized.

Copy link

xambas84 commented Feb 7, 2019

sorry for the bump. but are there any plans to fix this?

@jseldess

This comment has been minimized.

Copy link
Contributor

jseldess commented Feb 7, 2019

Thanks for the bump, @xambas84. I'm looking into this now. I think this issue encompasses two separate issues. I'll update again soon.

@jseldess

This comment has been minimized.

Copy link
Contributor

jseldess commented Feb 7, 2019

@jazoom, @dlouwers, @xambas84, I've split the Kubernetes-specific problem into a new issue: #4333. We're planning to update our docs for the Docker problem for our April release. We'll aim to address the Kubernetes issue as well, if time allows.

@jseldess

This comment has been minimized.

Copy link
Contributor

jseldess commented Feb 7, 2019

For this issue:

  • Remove docker and kubernetes from the install-cockroachdb-windows page and replace with suggestion that you can run a linux vm.
  • Figure out and document more complete instructions (exploring @quartex's docker-compose suggestion first).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.