New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Does not appear to work in a Linux Docker container on Windows #9833

Closed
Frogmore42 opened this Issue May 11, 2018 · 1 comment

Comments

Projects
None yet
2 participants
@Frogmore42

Frogmore42 commented May 11, 2018

Bug report

System info: [Include InfluxDB version, operating system name, and other relevant details]
ts=2018-05-11T19:13:53.903815Z lvl=info msg="InfluxDB starting" log_id=0809pOwl000 version=1.5.2 branch=1.5 commit=02d7d4f043b34ecb4e9b2dbec298c6f9450c2a32
ts=2018-05-11T19:13:53.903975Z lvl=info msg="Go runtime" log_id=0809pOwl000 version=go1.9.2 maxprocs=2

Windows 10 Pro 1709 (build 16299.431)
Docker 18.03.1-ce

Steps to reproduce:

  1. Using Docker-Compose to create a container for influxdb and another for grafana
  2. Using Telgraf for Windows to write to influxdb
  3. Works fine from a clean state
  4. Restart computer
  5. Wait for Docker to be ready
  6. docker-compose up

__Expected behavior:
grafana to show the data from earlier when the system was up with a gap for the time it was down

__Actual behavior:
while the logs indicate influxdb shut down gracefully, it was unable to start correctly.

Additional info: [Include gist of relevant config, logs, etc.]
[httpd] 172.21.0.1 - - [11/May/2018:19:09:53 +0000] "POST /write?db=telegraf HTTP/1.1" 204 0 "-" "telegraf" e28dffb0-554e-11e8-84ba-000000000000 24829
[httpd] 172.21.0.1 - - [11/May/2018:19:10:03 +0000] "POST /write?db=telegraf HTTP/1.1" 204 0 "-" "telegraf" e883a87b-554e-11e8-84bb-000000000000 26254
[httpd] 172.21.0.1 - - [11/May/2018:19:10:13 +0000] "POST /write?db=telegraf HTTP/1.1" 204 0 "-" "telegraf" ee79a8a4-554e-11e8-84bc-000000000000 25816
ts=2018-05-11T19:11:16.518646Z lvl=info msg="Signal received, initializing clean shutdown..." log_id=07z8gGl000
ts=2018-05-11T19:11:16.518681Z lvl=info msg="Waiting for clean shutdown..." log_id=07
z8gGl000
ts=2018-05-11T19:11:16.518701Z lvl=info msg="Shutting down monitor service" log_id=07z8gGl000 service=monitor
ts=2018-05-11T19:11:16.518753Z lvl=info msg="Listener closed" log_id=07
z8gGl000 service=snapshot
ts=2018-05-11T19:11:16.518760Z lvl=info msg="Terminating storage of statistics" log_id=07z8gGl000 service=monitor
ts=2018-05-11T19:11:16.518769Z lvl=info msg="Terminating precreation service" log_id=07
z8gGl000 service=shard-precreation
ts=2018-05-11T19:11:16.518777Z lvl=info msg="Terminating continuous query service" log_id=07z8gGl000 service=continuous_querier
ts=2018-05-11T19:11:16.518789Z lvl=info msg="Closing retention policy enforcement service" log_id=07
z8gGl000 service=retention
ts=2018-05-11T19:13:53.903815Z lvl=info msg="InfluxDB starting" log_id=0809pOwl000 version=1.5.2 branch=1.5 commit=02d7d4f043b34ecb4e9b2dbec298c6f9450c2a32
ts=2018-05-11T19:13:53.903975Z lvl=info msg="Go runtime" log_id=0809pOwl000 version=go1.9.2 maxprocs=2
ts=2018-05-11T19:13:54.108029Z lvl=info msg="Using data dir" log_id=0809pOwl000 service=store path=/var/lib/influxdb/data
ts=2018-05-11T19:13:54.109450Z lvl=info msg="Open store (start)" log_id=0809pOwl000 service=store trace_id=0809pPkG000 op_name=tsdb_open op_event=start
ts=2018-05-11T19:13:54.303542Z lvl=info msg="Reading file" log_id=0809pOwl000 engine=tsm1 service=cacheloader path=/var/lib/influxdb/wal/_internal/monitor/1/_00001.wal size=4228820
ts=2018-05-11T19:13:54.816503Z lvl=info msg="Reading file" log_id=0809pOwl000 engine=tsm1 service=cacheloader path=/var/lib/influxdb/wal/telegraf/autogen/2/_00001.wal size=2391011
ts=2018-05-11T19:13:55.114607Z lvl=info msg="Failed to open shard" log_id=0809pOwl000 service=store trace_id=0809pPkG000 op_name=tsdb_open db_shard_id=2 error="[shard 2] sync /var/lib/influxdb/data/telegraf/autogen/2: invalid argument"
ts=2018-05-11T19:13:55.124054Z lvl=info msg="Failed to open shard" log_id=0809pOwl000 service=store trace_id=0809pPkG000 op_name=tsdb_open db_shard_id=1 error="[shard 1] sync /var/lib/influxdb/data/_internal/monitor/1: invalid argument"
ts=2018-05-11T19:13:55.124168Z lvl=info msg="Open store (end)" log_id=0809pOwl000 service=store trace_id=0809pPkG000 op_name=tsdb_open op_event=end op_elapsed=1014.784ms
ts=2018-05-11T19:13:55.124202Z lvl=info msg="Opened service" log_id=0809pOwl000 service=subscriber
ts=2018-05-11T19:13:55.124263Z lvl=info msg="Starting monitor service" log_id=0809pOwl000 service=monitor
ts=2018-05-11T19:13:55.124283Z lvl=info msg="Registered diagnostics client" log_id=0809pOwl000 service=monitor name=build
ts=2018-05-11T19:13:55.124289Z lvl=info msg="Registered diagnostics client" log_id=0809pOwl000 service=monitor name=runtime
ts=2018-05-11T19:13:55.124294Z lvl=info msg="Registered diagnostics client" log_id=0809pOwl000 service=monitor name=network
ts=2018-05-11T19:13:55.124501Z lvl=info msg="Registered diagnostics client" log_id=0809pOwl000 service=monitor name=system
ts=2018-05-11T19:13:55.124718Z lvl=info msg="Starting precreation service" log_id=0809pOwl000 service=shard-precreation check_interval=10m advance_period=30m
ts=2018-05-11T19:13:55.124738Z lvl=info msg="Starting snapshot service" log_id=0809pOwl000 service=snapshot
ts=2018-05-11T19:13:55.124743Z lvl=info msg="Starting continuous query service" log_id=0809pOwl000 service=continuous_querier
ts=2018-05-11T19:13:55.124902Z lvl=info msg="Starting HTTP service" log_id=0809pOwl000 service=httpd authentication=false
ts=2018-05-11T19:13:55.124911Z lvl=info msg="opened HTTP access log" log_id=0809pOwl000 service=httpd path=stderr
ts=2018-05-11T19:13:55.125379Z lvl=info msg="Listening on HTTP" log_id=0809pOwl000 service=httpd addr=[::]:8086 https=false
ts=2018-05-11T19:13:55.125420Z lvl=info msg="Starting retention policy enforcement service" log_id=0809pOwl000 service=retention check_interval=30m
ts=2018-05-11T19:13:55.124893Z lvl=info msg="Storing statistics" log_id=0809pOwl000 service=monitor db_instance=_internal db_rp=monitor interval=10s
ts=2018-05-11T19:13:55.133319Z lvl=info msg="Listening for signals" log_id=0809pOwl000
ts=2018-05-11T19:13:55.136564Z lvl=info msg="Sending usage statistics to usage.influxdata.com" log_id=0809pOwl000
ts=2018-05-11T19:14:10.024165Z lvl=info msg="Reading file" log_id=0809pOwl000 engine=tsm1 service=cacheloader path=/var/lib/influxdb/wal/_internal/monitor/1/_00001.wal size=4228820
ts=2018-05-11T19:14:10.657776Z lvl=info msg="Write failed" log_id=0809pOwl000 service=write shard=1 error="[shard 1] sync /var/lib/influxdb/data/_internal/monitor/1: invalid argument"
ts=2018-05-11T19:14:10.658092Z lvl=info msg="failed to store statistics" log_id=0809pOwl000 service=monitor error="[shard 1] sync /var/lib/influxdb/data/_internal/monitor/1: invalid argument"
ts=2018-05-11T19:14:20.051376Z lvl=info msg="Reading file" log_id=0809pOwl000 engine=tsm1 service=cacheloader path=/var/lib/influxdb/wal/_internal/monitor/1/_00001.wal size=4228820
ts=2018-05-11T19:14:20.306750Z lvl=info msg="Write failed" log_id=0809pOwl000 service=write shard=1 error="[shard 1] sync /var/lib/influxdb/data/_internal/monitor/1: invalid argument"
ts=2018-05-11T19:14:20.306961Z lvl=info msg="failed to store statistics" log_id=0809pOwl000 service=monitor error="[shard 1] sync /var/lib/influxdb/data/_internal/monitor/1: invalid argument"
ts=2018-05-11T19:14:30.038802Z lvl=info msg="Reading file" log_id=0809pOwl000 engine=tsm1 service=cacheloader path=/var/lib/influxdb/wal/_internal/monitor/1/_00001.wal size=4228820
ts=2018-05-11T19:14:30.308278Z lvl=info msg="Write failed" log_id=0809pOwl000 service=write shard=1 error="[shard 1] sync /var/lib/influxdb/data/_internal/monitor/1: invalid argument"
ts=2018-05-11T19:14:30.308347Z lvl=info msg="failed to store statistics" log_id=0809pOwl000 service=monitor error="[shard 1] sync /var/lib/influxdb/data/_internal/monitor/1: invalid argument"
ts=2018-05-11T19:14:40.042630Z lvl=info msg="Reading file" log_id=0809pOwl000 engine=tsm1 service=cacheloader path=/var/lib/influxdb/wal/_internal/monitor/1/_00001.wal size=4228820
ts=2018-05-11T19:14:40.295905Z lvl=info msg="Write failed" log_id=0809pOwl000 service=write shard=1 error="[shard 1] sync /var/lib/influxdb/data/_internal/monitor/1: invalid argument"
ts=2018-05-11T19:14:40.295961Z lvl=info msg="failed to store statistics" log_id=0809pOwl000 service=monitor error="[shard 1] sync /var/lib/influxdb/data/_internal/monitor/1: invalid argument"
ts=2018-05-11T19:14:50.048718Z lvl=info msg="Reading file" log_id=0809pOwl000 engine=tsm1 service=cacheloader path=/var/lib/influxdb/wal/_internal/monitor/1/_00001.wal size=4228820
ts=2018-05-11T19:14:50.363324Z lvl=info msg="Signal received, initializing clean shutdown..." log_id=0809pOwl000
ts=2018-05-11T19:14:50.363473Z lvl=info msg="Waiting for clean shutdown..." log_id=0809pOwl000
ts=2018-05-11T19:14:50.363636Z lvl=info msg="Listener closed" log_id=0809pOwl000 service=snapshot
ts=2018-05-11T19:14:50.425885Z lvl=info msg="Write failed" log_id=0809pOwl000 service=write shard=1 error="[shard 1] sync /var/lib/influxdb/data/_internal/monitor/1: invalid argument"
ts=2018-05-11T19:14:50.426024Z lvl=info msg="failed to store statistics" log_id=0809pOwl000 service=monitor error="[shard 1] sync /var/lib/influxdb/data/_internal/monitor/1: invalid argument"
ts=2018-05-11T19:14:50.426090Z lvl=info msg="Shutting down monitor service" log_id=0809pOwl000 service=monitor
ts=2018-05-11T19:14:50.426163Z lvl=info msg="Terminating storage of statistics" log_id=0809pOwl000 service=monitor
ts=2018-05-11T19:14:50.426208Z lvl=info msg="Terminating precreation service" log_id=0809pOwl000 service=shard-precreation
ts=2018-05-11T19:14:50.426254Z lvl=info msg="Terminating continuous query service" log_id=0809pOwl000 service=continuous_querier
ts=2018-05-11T19:14:50.426303Z lvl=info msg="Closing retention policy enforcement service" log_id=0809pOwl000 service=retention
ts=2018-05-11T19:14:50.432726Z lvl=info msg="Closed service" log_id=0809pOwl000 service=subscriber
ts=2018-05-11T19:14:50.433336Z lvl=info msg="Server shutdown completed" log_id=0809pOwl000

ts=2018-05-11T19:15:00.242190Z lvl=info msg="InfluxDB starting" log_id=0809tS4W000 version=1.5.2 branch=1.5 commit=02d7d4f043b34ecb4e9b2dbec298c6f9450c2a32
ts=2018-05-11T19:15:00.242498Z lvl=info msg="Go runtime" log_id=0809tS4W000 version=go1.9.2 maxprocs=2
ts=2018-05-11T19:15:00.345745Z lvl=info msg="Using data dir" log_id=0809tS4W000 service=store path=/var/lib/influxdb/data
ts=2018-05-11T19:15:00.351405Z lvl=info msg="Open store (start)" log_id=0809tS4W000 service=store trace_id=0809tSVl000 op_name=tsdb_open op_event=start
ts=2018-05-11T19:15:00.469529Z lvl=info msg="Reading file" log_id=0809tS4W000 engine=tsm1 service=cacheloader path=/var/lib/influxdb/wal/_internal/monitor/1/_00001.wal size=4228820
ts=2018-05-11T19:15:00.652798Z lvl=info msg="Reading file" log_id=0809tS4W000 engine=tsm1 service=cacheloader path=/var/lib/influxdb/wal/telegraf/autogen/2/_00001.wal size=2391011
ts=2018-05-11T19:15:00.974440Z lvl=info msg="Failed to open shard" log_id=0809tS4W000 service=store trace_id=0809tSVl000 op_name=tsdb_open db_shard_id=2 error="[shard 2] sync /var/lib/influxdb/data/telegraf/autogen/2: invalid argument"
ts=2018-05-11T19:15:01.044153Z lvl=info msg="Failed to open shard" log_id=0809tS4W000 service=store trace_id=0809tSVl000 op_name=tsdb_open db_shard_id=1 error="[shard 1] sync /var/lib/influxdb/data/_internal/monitor/1: invalid argument"
ts=2018-05-11T19:15:01.044297Z lvl=info msg="Open store (end)" log_id=0809tS4W000 service=store trace_id=0809tSVl000 op_name=tsdb_open op_event=end op_elapsed=692.898ms
ts=2018-05-11T19:15:01.044466Z lvl=info msg="Opened service" log_id=0809tS4W000 service=subscriber
ts=2018-05-11T19:15:01.044566Z lvl=info msg="Starting monitor service" log_id=0809tS4W000 service=monitor
ts=2018-05-11T19:15:01.044739Z lvl=info msg="Registered diagnostics client" log_id=0809tS4W000 service=monitor name=build
ts=2018-05-11T19:15:01.044886Z lvl=info msg="Registered diagnostics client" log_id=0809tS4W000 service=monitor name=runtime
ts=2018-05-11T19:15:01.045003Z lvl=info msg="Registered diagnostics client" log_id=0809tS4W000 service=monitor name=network
ts=2018-05-11T19:15:01.045124Z lvl=info msg="Registered diagnostics client" log_id=0809tS4W000 service=monitor name=system
ts=2018-05-11T19:15:01.045358Z lvl=info msg="Starting precreation service" log_id=0809tS4W000 service=shard-precreation check_interval=10m advance_period=30m
ts=2018-05-11T19:15:01.045625Z lvl=info msg="Starting snapshot service" log_id=0809tS4W000 service=snapshot
ts=2018-05-11T19:15:01.045794Z lvl=info msg="Starting continuous query service" log_id=0809tS4W000 service=continuous_querier
ts=2018-05-11T19:15:01.045947Z lvl=info msg="Starting HTTP service" log_id=0809tS4W000 service=httpd authentication=false
ts=2018-05-11T19:15:01.046141Z lvl=info msg="opened HTTP access log" log_id=0809tS4W000 service=httpd path=stderr
ts=2018-05-11T19:15:01.045527Z lvl=info msg="Storing statistics" log_id=0809tS4W000 service=monitor db_instance=_internal db_rp=monitor interval=10s
ts=2018-05-11T19:15:01.046711Z lvl=info msg="Listening on HTTP" log_id=0809tS4W000 service=httpd addr=[::]:8086 https=false
ts=2018-05-11T19:15:01.046756Z lvl=info msg="Starting retention policy enforcement service" log_id=0809tS4W000 service=retention check_interval=30m
ts=2018-05-11T19:15:01.049235Z lvl=info msg="Sending usage statistics to usage.influxdata.com" log_id=0809tS4W000
ts=2018-05-11T19:15:01.049334Z lvl=info msg="Listening for signals" log_id=0809tS4W000
ts=2018-05-11T19:15:20.021776Z lvl=info msg="Reading file" log_id=0809tS4W000 engine=tsm1 service=cacheloader path=/var/lib/influxdb/wal/_internal/monitor/1/_00001.wal size=4228820
ts=2018-05-11T19:15:20.258580Z lvl=info msg="Write failed" log_id=0809tS4W000 service=write shard=1 error="[shard 1] sync /var/lib/influxdb/data/_internal/monitor/1: invalid argument"
ts=2018-05-11T19:15:20.259115Z lvl=info msg="failed to store statistics" log_id=0809tS4W000 service=monitor error="[shard 1] sync /var/lib/influxdb/data/_internal/monitor/1: invalid argument"

Please note, the quickest way to fix a bug is to open a Pull Request.

This appears related to the issue raised in: #9630

It appears to work okay if I use a volume for influxdb's data, but I am not quite sure where that data really gets stored and how reliable/easy to manage (size/backup) that will be.

zeebo added a commit that referenced this issue May 15, 2018

attempt to work on docker on windows
multiple users have attempted to run influxdb in a docker container
with a windows host and a volume mounted from windows. that causes
problems because it apparently uses samba/cifs which does not
support fsync on directories. this patchset will, once, at startup,
check if it can fsync "/". if it receives an EINVAL as is what
appears to happen on samba/cifs, then it will not attempt to fsync
any more. this should help.

updates #9833.
updates #9630.

zeebo added a commit that referenced this issue May 16, 2018

attempt to work on docker on windows
multiple users have attempted to run influxdb in a docker container
with a windows host and a volume mounted from windows. that causes
problems because it apparently uses samba/cifs which does not
support fsync on directories. this patchset will, once, at startup,
check if it can fsync "/". if it receives an EINVAL as is what
appears to happen on samba/cifs, then it will not attempt to fsync
any more. this should help.

updates #9833.
updates #9630.
@zeebo

This comment has been minimized.

Member

zeebo commented May 30, 2018

Thanks for the bug report. I think I've pushed a fix in #9847. Is it possible for you to build a version of the binary with that commit and checking to see if it fixes your issue? If you'd like, I can also build a windows binary for you. That may help speed up the process to get it merged.

zeebo added a commit that referenced this issue Jun 1, 2018

attempt to work on docker on windows
multiple users have attempted to run influxdb in a docker container
with a windows host and a volume mounted from windows. that causes
problems because it apparently uses samba/cifs which does not
support fsync on directories. this patchset will, once, at startup,
check if it can fsync "/". if it receives an EINVAL as is what
appears to happen on samba/cifs, then it will not attempt to fsync
any more. this should help.

updates #9833.
updates #9630.

zeebo added a commit that referenced this issue Jun 1, 2018

attempt to work on docker on windows
multiple users have attempted to run influxdb in a docker container
with a windows host and a volume mounted from windows. that causes
problems because it apparently uses samba/cifs which does not
support fsync on directories. this patchset will, once, at startup,
check if it can fsync "/". if it receives an EINVAL as is what
appears to happen on samba/cifs, then it will not attempt to fsync
any more. this should help.

updates #9833.
updates #9630.

zeebo added a commit that referenced this issue Jun 1, 2018

attempt to work on docker on windows
multiple users have attempted to run influxdb in a docker container
with a windows host and a volume mounted from windows. that causes
problems because it apparently uses samba/cifs which does not
support fsync on directories. this patchset will, once, at startup,
check if it can fsync "/". if it receives an EINVAL as is what
appears to happen on samba/cifs, then it will not attempt to fsync
any more. this should help.

updates #9833.
updates #9630.

zeebo added a commit that referenced this issue Jun 1, 2018

fix(tsdb): attempt to work on docker on windows
multiple users have attempted to run influxdb in a docker container
with a windows host and a volume mounted from windows. that causes
problems because it apparently uses samba/cifs which does not
support fsync on directories. this patchset will, if it receives an EINVAL
on directory fsync, as is what appears to happen on samba/cifs, then it
will ignore it. this should help.

fixes #9833.
fixes #9630.

@zeebo zeebo closed this in #9847 Jun 1, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment