New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docker on Windows: CIFS v3.02 mounts failing with big file count #2285

Open
blurayne opened this Issue Jul 26, 2018 · 3 comments

Comments

Projects
None yet
1 participant
@blurayne

blurayne commented Jul 26, 2018

Description

We use containers for our developer environments. The projects are built with tools shipped within the containers. While that works like a charm for OSX and Linux, we face problems with some containers on Windows which have a heavy load of source files. The project directory is mounted to the container. This is not only npm related . It also affects containrs with PHP phing build system. Yet I can only provide the npm-example to reproduce.

As workaround we have found out, that builds do work with CIFS 2.0, but not with 2.1 or 3.02 mounts. If the build is failing it complains about "File not found". Yet the file is there and can be read!

I am certain the bug did not occur with Docker for Windows 1.13.0 (2017-01-19), yet already 1.12.1 (2016-09-16) introduced CIFS version 3.02. Maybe it is CIFS kernel module related or relies on communciation with windows CIFS server? I guess LinuxKit v0.4 introduced the problem.

  • I have tried with the latest version of my channel (Stable or Edge)
  • I have uploaded Diagnostics 5DE54CB5-8C76-4DCC-BD8D-4140C021E11C/20180726145107

Expected behavior

On long running builds files are accessible and and build does not fail since a file-not-found or similar is encountered.

Actual behavior

Build fails since it encounters 'file-not-found-error'. Yet, the file can be read in the terminal by cat or other tools.

Information

Microsoft Windows [Version 10.0.17134.165]
Version 18.06.0-ce-win70 (19075)
Channel: stable
501df65

Running Linux Containers only.

I tried running Moby with with CPU=1. Same results.

Also playing around with CIFS mount options within moby /proc/fs/cifs/** did bring no change for the better.

I did following to change that options:

docker run --net=host --ipc=host --uts=host --pid=host -it --security-opt=seccomp=unconfined --privileged --rm alpine /bin/sh -c  nsenter --target 1 --mount --uts --ipc --net --pid

Steps to reproduce the behavior

0. Setup

I tried following test setup. Please replace CIFS vers=2.0 with 3.02 or 2.1 and you'll see it fails. These commands are run by Windows WSL Ubuntu18.04 with DOCKER_HOST=tcp://localhost:2375.

dockerfile+faillog.zip

1. Build test image

docker build -t dockerbug .

2. Create test volumes with CIFS 2.0 and 3.02 protocol.

Please adjust host, username, password and domain to your Windows environment so the shares can be mounted.

The problem is NOT symlink related. I also tried switching mount options like mfsymlinks and so on.

docker volume create \
   --driver local \
   --opt type=cifs \
   --opt device=//10.0.75.1/C \
   --opt o=username=Markus,password=XXXXXXXX,relatime,vers=3.02,sec=ntlmsspi,cache=strict,username=Markus,domain=REFDESK-01,uid=0,noforceuid,gid=0,noforcegid,addr=10.0.75.1,file_mode=0777,dir_mode=0777,iocharset=utf8,nounix,serverino,mapposix,mfsymlinks,noperm,rsize=1048576,wsize=1048576,echo_interval=60,actimeo=1,nobrl \
   testcifs3.02
docker volume create \
   --driver local \
   --opt type=cifs \
   --opt device=//10.0.75.1/C \
   --opt o=username=Markus,password=XXXXXXXX,relatime,vers=2.0,sec=ntlmsspi,cache=strict,username=Markus,domain=REFDESK-01,uid=0,noforceuid,gid=0,noforcegid,addr=10.0.75.1,file_mode=0777,dir_mode=0777,iocharset=utf8,nounix,serverino,mapposix,mfsymlinks,noperm,rsize=1048576,wsize=1048576,echo_interval=60,actimeo=1,nobrl \
   testcifs2.0

3. Try a containered npm install

Attention! This may take a while to run! It also may rollback some files. This is not the error!

Failing with CIFS 3.02

docker run -it --rm -v testcifs3.02:/mnt/c --workdir $(pwd) dockerbug /bin/sh -c '$(getent passwd $(whoami) | cut -d: -f7)'
bash-4.2# mount | grep cifs
//10.0.75.1/C on /mnt type cifs (rw,relatime,vers=3.02,sec=ntlmsspi,cache=strict,username=Markus,domain=REFDESK-01,uid=0,noforceuid,gid=0,noforcegid,addr=10.0.75.1,file_mode=0777,dir_mode=0777,iocharset=utf8,nounix,serverino,mapposix,nobrl,mfsymlinks,noperm,rsize=1048576,wsize=1048576,echo_interval=60,actimeo=1)
npm install node-sass
[...]
Error: EINVAL: invalid argument, open '/mnt/c/Users/Markus/Desktop/docker-containered/node_modules/node-sass/package.json'
    at Error (native)
    at Object.fs.openSync (fs.js:642:18)
    at Object.fs.readFileSync (fs.js:510:33)
    at Object.Module._extensions..json (module.js:592:20)
    at Module.load (module.js:494:32)
    at tryModuleLoad (module.js:453:12)
    at Function.Module._load (module.js:445:3)
    at Module.require (module.js:504:17)
    at require (internal/module.js:20:19)
    at Object.<anonymous> (/mnt/c/Users/Markus/Desktop/docker-containered/node_modules/node-sass/lib/extensions.js:7:9)
[...]
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! node-sass@4.9.2 install: `node scripts/install.js`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the node-sass@4.9.2 install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2018-07-26T13_49_04_149Z-debug.log

exit code 1
/root/.npm/_logs/2018-07-26T13_49_04_149Z-debug.log
33833 verbose stack Error: node-sass@4.9.0 install: `node scripts/install.js`
33833 verbose stack Exit status 1
33833 verbose stack     at EventEmitter.<anonymous> (/usr/local/share/.config/yarn/global/node_modules/npm/node_modules/npm-lifecycle/index.js:304:16)
33833 verbose stack     at emitTwo (events.js:106:13)
33833 verbose stack     at EventEmitter.emit (events.js:191:7)
33833 verbose stack     at ChildProcess.<anonymous> (/usr/local/share/.config/yarn/global/node_modules/npm/node_modules/npm-lifecycle/lib/spawn.js:55:14)
33833 verbose stack     at emitTwo (events.js:106:13)
33833 verbose stack     at ChildProcess.emit (events.js:191:7)
33833 verbose stack     at maybeClose (internal/child_process.js:920:16)
33833 verbose stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:230:5)
33834 verbose pkgid node-sass@4.9.0
33835 verbose cwd /mnt/c/Users/Markus/Projects/bugreport
33836 verbose Linux 4.9.93-linuxkit-aufs
33837 verbose argv "/usr/bin/node" "/usr/local/bin/npm" "install"
33838 verbose node v6.14.3
33839 verbose npm  v6.2.0
33840 error code ELIFECYCLE
33841 error errno 1
33842 error node-sass@4.9.0 install: `node scripts/install.js`
33842 error Exit status 1
33843 error Failed at the node-sass@4.9.0 install script.
33843 error This is probably not a problem with npm. There is likely additional logging output above.
33844 verbose exit [ 1, true ]

From a run within a different container

dmesg

[ 2828.299719] 0000 4800 53fe 424d 0040 0001 0034 c000  ...H.SMB@...4...
[ 2828.299720] 0005 0001 0009 0000 0000 0000 b470 0001  ............p...
[ 2828.299721] 0000 0000 4238 0000 0001 0000 0015 0000  ....8B..........
[ 2828.299723] 4c00 0000 3377 4dfd 4a86 50e0 64f8 2bbe  .L..w3.M.J.P.d.+
[ 2828.299723] 1d92 7f70 0009 0000 0000 0000            ..p.........
[ 2828.299757] 0000 4800 53fe 424d 0040 0001 0034 c000  ...H.SMB@...4...
[ 2828.299759] 0005 0001 0009 0000 0000 0000 b470 0001  ............p...
[ 2828.299760] 0000 0000 4238 0000 0001 0000 0015 0000  ....8B..........
[ 2828.299761] 4c00 0000 3377 4dfd 4a86 50e0 64f8 2bbe  .L..w3.M.J.P.d.+
[ 2828.299762] 1d92 7f70 0009 0000                      ..p.....
[ 2828.299766] Status code returned 0xc0000034 STATUS_OBJECT_NAME_NOT_FOUND

Successful with CIFS 2.0

docker run -it --rm -v testcifs2.0:/mnt/c --workdir $(pwd) dockerbug /bin/sh -c '$(getent passwd $(whoami) | cut -d: -f7)'
bash-4.2# mount | grep cifs
//10.0.75.1/C on /mnt/c type cifs (rw,relatime,vers=2.0,sec=ntlmsspi,cache=strict,username=Markus,domain=REFDESK-01,uid=0,noforceuid,gid=0,noforcegid,addr=10.0.75.1,file_mode=0777,dir_mode=0777,iocharset=utf8,nounix,serverino,mapposix,nobrl,mfsymlinks,noperm,rsize=65536,wsize=65536,echo_interval=60,actimeo=1)
npm install
[...]
+ node-sass@4.9.2
added 199 packages from 134 contributors and audited 531 packages in 45.896s
found 4 moderate severity vulnerabilities
  run `npm audit fix` to fix them, or `npm audit` for details

real	0m46.728s
user	0m13.020s
sys	0m6.630s

exit code 0

@blurayne blurayne changed the title from Builds within containers failing fi to Builds within containers failing (CIFS 3.02 vs 2.0?!) Jul 26, 2018

@blurayne

This comment has been minimized.

Show comment
Hide comment
@blurayne

blurayne Jul 26, 2018

The issue can be provked by npm install node-sass alone. I updated the main description.

blurayne commented Jul 26, 2018

The issue can be provked by npm install node-sass alone. I updated the main description.

@blurayne

This comment has been minimized.

Show comment
Hide comment
@blurayne

blurayne Jul 27, 2018

It would be a nice feature to place default mount options somewhere in a config for troubleshooting. I did grep the source of docker and moby so far and just found out the moby is reading out mount options.

I did test with NFS volumes, yet only using NFSv3 3rd party servers like WinNFSd, FreeNFS on Windows. They were sucessful and the builds did run through. It would be cool if you guys could get Microsoft to open their NFSv4.1 Windows Server Feature to Windows 10 Pro installations.
https://docs.microsoft.com/en-us/windows-server/storage/nfs/deploy-nfs#deploy-nfs-infrastructure

blurayne commented Jul 27, 2018

It would be a nice feature to place default mount options somewhere in a config for troubleshooting. I did grep the source of docker and moby so far and just found out the moby is reading out mount options.

I did test with NFS volumes, yet only using NFSv3 3rd party servers like WinNFSd, FreeNFS on Windows. They were sucessful and the builds did run through. It would be cool if you guys could get Microsoft to open their NFSv4.1 Windows Server Feature to Windows 10 Pro installations.
https://docs.microsoft.com/en-us/windows-server/storage/nfs/deploy-nfs#deploy-nfs-infrastructure

@blurayne

This comment has been minimized.

Show comment
Hide comment
@blurayne

blurayne Jul 31, 2018

I think the bug came with the redstone update to windows. Yet, not verified.

WORKAROUND:

  1. Add capabilities to containers: SYS_ADMIN DAC_READ_SEARCH
  2. Remount within entrypoint to cifs vers=2.0

Also I have seen directio is not as as default within docker-compose for vers=3.02 mounts.

blurayne commented Jul 31, 2018

I think the bug came with the redstone update to windows. Yet, not verified.

WORKAROUND:

  1. Add capabilities to containers: SYS_ADMIN DAC_READ_SEARCH
  2. Remount within entrypoint to cifs vers=2.0

Also I have seen directio is not as as default within docker-compose for vers=3.02 mounts.

@blurayne blurayne changed the title from Builds within containers failing (CIFS 3.02 vs 2.0?!) to Docker on Windows: CIFS v3.02 mounts failing with big file count Jul 31, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment