NTFS hidden files are not visible when shared through gRPC Fuse

[dbolkensteyn]

I could reproduce with both the stable and edge channels:

• Stable 2.2.0.3 (42716)
• Edge 2.21.0 (42746)

Expected behavior

All shared files, regardless of whether or not they have the hidden NTFS attribute, should show up in the file listing produced by ls -al from a container on a shared folder.

Actual behavior

Hidden files are omitted from ls -al's listing.

Information

Because of the new caching mechanism that comes with gRPC Fuse, simply removing the hidden attribute from the file will not make it appear in ls -al's output. One way to workaround that is to create and delete a random file in the same folder, which seems to invalidate the cache.

[dbolkensteyn]

I've just tested with Docker Stable 2.1.0.5 (40693) which was pre-gRPC FUSE and I confirm that hidden files were indeed showing up.

[stephen-turner]

Thanks for the report, @dbolkensteyn. @djs55, is there a simple flag we can toggle in the FUSE library that would just make this work?

[djs55]

We're currently filtering out files with FILE_ATTRIBUTE_SYSTEM and FILE_ATTRIBUTE_HIDDEN. I've made a PR to remove the filter for FILE_ATTRIBUTE_HIDDEN.

[djs55]

Sorry for the delay getting back to you. I've got a development build suitable for testing: https://download-stage.docker.com/win/edge/43083/Docker%20Desktop%20Installer.exe It reports as version 2.2.2.0 (43083). It has prototype fixes for

• missing inotify MODIFY event
• missing hidden files
• filesystem events on long paths
• case-insensitivity fix for create
• fix a panic when removing a watch that doesn't exist
• ability to see filesystem events across directory junctions
• represent directory junctions as directories rather than symlinks

If you have a chance to test the build, let me know how it goes! If something goes wrong, please describe what happened in as much detail as you can and please include diagnostics.

[dbolkensteyn]

Thank you @djs55 , I have tested the build and I can confirm that this issue is indeed solved.

I still am getting another issue though for which I'll file a separate issue with a diagnostics, but it'll take a bit more time to come up with a minimal reproducer. This is libgit2 failing to clone a Git repository, note however that the git command line itself works. Just putting it out there in case it rings a bell:

failed to truncate pack file /github/zzz40500/Android-PullToNextLayout/objects/pack/pack_git2_xG7FzD: Operation not permitted

Thanks a lot, I'll close this issue as it is resolved!

[djs55]

@dbolkensteyn thanks for letting me know about the hidden files issue.

Regarding the failed to truncate: there was a known bug/limitation to do with concurrent file access and locking which I've recently fixed. If you have time and would like to try it the most recent development build is here: https://download-stage.docker.com/win/edge/43358/Docker%20Desktop%20Installer.exe

Thanks again for your help!

[stephen-turner]

This fix is now released in 2.2.0.4.

[eugenioestrada]

Hi @stephen-turner, i am having the same issue than @dbolkensteyn cloning a git repository with libgit2 using 2.2.0.5 release. Last release working properly is 2.1.0.5. Same error:

LibGit2Sharp.LibGit2SharpException: 'failed to truncate pack file '/app/repository/.git/objects/pack/pack_git2_gMlADA': Operation not permitted'

[stephen-turner]

@djs55 I know the "failed to truncate" was not the primary issue in this ticket, but where did the fix reach? I'm guessing it's on Edge 2.2.3.0 but not Stable 2.2.0.5 so that @eugenioestrada's observation is expected?

[djs55]

@stephen-turner @eugenioestrada yes I believe that's correct -- the fix for the read-only file issue (which caused git to fail) was not merged to 2.2.0.5. I did make test builds based off that branch but the actual fix is only released in edge 2.2.3.0.

[eugenioestrada]

@djs55 @stephen-turner I've tested the 2.2.3.0 edge release, still failing when cloning a repo but different error.

"LibGit2Sharp.LibGit2SharpException: 'failed to mmap. Could not write data: Permission denied'"

[djs55]

@eugenioestrada could you reproduce the error and then upload a diagnostics report? The diagnostics option is in the whale menu in the system tray -> Troubleshoot -> Diagnostics. Once it uploads it provides an ID which you can quote here -- this will allow me to look at your logs. Thanks in advance!

[eugenioestrada]

hi @djs55, i've published a repo with an example of the issue:

https://github.com/eugenioestrada/docker-for-win-issue-example

I've also uploaded the diagnostics data with ID: 41FE822B-5B64-4E26-BE56-FCE8C8E9F548/20200408143336

Hope it can help you to solve this :)

[eugenioestrada]

hi @djs55 @stephen-turner, was it useful to reproduce the issue?

[eugenioestrada]

hello @djs55 @stephen-turner issue is already happening in 2.3 release using WSL2

[djs55]

@eugenioestrada sorry for the delay getting back to you and thanks for the repro example. On latest edge 2.3.0.0 (44472) I get the error:

Unhandled exception. LibGit2Sharp.LibGit2SharpException: failed to truncate pack file '/shared/repo/.git/objects/pack/pa
ck_git2_dohfjH': Operation not permitted

which corresponds to the low-level trace

{
    "ID": 7718,
    "Name": "Truncate",
    "Time": "2020-04-24T14:29:25.544172232Z",
    "Request": {
      "Name": "c/Users/djs/bugs/5808/docker-for-win-issue-example/shared/repo/.git/objects/pack/pack_git2_dohfjH",
      "Size": 14168047,
      "Context": {
        "Owner": {},
        "Pid": 20251,
        "Export": 2
      }
    },
    "Response": null
  },
  {
    "ID": 7718,
    "Name": "Truncate",
    "Filesystem": "remote",
    "Time": "2020-04-24T14:29:25.546498432Z",
    "Request": null,
    "Response": {
      "Status": {
        "Code": 1
      }
    }
  },

where "Code": 1 is EPERM i.e. "Permission denied". I need to check further where the error is coming from.
Thanks for making a repro example!

[djs55]

@eugenioestrada I've made a prototype fix and have a test build: https://download-stage.docker.com/win/edge/44749/Docker%20Desktop%20Installer.exe For me the repro example runs to completion and prints "Hello World". If you have time, could you let me know if it fixes the problem for you? If not, could you upload a fresh set of diagnostics. Thanks again!

[eugenioestrada]

Hi @djs55, with the edge release 44749 is working in the demo project and also in my real life project. So it's fixed ;) I've also tested the new edge release 44875 and it's fixed too there.

Thanks you!!!

[djs55]

@eugenioestrada I'm glad to hear the problem is fixed! Thanks for your help tracking this down -- the repro case was invaluable.

[eugenioestrada]

@djs55 sorry, i think i made a mistake testing the issue. I tested it locally instead of running on Docker :(

44749 and 44875 still failing on my computer in demo project, the real life project fails in another point of the code but it could be because a change of code.

Running the repro case I've still getting the exception:

LibGit2Sharp.LibGit2SharpException: 'failed to mmap. Could not write data: Permission denied'

Sorry for my previous message!!!

[eugenioestrada]

hi @djs55, when docker is running using Hyper-V doesn't fail but when you use the WSL2 backend it's still failing with the error "failed to mmap. Could not write data: Permission denied".

[djs55]

@eugenioestrada I assume the error is coming from the WSL 2 built-in 9P filesystem which can be used to access files from the Windows host.

To make the best use of WSL 2 I recommend using the "Linux workspaces" feature described here. The trick is to

• enter the Linux shell with wsl
• switch to the Linux home directory (e.g. cd ~)
• clone the source code there
• run either explorer . or code . to launch file explorer or VSCode on the Windows host: this shows you where the Linux files are mounted on Windows
• edit the files on Windows, through this remote mount
• run docker run inside the Linux shell: this ensures that the container runs directly on the Linux filesystem which is the fastest (and most semantically correct) configuration.

Let me know if that works for you!

[eugenioestrada]

Hi @djs55, the Linux workspaces can be used with VS Code but not with Visual Studio 2019 because it can't load the solution:

 error  : The project file could not be loaded. Invalid URI: The hostname could not be parsed.  \\wsl$\Ubuntu\home\eugenio\github...

\wsl$ host can't be mapped to a letter... I don't know if the new Tux option in Windows explorer could solve this. I will give it a try.

[docker-robott]

Closed issues are locked after 30 days of inactivity.
This helps our team focus on active issues.

If you have found a problem that seems similar to this, please open a new issue.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle locked