Manipulate Linux/Ubuntu subsystem files from via the Windows environment

[BJobsis]

• A brief description:
  When I try to create a file from Windows10 (using Sublime Text, etc) into the Linux Ubuntu subsystem
  the file is created, but afterwards the file can't be seen from within the subsystem.
  I bypass this by creating (and cutting/pasting) the file via the NANO cmd.

• Expected results
  It would be nice (like in Apple's OSX terminal environment) to manipulate files from outside the Linux subsystem.
  In my opinion (as a developer) this would mean a big step forward in the usability of the Ubuntu subsystem.

*My Windows build number: W10 Build 1607 14393.576.

Regards, Barry Jobsis (barry.jobsis@gmail.com)

[aseering]

Thanks @BJobsis for reporting this! For what it's worth, I think this is the same basic issue as #1239 , though the way you have phrased it is more general.

If you (or anyone else reading this particular issue) has not already seen this blog post, I'd recommend reading it:
https://blogs.msdn.microsoft.com/commandline/2016/11/17/do-not-change-linux-files-using-windows-apps-and-tools/

The intent, in short, is that you put your files in /mnt/c/ somewhere if you want to get at them from Windows.

[aseering]

@benhillis -- this UserVoice appears to indicate that this issue is WONTFIX. Could you (or could someone else at Microsoft) make sure this ticket and that UserVoice are flagged the same way?

[fpqc]

@aseering It looks like their thinking has moved from "wontfix" to "might eventually fix in RS3 or RS4", based on some hints I've seen dropped.

[benhillis]

@aseering - I'm not sure why this was ever marked as won't fix. This is a huge limitation of the current design and definitely something we're looking at improving. I'll get that user voice status changed. Thanks for the heads up.

[fpqc]

Lol, and that's about as close to a confirmation as we've seen!

[benhillis]

It's a hard problem to solve (mapping Windows permissions to Linux permissions or vise versa). Just ask anybody who's worked with Samba.

[fpqc]

@benhillis Yup, the Samba people blamed you guys for it though!! (ACL format is not stable between windows versions).

[benhillis]

I'm pretty sure I wasn't born yet when those decisions were made :)

[therealkenc]

But if one shrinks the problem space to /home, ie "development related scenarios", the permissions problem solves itself. Just ask anybody who's worked with OneDrive. 😃

Longer term, there is no reason why users or permissions need to be mapped in the first place (but I digress).

[pharmmd-rich]

do what I do, create a symlink to your documents folder and just work on stuff like that. If you are really brave, you could try symlinking your windows user directory to your home directory. I haven't tried that though, so it might break things.

[fpqc]

Will break everything. Don't do it.

[therealkenc]

Here, fixed that for you. This also represents the third way I alluded to a month ago in #1397 (comment). Options available to have VSCode open a file on a WSL path:

1. VSCode would have to have built in logic to understand WSL file paths.
2. WSL could do some magic to the command line before creating the NT process. [Ben] outlined some of the trouble with that in Launching Win32 applications from within WSL #1243
3. Orrrr... Use a Window Filesystem that has built in logic that understands WSL file paths.

I give you, nfs-ganesha on WSL and nfs-fuse in Windows via dokan-dev. That's VSCode editing .bashrc, launched from WSL home directory on L: drive (natch) in Windows Explorer. Code to follow when it sucks less.

[fpqc]

@therealkenc Lol, was waiting for someone to do this. I figured you could do it horribly by setting up a local Samba share and mounting it on a Windows network mount, but then I considered what a hassle it would be. I'm guessing you've essentially streamlined this using dokan?

Hope I'm doing it justice.

[therealkenc]

Samba is problematic because Windows is a CIFS server and those ports are spoken for. This is NFS not CIFS. There is a NFS client available for Windows, but Microsoft considers the technology all Enterprisey 'n stuff (as opposed to the thing your teenager uses to share XBMC videos throughout the house).

I considered a fusesmb port to dokan, with which you could choose a nonstandard port, but that was just a bridge too silly to contemplate seriously. That is thread #764 (message). In fact if I knew how to redirect the CIFS ports on Windows this hack probably wouldn't have happened.

[fpqc]

So KenC, does this automagically fix paths (so launching code.exe ~/blah opens the file with a path relative to the nfs mount?

[therealkenc]

You'd get automagic translation if you mount / on L:, because then there's simply nothing to translate. That's problematic for other reasons though. I'll start playing with path translation after (if) I get more pressing stuff addressed. Truth is I haven't looked into how the interop stuff deals with paths much; notably $PWD. As of this writing it is mostly useful (if that is the word) for having write-access to VolFS files from the CMD prompt not the WSL prompt.

It's good to know they have their top men on this however. Though I would prefer if it were someone with a very particular set of skills.

[fpqc]

@therealkenc By the way, you don't need dokany if you have Win10 pro. There is some incorrect info, but the NFS client is still available for installation in the optional Windows features.

By the way, I built ganesha and set up a pkgbuild that I think is right, but I was wondering what you did and how you set up your config to get it to work (this won't build locally given the screwed up state of my build tools, but I do have a server running Arch, so I built it there).

PKGBUILD.txt

[therealkenc]

Ganesha doesn't work out of the box because of #1530 and to a lesser degree #1492. It was a relatively large work-around (basically implementing a dcache in userspace). Rather than "going public" I shelved this for the time being. #1529 makes interop painful unless you disable fd caching entirely. And there's a bug in ganesha itself that needs to be worked out. I get the (possibly unfair) impression ganesha is used mostly for cluster stuff; which is understandable because Real Linux™ has a perfectly serviceable NFS kernel module.

Upshot is that I'm presently looking into porting libsmbclient to Windows, even if that means they might have to haul me off to a padded room.

Funny story about Win10 Pro. I tried to give myself a Pro upgrade for my home machine as a Christmas present, but Microsoft wouldn't take my money because no US address to match my credit card. So I ported nfs-fuse to Dokan instead.

[fpqc]

@therealkenc =.= Yes, Microsoft wouldn't take our money at work to upg to Enterprise. They wanted us to contact a partner or something so we gave up.

We should have lied & told them we were deficient in our licenses or something lol. Then they'd have to give us something we could buy to move back into compliance!

Anyway, will keep an eye on those bugs. And yes, you are probably insane to try to port libsmbclient back to Windows!

[kayakyakr]

Late, but glad to hear that the perception has shifted toward allowing windows to access VolFS.

Still hoping that DriveFS can eventually support all the calls expected. The use case of IDE of choice + WSL + Docker is a really good environment, but is still just barely not supporting a lot of the frameworks.

Stock Rails works! Ember-Cli does not :(

[therealkenc]

Okay; there is one solution using stuff in everyone's toolbelt. Apache2 on WSL works alright enough out of the box these days. Which means WebDAV works. For those not familiar, friendly instructions for Apache here and WebDAV here (among many other sources). Mounting a WebDAV drive on Windows 10 here.

It works better than passably well for file editing and copying -ish use cases. I dragged-and-dropped a 1.5GB iso image and performance wasn't bad to the naked eye. Doing something like find-in-files on a big tree with Windows VSCode does work, but won't win any benchmark competitions. I'd post another screenshot, but it looks a lot like the last one.

[BJobsis]

Is there someone at Microsoft who can confirm that this request will be implemented in the (i hope near) future?

[fpqc]

@BJobsis Not yet It's something that they are looking at how to do for Redstone 3 (post-April). It will not make it into this series of insider builds. Read Rich Turner's Microsoft twitter feed @richturnms . Also Ben Hillis (MS Developer) has said the same a few times on this issues board. Please look for yourself.

[markallasread]

Subscribed. This is really my only impediment in swapping to windows full time (which, admittedly, I did — but am now regretting as a result of this).

• Creating a file from windows on the wsl folder makes it invisible & gives it really weird properties. (Edit: I should now note I see this is not supported, and heavily recommended against — don't be me!)
• Mounting windows drives on linux (or viewing them) shows everything with 777 permissions (and makes my eyes twitch, as a long-time linux/macos user)
• I tried running my daemons from windows as an alternative (mkdocs, jekyll, etc) but they hate backslashes and pretend none of my files exist.

Altogether, these aren't showstoppers, but when I use my IDE/text editor in windows (atom/sublime text/etc) — and create a file, it takes a few hours before I realize my mistake and have to undo a few hours of work.

For now, I'm simply using dropbox as a go-between — yes, everything has 777 permissions & drives me batty, but I can solve that with a simple script when I provision my work.

[fpqc]

@DavidandGoliath My understanding is that they plan on removing some of the limitations on editing lxss files from Windows in the future, but I have no idea how they plan to do so. There have been a few suggestions from the community, but however the Devs are going to do it, they'll make sure it makes sense. We do know for sure that this won't be in the insiders builds at least until RS2 RTM (End of April). It might take months after that before we see anything in the RS3 Insider builds. I do not know, and the devs are loath to speculate. Addressing 777 mounts will likely flow from however they decide to do this.

As for daemons, this is going to rely on two things: Finishing syscalls and kernel features required by systemd, and the integration of the WSL init daemon (which manages Linux-side of the LXSSBus and provides the Win32 control and console interfaces) as a standalone daemon service run by systemd. This target was alluded to in passing, I think, in Ben Hillis's blogpost about interop, but we have rather less information about it aside from the uservoice response and Ben's mention of WSL-init outside of PID1.

So keep your fingers and toes crossed.

Also, the 777 factoid is not relevant, since cmd can be launched interactively from WSL and used to edit /etc/shadow through the Windows-side interface. If someone knows you are running WSL, root is effectively equivalent to any other user.

[pharmmd-rich]

IMO it's such a low priority. You can mount any Windows folder you need inside your linux system at any point in the filesystem. I have my Windows Documents folder mounted in ~/Documents for instance. I personally would rather see them work on improving compatibility, adding support for starting linux services at WIndows startup, and performance improvements (compiling a project on Ubuntu native is currently much faster than on WSL). I can't think of a single scenario where you would need to do the opposite and edit files directly on the WSL FS.

At any rate, to fix editing they could easily create a virtual (.vhd) EXT4 drive for the FS and store it in the User folder in Windows. Double click the VHD, it will mount, and you can edit files.

[kayakyakr]

Agree with @pharmmd-rich: I think the goal right now should be getting DriveFS up to speed so it performs comparably with VolFS and supports all the ops necessary. It seems that's a goal of theirs too as the most recent build came with DriveFS performance improvements.

Think this is a good place to mention really hoping that #1357 is fixed soon as it's the last hard block (that I know of) to my swapping to windows development full time. Though DriveFS performance is a soft block.

Linux services are better handled in Docker than WSL, imo.

[markallasread]

The underlying issue is that when I commit my code to repos, my permissions (and, by default, everything being 777 is committed as well). Perhaps this is more part of my issue: #352

[aseering]

@DavidandGoliath -- as a workaround, that behavior is configurable for git via git config core.fileMode false. I would expect at least some other version-control systems to have similar workarounds, though I don't know them offhand.

[marcelgwerder]

At this point this is the most important feature for me as a web developer. All I want is to edit my code from windows, everything else should happen within WSL. Having the code on the Windows filesystem works 99% of the time but there is that one percent where the application thinks it's running on Linux and does all the crazy things the Windows filesystem doesn't handle properly (permissions, directory separators etc.). This reminds me of working with Docker for Windows which had the same issues because the code was residing on Windows and I ultimately had to abandon that solution. The only way all the code I use runs seamlessly is when it not only runs in Linux but also is on such a filesystem.

Just a small example of some code from the core of a PHP CMS I have to work with:

// Replace \ with _ if on Windows
if (DIRECTORY_SEPARATOR == '\\')
{
	$key = str_replace('\\', '_', $key);
}

You can imagine that this doesn't behave as intended on WSL because the Windows check will fail and the resulting cache key, which is used as the file name, keeps the backslashes in it and will create a number of directories instead of a file.

Definitely a huge 👍 for this!

[pharmmd-rich]

I still don't understand the desire to directly edit the files on the WSL filesystem. WSL provides a share (/mnt/c or similar) that allows you to perform the same thing, safely and without issue (I've never encountered one at least).

However, performance of the WSL filesystem in general is still very slow, I still use a CentOS VM on VMWare and use shared folders. It is exponentially faster and more stable to do stuff that way.

Hopefully the WSL team focuses on speeding things up in the future.

[kayakyakr]

@pharmmd-rich For me, it's mostly a speed issue. DriveFS is significantly slower than VolFS. If I was going straight Vim, I'd be able to manage, but if I want to run an integrated environment (Atom/VisualStudio Code, Docker services, linux server), I've gotta be on DriveFS and it's at times painfully slow.

[teras]

I am also in the need to edit files directly on WSL fiesystem, and I will give you an example why is that. I need to host my project in WSL for hundreds of reasons, some of which are unix permissions, symbolic links, compatibility with other open source tools, fixed directory structure etc. Now, I need an IDE to edit my project (let's pretend that this is IntelliJ or Netbeans). Then I am stucκ to use the Linux version of the IDE (which again means gigabytes of dependencies, not to mention the ugly graphics). This is my only reason up to now that I continue developing on my Linux box, instead of booting into my (test-only) Windows partition. 2017-12-08 20:12 GMT+02:00 Richard Campbell <notifications@github.com>:

…

I still don't understand the desire to directly edit the files on the WSL filesystem. WSL provides a share (/mnt/c or similar) that allows you to perform the same thing, safely and without issue (I've never encountered one at least). However, performance of the WSL filesystem in general is still very slow, I still use a CentOS VM on VMWare and use shared folders. It is exponentially faster and more stable to do stuff that way. Hopefully the WSL team focuses on speeding things up in the future. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1524 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAkKuC6sKQ2Ncw1819JuSiHjItC6ccPPks5s-Xv4gaJpZM4LR5BU> .

[Gannalech]

Here's a workaround: after setting obcaseinsensitive=0 in the registry and fixing any path issues install Sandboxie (any sandboxing program will do). Run the IDE / text editor sandboxed so that it does not directly save to lxss. Use rsync under Bash to update the files.

[leo-cydar]

@Gannalech that is a terrible workaround.

[Deltrego]

@b71729 maybe you need more detail:
I enabled case sensitivity on an almost new w10 install and luckily had no issues. Without it you can still edit linux files with non-conflicting names.
I tried to avoid third party tools but could not make VirtualStore do what was needed (tried a few things like shimming the text editor so that run as non-admin any writes to lxss folder would be virtualized).
With Sandboxie I created a custom "lxss" sandbox w/ no auto recovery, no deletion, protecting only the lxss folder. Then I defined the following aliases under bash:

alias *1='"/mnt/c/Program Files/Sandboxie/Start.exe" /box:lxss explorer.exe /root,C:\\Users\\$USER\\AppData\\Local\\lxss'
alias *2='rsync -r /mnt/c/Sandbox/$USER/lxss/user/current/AppData/Local/lxss/ /.'
alias *3='rm -rf /mnt/c/Sandbox/$USER/lxss/user/current/AppData/Local/lxss/*'

$USER represents the Windows username, no spaces. Not a predefined variable.
*1 opens a sandboxed Explorer window from bash (any opened file is also sandboxed).
*2 syncs changes, this should preserve original permissions, using mv would make them 777
*3 deletes lxss folder from the sandbox (to avoid stale files reverting any changes done in bash)
Hope someone finds this useful.

[BJobsis]

Ls,

Tested create file in W10 1903 (via WSL explore.exe) with Sublime Text and read back in WSL with Nano. And vice versa. Both actions worked perfect.
I close this request now as I consider it as solved.
Development team thanks for your effort.

Regards, Barry

[ilqvya]

@BJobsis what about symlink folders ?

[therealkenc]

Shipped in 18342 aka 1903.

[BJobsis]

Ilya, I never used symlinks. I understand what they do but don't see any added value in this case. For me: I needed a way to edit a file in WSL and use in in W10 and vica versa. That works now and so my problem is solved. Regards,Barry

…

On Thu, Oct 31, 2019 at 4:55 PM Ilya Polishchuk ***@***.***> wrote: @BJobsis <https://github.com/BJobsis> what about symlink folders ? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1524?email_source=notifications&email_token=ACFBUZIFKCL42SZ5PK3MR3LQRL5XBA5CNFSM4C2HSBKKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECYJKLA#issuecomment-548443436>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACFBUZJFKFRCOCAW6NWIJPTQRL5XBANCNFSM4C2HSBKA> .