Various issues associated with Windows/POSIX synchronization

[Toilal]

It seems mutagen doesn't support sync from windows to linux because chmod is not supported on windows.

It also cause problems with symbolic links.

See the following status, Alpha is running Windows and Beta is running linux.

Session: 4cbdd611-a26a-4a46-9623-badffbac15d6
Status: Scanning files on beta
Alpha:
        URL: C:\devel\projects\AFB-oison
        Status: Connected
        Problems:
                mapserver/mapserver.map: unable to ensure path does not exist: path exists
                oison-backend/liquibase.properties: unable to ensure path does not exist: path exists
                oison-backend/phpunit.xml: unable to ensure path does not exist: path exists
                abc: unable to create file: unable to relocate staged file: unable to set file mode: chmod C:\devel\projects\AFB-oison\abc252047704: not supported by windows
                def: unable to create file: unable to relocate staged file: unable to set file mode: chmod C:\devel\projects\AFB-oison\def766032087: not supported by windows
                docker-compose.override.yml: unable to ensure path does not exist: path exists
                .docker/.ca-certificates/GFI_Informatique_Orléans_Root_CA.crt: unable to create file: unable to relocate staged file: unable to set file mode: chmod C:\devel\projects\AFB-oison\.docker\.ca-certificates\GFI_Informatique_Orléans_Root_CA.crt941617226: not supported by windows
Beta:
        URL: vagrant@192.168.1.100:/home/vagrant/test
        Status: Connected

"unable to ensure path does not exist: path exists" => Symbolic links
"unable to set file mode: chmod C:\devel\projects\AFB-oison\abc252047704: not supported by windows" => New file created on Beta (linux). chmod should be ignored on windows, or only set readonly if "w" flag is missing on the linux counterpart.

It also creates a bunch of files when it fails, it seems temporary file is not removed when those failures occurs.

I'll be glad to contribute with a Pull Request for this issue, but I never wrote anything in Go lang.

[xenoscopic]

Hi @Toilal, thanks again for the report. Support for Windows <-> POSIX synchronization is a major goal, and things should work, so I definitely want to figure out what's going on. I don't use Windows on a daily basis, so the testing on that side of things doesn't get the attention that it should. I'm hoping to change that in the next release.

I think there are a few issues here:

1. Something is wrong with the symbolic link logic on the Windows side of things, or at least the error isn't very clear. Mutagen doesn't support symbolic links on Windows, and it will indicate a problem if it can't propagate them. I think the problem it's indicating is not particularly clear. I'll play around with this and see if I can't make things more sane.

2. It looks like Chmod behavior has changed on Windows since Go 1.8, which was the last time I thoroughly tested this. I'll have a look and see what's going on. My goal right now is to bring up test coverage before the next release, so hopefully I'll suss out most of these issues.

3. The atomic file rename function (which is used when moving across volumes) is not removing the temporary files it creates on chmod failure. That's why all of these extra files are being created. I'll fix this.

I appreciate your offer for help, but I should be able to sort these issues out in the next few days. As I mentioned in the other issue you filed, I'm trying to have a new version released by early next week, and I'd be grateful if you could try it again.

[xenoscopic]

Hi @Toilal,

I've had a chance to address most of these issues with 672263f and 1e842d2. I'm still trying to figure out what's causing the "unable to ensure path does not exist: path exists" error for symlinks.

Windows does support symlinks, sort of. They aren't really useful until Windows 10 because they required complex permissions to create before that. Consequently, Mutagen doesn't try to support them on Windows (at the moment). But if they already exist, Mutagen will show the error "unable to ensure path does not exist: path exists" before it shows the error "unable to create symlink: symlinks not supported on this platform".

Can you tell me a little bit more about the folders you're synchronizing? What is the origin of the symlink files? Were they created on the Linux side? Did they already exist on the Windows side as native Windows symlinks (reparse points)?

[Toilal]

Symlinks are coming from the windows side (alpha), Linux side (beta) was empty at first. These are native windows links.

[Toilal]

I'm investigating alternatives to NFS shares between windows and Ubuntu (through Vagrant and winnfsd) for docker based development environments.

This project seems really good and if it work properly, i'll try to wrap it into a vagrant plugin to share folders between host and VM trough Vagrantfile configuration. This should improve the IO performance of the application being develop in such environment.

[Toilal]

If symlinks can be supported properly on Windows 10+ only, it will be just fine. WinNFSD supports symlinks properly when processing is running as admin only.

Maybe you could perform a beta/rc release so I can test your fixes ?

[xenoscopic]

Okay, I think I've figured out a strategy for Windows symlink support, and I've implemented it in 2e82f53. It took a little thinking about, but I think it actually makes symlink support better overall, even on POSIX systems.

Mutagen now has three symlink synchronization modes: sane (the default), ignore, and POSIX raw. These modes can be specified when a session is created.

The sane mode synchronizes only those symlinks that can be reliably round-tripped between POSIX and Windows. This essentially limits support to those symlinks which are relative paths, don't contain certain special characters (namely : and \), and don't point outside the synchronization root at any point in their target. The test for this property is normalizeSymlinkAndEnsureSane. It may need refinement if users run into corner cases, but I think it should cover 99.99% of cases. Sane mode has the downside that if any "non-sane" symlinks exist in the synchronization root, the synchronization has to be halted until they're removed or fixed. Unfortunately they can't just be ignored, because if they suddenly became "sane" or "non-sane," they would appear or disappear in Mutagen's eyes, and it would break the three-way merge. That being said, I think this will work for the vast majority of cases, especially those cases where something like a source tree or an npm_modules directory contains symlinks into itself. I think this is the best we'll ever be able to do with supporting symlinks on Windows. Even winnfsd applies similar restrictions (the symlink can't point outside the mount).

Sane mode on Windows also still requires the SeCreateSymbolicLinkPrivilege permission for the user executing the process creating the symlink (either the daemon or the agent). Windows 10 (in Developer Mode) allows users to create symbolic links without this permission, but it requires the SYMBOLIC_LINK_FLAG_ALLOW_UNPRIVILEGED_CREATE flag to be specified to CreateSymbolicLinkW, and the Go change that implements this won't land until Go 1.11.

Ignore mode is exactly that: any symlinks seen in either synchronization root are ignored and not propagated.

POSIX raw mode is a mode that only works between POSIX systems: it copies the symlink target exactly, without any validation or normalization. POSIX symlinks are just opaque blobs, and they can be round-tripped reliably.

I think the combination of these three modes should cover most use cases.

I want to finish a few more unrelated tweaks tonight, but I will put together a beta build for you to test and let you know when it's ready. Once you've verified that looks better, I'll close out this issue.

[Toilal]

It seems great, just ping me when a new release candidate is available and i'll then test this in my environment as soon as possible.

[Toilal]

Any chance to get a release soon ? even a beta/rc one ?

[xenoscopic]

Yes, sorry! I've been caught up with work these last few weeks, but I should be able to tag a new release by Wednesday morning.

[xenoscopic]

Hey, I know I'm late, but I've got a version that I'm going to play with a bit this evening and then tag and release as v0.3.0. It'll still be a beta release, but I'd be thrilled if you could give it another shot. I'll ping you as soon as I've tagged it.

[Toilal]

Thanks ! No problem, take it easy :)

[xenoscopic]

Hi @Toilal, I've just tagged and released v0.3.0. Sorry it took so long, but I wanted to make sure it was ready to go. It's also quite a big release with a lot of new features, so the documentation has been updated accordingly. It should fix all of the problems described in this issue, but I would be grateful if you could try it when you have a chance. It breaks compatibility with v0.2.x, so please delete ~/.mutagen if it exists and recreate sessions afterward.

Definitely let me know if it doesn't fix the issues you've described or if you run into any other issues. Thanks again for your help!

[Toilal]

I gave a first try and it seems really awesome now ! Issues on windows are now fixed, and it's definitively the best replacement option for Unison :). First sync was as fast as a git clone, and new monitor is brilliant too ! Good job !

Is there any option/configuration to enable faster change detection on the windows side, based on filesystem notifications ? Initial synchronisation is damn fast, but when working on files, it has some delay (a few seconds) before the file change is propagated on the other side.

Does mutagen list -l display effective watch mode, or only requested ones based on configuration ?

mutagen list -l

Session: bf326751-27f4-4073-8d8f-4207bae9c33a
Ignore VCS mode: Default (Propagate)
Symlink mode: Default (Portable)
Watch mode: Default (Portable)
Watch polling interval: Default (10 seconds)
Alpha:
        URL: C:\devel\projects\SOFAXIS-EspaceClient
        Status: Connected
Beta:
        URL: vagrant@192.168.1.100:~/mutagen/SOFAXIS-EspaceClient
        Status: Connected

anyway I'll still try to use it on my actual dev projects, and if everything is ok i'll try to write a Vagrant plugin to implement SyncFolder with mutagen.

[xenoscopic]

Hey, thanks for the initial feedback. Please feel free to report any issues, no matter how small. If things aren't simple and functional, then they aren't working correctly.

Right now the native file monitoring is only used on Windows if the path is a subdirectory of the user's home directory. The reasons for this are outlined here, but I've discovered an unrelated issue on macOS, so I will try to put in time to remove this limitation over the weekend. Please stand by for more information.

mutagen list -l doesn't really display effective watch mode. It does display the watch mode requested by the configuration (or the default value if none is requested), but a watch mode of "portable" doesn't really tell you what's actually being used. I will think about this over the weekend and think if I can come up with a better mechanism for displaying this. Perhaps I can list the watch mode being used under each endpoint.

Thanks again for your input!

[xenoscopic]

Hi @Toilal, I've just pushed out v0.4.0 which implements a completely new filesystem watching mechanism. Recursive watching should now be used in every directory on Windows. There's one caveat due to a bug in another package that I'll try to get fixed soon. But in any case, it should be better than the old watching mechanism in every way.

[xenoscopic]

The aforementioned Windows filesystem watching caveat has been fixed in v0.4.1, so that's no longer an issue.