Support OneDrive folders on recent Windows 10

[jarppiko]

Syncthing (v0.14.40) does not work with Onedrive on Windows 10 Fall Creator Update (version 1709) properly in some cases.

In Win 10 Fall Creators Update, Microsoft changed how Onedrive operates NTFS filesystem. Now OneDrive creates reparsepoints (an NTFS feature). Syncthing forum user "geeksquad" explained it here. It seems Syncthing is refusing to recognize a OneDrive directory using reparsepoints and reports an error message "folder path missing".

[ZEER7] 12:49:24 WARNING: Creating folder marker: folder path missing
[ZEER7] 12:49:24 INFO: Restarted folder "EXAMPLE" (xxxxx-yyyyy) (readwrite)
[ZEER7] 12:49:24 INFO: Connection to WJXMAFI closed: reading length: read tcp 192.168.1.2:51413->192.168.1.3:22000: use of closed network connection
[ZEER7] 12:49:24 WARNING: Error on folder "EXAMPLE" (xxxxx-yyyyy): folder path missing
[ZEER7] 12:49:24 INFO: Failed initial scan of readwrite folder "EXAMPLE" (xxxxx-yyyyy)

I have 2 OneDrive synced directories on a same computer synced with Syncthing to a Linux server. Based on my experiments the following factors may impact on the success of getting Syncthing to sync a OneDrive folder:

1. I have OneDrive Business and standard OneDrive folders and only the ordinary OneDrive folder syncs properly. There might or might not be a issue related to Onedrive Business.
2. In case of the (standard) OneDrive, the synced folder is not the root OneDrive folder, but a folder under the Onedrive root folder. Syncthing syncs this folder under the OneDrive root folder OK.
3. In the case of the OneDrive Business folder, I am trying to sync the OneDrive Business root folder, but after updating to Win10 Fall Creators Update, Syncthing stopped syncing this folder.
4. Syncthing does not sync properly any OneDrive or OneDrive Business folder IF Files On-demand feature has been enabled on the OneDrive. Microsoft introduced this feature in Win 10 Fall Creators Update and it is enabled by default. I have disabled it manually.

Version Information

Syncthing Version: v0.14.40 Win 64-bit
OS Version: Windows 10 Fall Creator Update (Version >= 1709 )

[clee231]

Adding to the conversation...

Note that accessing the OneDrive folder from Linux distributions utilizing the ntfs-3g driver to read/write to NTFS filesystems will also experience similar issues. This is because ntfs-3g is not currently able to understand folders/files with reparse points. It would be unlikely that Syncthing would be able to get around it without removing the reparse point. (microsoft/WSL#1941 (comment))
I haven't done extensive testing on this, but I've heard another user mention that using the NTFS kernel implementation instead of the ntfs-3g driver will allow you to at least view the contents of the folder/file. Unfortunately the kernel implementation doesn't support writing to NTFS filesystems.

If people are syncing their OneDrive folder on Linux using the Windows partition copy, a quick workaround would be to use a script to execute the deletion of the reparse point on restart/shutdown on the Windows side. Deletion of the reparse point doesn't effect anything, as far as I can tell... But perhaps the story is different when you enable OneDrive's new "Files on Demand" feature.

[calmh]

I think this is a conceptual problem, not a Syncthing problem. It's true that the reparse point is the direct cause of the failure, as that isn't presented in a way we expect in either the Go runtime or the lower level Windows APIs (I don't know which). But I expect that the point of the reparse point is for Windows to present OneDrive as some sort of virtual filesystem, with the files being on disk or in the cloud and being downloaded automatically as necessary.

As such I fully expect things to be completely broken even if we understood the reparse point, as the files are probably not even on disk, necessarily. So all in all I think this is just something that will remain an incompatibility.

But someone more on the Windows side of things is welcome to prove me wrong and fix it.

[jarppiko]

It is possible to disable the "Files on Demand" feature in OneDrive settings. Therefore from user perspective this is bug/unsupported feature. In my case all the files and folder are locally on windows hard drive, but Syncthing still can't sync those.

I ditched Syncthing and went back to manual Unison sync. Unison can sync the files without any problem so I do not understand what would prevent Syncthing from doing so.

[coolVariable]

Not sure why this was marked as closed, since the issue still persists.

I understand that Syncthing may need to be limited to files ON the device, ie not only in the cloud, but if a folder is fully synced then this should "just work".

Not fixing this severely limits the appeal of syncthing.
This, plus the inability by syncthing on android to work with SD cards outside of root access, severely limits the usability of syncthing - Syncthing used to be a must have program for me. Now it is turning into ... oh Syncthing has yet another issue/bug.

[AudriusButkevicius]

OneDrive should work if you disable the files on demand feature, and point the folder at the root of one drive and not Documents or something like that where OneDrive is a subfolder.

There is usually never a good reason pointing two syncing applications to the same path, so I am not sure I am buying the appeal argument.

[coolVariable]

Sorry - saying to not use the other (more useful app) is not a solution.

Like I said: syncthing now has TWO fundamental bugs/issues:

1. Syncthing should work with Onedrive folder/files (WITH files on demand) as long as they are synced to the device
2. Syncthing should be able to work with SD cards on android (this is now a 3 year old problem)

[AudriusButkevicius]

It has more issues than that, yet I fail to understand your point.

[coolVariable]

I know Syncthing has more issues than that. I agree.
It seems that recently syncthing has nothing but issues.

For people to close their eyes and say that an issue doesn't exist is not a solution, so this BUG should not be marked as closed.
Every other program can work with Win10 Fall Update just fine.

[AudriusButkevicius]

It's not a use case we are targeting, same way we don't sync directory mtimes, or ownership, or ACLs, or xattrs or resource forks/streams or symlinks on windows.

We have to pick our battles, and this is not one of them.

[calmh]

My understanding was that this issue was fixed. Is it not? In what way is it not?

[AudriusButkevicius]

He's talking about files inside the OneDrive folder. If I recall correctly, with the cloud business enabled, they don't look like normal files, even though they are downloaded on the local machine.

I had to revert the fall creators upgrade, so I can't verify now.

[calmh]

Right, so that’s the same as #1845 (deduplicated files on ntfs then). Which in that case indeed makes this issue non-solved; and likely non-solvable if windows insists on presenting these files as symlinks.

[clee231]

In my setup (Post Fall Creators Update), the files look like normal files, but the containing OneDrive folder looks like a symlink (at least in Golang). I don't actually think it's actually a symlink though... Since even Windows tools don't report it as being a link to another location. (It's marked as a reparse point, which isn't the same thing as a symlink.)

[AudriusButkevicius]

The upstream issue on #1845 is solved, yet we've had no feedback if it's fixed in our case or not.

[clee231]

Well, in Syncthing v0.14.41, I still exhibit the same "folder path missing" error message on my machine. (I still see very similar log messages as described initially in this issue and in the forums. )

Once I remove the reparse point metadata on the OneDrive folder, Syncthing functions normally again. Perhaps a check for reparse points in addition to symlinks is needed?

If you need me to test something on my system, let me know. Keep in mind that I don't really know Golang (but I'm learning), so simple checks is probably all I can manage for now. (File listing and symlink check from forums)

[Ferroin]

It all seems to end up being an issue with handling of NTFS reparse points, which provide (among other things) symbolic links, volume mount points, directory junctions, and single instance storage (the official MS name for deduplicated files). They are also used to provide the Windows-native equivalent of FUSE (more specifically, they act as an indirection to another filesystem translation driver which reads the associated data in the reparse point and then decides what to return), which in turn is how OneDrive works as of Windows 10 build 1709.

More generic info on reparse points can be found in the wikipedia page. I can't give much advice on handling though, except to comment that it's a serious pain in the arse.

[calmh]

The folder path being a "symlink" should be fine from our point of view; if it's something other magic that is a reparse point yet not a symlink that could be the issue. It would be neat if someone with the appropriate Windows could confirm one way or the other. Likewise on #1845.

I was going to propose (configurably, maybe) disabling the symlink check on Windows as today it mostly would not be a security issue to do so (only admins and devs can create them). But if it doesn't solve these issues it's pointless.

[Ferroin]

I can't confirm because I don't use OneDrive myself, but I'm fairly certain that the reparse point that is the root of a OneDrive directory is not a symlink, but instead an indirection to a filesystem translation filter that provides the sync integration with OneDrive. If it were a symlink, that would imply that it ultimately works in a manner similar to Dropbox or Syncthing, which is inconsistent with everything else I've heard about OneDrive low-level operation.

[AudriusButkevicius]

It used to be the case that reparse points were represented as symlinks in Go, I am not sure what's the current state.

[clee231]

@Ferroin - Is there any quick way to check this? To differentiate a reparse points from a Symlink-type or any of the other types of reparse points? I'm currently using OneDrive and Syncthing... but Syncthing reports the aforementioned error. I can attempt to test things if you let me know how.

When I use fsutils, I get the following when querying the OneDrive folder:

>fsutil reparsepoint query C:\Users\Chase\OneDrive
Reparse Tag Value : 0x9000701a
Tag value: Microsoft

Reparse Data Length: 0x00000054
Reparse Data:
0000:  01 00 54 00 46 65 52 70  6d 24 d3 6a 50 00 00 00  ..T.FeRpm$.jP...
0010:  02 00 07 00 07 00 01 00  48 00 00 00 0a 00 04 00  ........H.......
0020:  4c 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  L...............
0030:  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ................
0040:  00 00 00 00 00 00 00 00  00 00 00 00 01 00 00 00  ................
0050:  76 00 00 00                                       v...

>fsutil reparsepoint query C:\Users\Chase\OneDrive\.stfolder
Error:  The file or directory is not a reparse point.

@AudriusButkevicius - I think Go still reports the reparse points as a symlink... when it may not actually be truly a symlink. I still get the same results as what I listed in this gist.

[Ferroin]

I'm fairly certain that that's not a symlink based on what you've posted.

A symlink will give output similar to this:

Reparse Tag Value : 0xa000000c
Tag value: Microsoft
Tag value: Name Surrogate
Tag value: Symbolic Link

Reparse Data Length: 0x0000001c
Reparse Data:
0000:  08 00 08 00 00 00 08 00  01 00 00 00 74 00 65 00  ............t.e.
0010:  73 00 74 00 74 00 65 00  73 00 74 00              s.t.t.e.s.t.

So you'll have a Tag value: Symbolic Link line, and the reparse data will contain a UTF-16 path to the file (this one is a relative link to a file called 'test' in the same directory).

[zechnkaas]

@calmh

The folder path being a "symlink" should be fine from our point of view; if it's something other magic that is a reparse point yet not a symlink that could be the issue. It would be neat if someone with the appropriate Windows could confirm one way or the other. Likewise on #1845.

id did reply on 1845. it did not work in case of dedup filesystem.

[jarppiko]

Just wanting to document the exact settings that helped me to get Syncthing to sync a OneDrive root folder successfully with v0.14.41:

• Disabled OneDrive On-demand files feature
• Set OneDrive not to sync .stfolder
• Deleted the OneDrive cloud copy of .stfolder OneDrive had already synced

Afternoon these steps Syncthing v0.14.41 synced OneDrive root folder without any issues.

[Ferroin]

Hmm, that actually kind of makes sense that things would work without the on-demand files feature enabled. Without that feature on, OneDrive should work pretty similarly to Syncthing or Dropbox, and thus shouldn't have cause any issues for Syncthing.