PathTooLongException when using USNJournal

[jojje]

• [X ] I have searched open and closed issues for duplicates.

---

Environment info

• Duplicati version: 2.0.3.9_canary_2018-06-30
• Operating system: Windows 7 Pro (64 bit)
• Backend: "Local folder or drive"

Description

When re-running a backup set which contains long file/folder names, and using the Journal USN feature introduced in #3184, the backup fails with a System.IO.PathTooLongException.

Full scan backups of the set works however, such as when making the initial backup, or when changing the include / exclude filters for the set, so the problem only seems to occur when the USN Journal code branch is triggered.

Steps to reproduce

1. Create a directory or filename with an absolute path longer tan 240 or 255 characters respectively.
2. Create a backup set that includes the dir / folder and set the advanced option usn-policy to Required.
3. Run the backup set two times.

• Actual result:
  The second time the backup is executed it fails with the exception System.IO.PathTooLongException.

• Expected result:
  Successful backup without the mentioned exception.

Debug log

2018-07-05 18:39:42 +02 - [Error-Duplicati.Library.Main.Operation.BackupHandler-FatalError]: Fatal error
System.IO.PathTooLongException: The specified path, file name, or both are too long. The fully qualified file name must be less than 260 characters, and the directory name must be less than 248 characters.
   at System.IO.PathHelper.GetFullPathName()
   at System.IO.Path.LegacyNormalizePath(String path, Boolean fullCheck, Int32 maxPathLength, Boolean expandShortPaths)
   at System.IO.Path.GetFullPathInternal(String path)
   at System.IO.DirectoryInfo.Init(String path, Boolean checkHost)
   at Duplicati.Library.Snapshots.USNJournal.GetVolumeRootFromPath(String path)
   at Duplicati.Library.Snapshots.UsnJournalService.IsPathEnumerated(String path)
   at Duplicati.Library.Main.Operation.BackupHandler.<>c__DisplayClass12_0.<RunMainOperation>b__2(String path, Int64 fileSize)
   at Duplicati.Library.Main.Database.LocalBackupDatabase.AppendFilesFromPreviousSetWithPredicate(IDbTransaction transaction, Func`3 exclusionPredicate, Int64 fileSetId, Int64 prevFileSetId, DateTime
timestamp)
   at Duplicati.Library.Main.Database.LocalBackupDatabase.AppendFilesFromPreviousSetWithPredicate(IDbTransaction transaction, Func`3 exclusionPredicate)
   at Duplicati.Library.Main.Operation.Common.SingleRunner.<>c__DisplayClass6_0.<RunOnMain>b__0()
   at Duplicati.Library.Main.Operation.Common.SingleRunner.<>c__DisplayClass5_0`1.<<DoRunOnMain>b__1>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Duplicati.Library.Main.Operation.BackupHandler.<RunMainOperation>d__12.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>d__19.MoveNext()

[jojje]

Possible avenues for tackling this windows specific nuisance:

• A) Use different versions of Windows file APIs that allow up to 32K character length, instead of the insufficiently small 260 char limit otherwise enforced by the windows API and dotnet libraries. See this library for someone who has already wrapped the appropriate APIs.
• B) Have user's install a microsoft provided hotfix from 2014.
• C) Have users seek out and delete the offending files / dirs, as unhelpfully proposed by Crashplan support for this sort of issue.

It looks as if it may be possible to configure the (duplicati) dotnet assembly so that it copes with long paths, however I'm not a dotnet developer and don't have the MS tools nor knowledge to try this out. From the MS article linked regarding unicode.

<application xmlns="urn:schemas-microsoft-com:asm.v3">
    <windowsSettings xmlns:ws2="http://schemas.microsoft.com/SMI/2016/WindowsSettings">
        <ws2:longPathAware>true</ws2:longPathAware>
    </windowsSettings>
</application>

[verhoek]

Thanks for the detailed bug report :)

It looks like this originates from the call System.IO.DirectoryInfo(path).Root.FullName from within USNJournal.GetVolumeRootFromPath , which is called from several places. Possibly other places raise similar issues once this call is fixed though.

@kenkendk, @dgehri:
Why is Path.GetPathRoot(path) not used here instead? GetPathRoot doesn't seem to be able to throw a path too long exception from its specification, and seems to be more direct than to build a directoryinfo object.

I cannot test this right now since I'm on the road with only Linux at hand. On the weekend I will look into this.

[NTWarrior]

I have just encountered this error; while ideally the error wouldn't occur, it would be helpful if the log at least identified the specific path(s) which is too long.

[dgehri]

@verhoek : I just encountered the problem myself. I suggest we do as you suggest and switch to using Path.GetPathRoot(path). Can you submit a pull-request?

[verhoek]

Yes I will, either tonight or tomorrow.
Edit: I just saw you already have a pull request pending here, so I closed mine.

[Taomyn]

Any idea when a new canary build will be released with this fix? I just hit this issue and without knowing what file is the cause so I can maybe fix it, my backup is no longer completing successfully.

[NTWarrior]

After the first time this happened I used Everything by VoidTools to find and rename or delete the items with overlong paths in my backup range, but I am still getting the same error. I have some full paths that are between 248 and 260 characters long, but in none of those are the directories above 248 characters.

System.IO.PathTooLongException: The specified path, file name, or both are too long. The fully qualified file name must be less than 260 characters, and the directory name must be less than 248 characters.
   at System.IO.PathHelper.GetFullPathName()
   at System.IO.Path.LegacyNormalizePath(String path, Boolean fullCheck, Int32 maxPathLength, Boolean expandShortPaths)
   at System.IO.Path.GetFullPathInternal(String path)
   at System.IO.DirectoryInfo.Init(String path, Boolean checkHost)
   at Duplicati.Library.Snapshots.USNJournal.GetVolumeRootFromPath(String path)
   at Duplicati.Library.Snapshots.UsnJournalService.IsPathEnumerated(String path)
   at Duplicati.Library.Main.Operation.BackupHandler.<>c__DisplayClass12_0.<RunMainOperation>b__2(String path, Int64 fileSize)
   at Duplicati.Library.Main.Database.LocalBackupDatabase.AppendFilesFromPreviousSetWithPredicate(IDbTransaction transaction, Func`3 exclusionPredicate, Int64 fileSetId, Int64 prevFileSetId, DateTime timestamp)
   at Duplicati.Library.Main.Database.LocalBackupDatabase.AppendFilesFromPreviousSetWithPredicate(IDbTransaction transaction, Func`3 exclusionPredicate)
   at Duplicati.Library.Main.Operation.Common.SingleRunner.<>c__DisplayClass3_0.<RunOnMain>b__0()
   at Duplicati.Library.Main.Operation.Common.SingleRunner.<DoRunOnMain>d__2`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Duplicati.Library.Main.Operation.BackupHandler.<RunMainOperation>d__12.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at CoCoL.ChannelExtensions.WaitForTaskOrThrow(Task task)
   at Duplicati.Library.Main.Controller.<>c__DisplayClass13_0.<Backup>b__0(BackupResults result)
   at Duplicati.Library.Main.Controller.RunAction[T](T result, String[]& paths, IFilter& filter, Action`1 method)
   at Duplicati.Library.Main.Controller.Backup(String[] inputsources, IFilter filter)
   at Duplicati.Server.Runner.Run(IRunnerData data, Boolean fromQueue)

EDIT: Still failing with same error after getting directories and absolute paths under 240 and 255 character respectively as in OP

[Taomyn]

Just upgraded to 2.0.3.12_canary_2018-10-23 which I thought had the fix, got the same error:

System.IO.PathTooLongException: The specified path, file name, or both are too long. The fully qualified file name must be less than 260 characters, and the directory name must be less than 248 characters. at System.IO.Path.LegacyNormalizePath(String path, Boolean fullCheck, Int32 maxPathLength, Boolean expandShortPaths) at System.IO.Path.GetPathRoot(String path) at Duplicati.Library.Snapshots.UsnJournalService.IsPathEnumerated(String path) at Duplicati.Library.Main.Operation.BackupHandler.<>c__DisplayClass12_0.<RunMainOperation>b__2(String path, Int64 fileSize) at Duplicati.Library.Main.Database.LocalBackupDatabase.AppendFilesFromPreviousSetWithPredicate(IDbTransaction transaction, Func3 exclusionPredicate, Int64 fileSetId, Int64 prevFileSetId, DateTime timestamp)
at Duplicati.Library.Main.Database.LocalBackupDatabase.AppendFilesFromPreviousSetWithPredicate(IDbTransaction transaction, Func3 exclusionPredicate) at Duplicati.Library.Main.Operation.Common.SingleRunner.<>c__DisplayClass3_0.<RunOnMain>b__0() at Duplicati.Library.Main.Operation.Common.SingleRunner.<DoRunOnMain>d__21.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Duplicati.Library.Main.Operation.BackupHandler.d__12.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Duplicati.Library.Main.Operation.BackupHandler.d__19.MoveNext() --- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at CoCoL.ChannelExtensions.WaitForTaskOrThrow(Task task)
at Duplicati.Library.Main.Controller.<>c__DisplayClass13_0.b__0(BackupResults result)
at Duplicati.Library.Main.Controller.RunAction[T](T result, String[]& paths, IFilter& filter, Action1 method) at Duplicati.Library.Main.Controller.Backup(String[] inputsources, IFilter filter) at Duplicati.Server.Runner.Run(IRunnerData data, Boolean fromQueue)

[verhoek]

Apparently then also getpathroot triggers this exception, though this is not documented.

I don't see a clean solution to this problem: things like adding to the registry a certain option to fix this for .NET is not so clean (and only works on windows 10?).
Best would be imho to use a third party lib like ZetaLongPaths that works for all windows users.

EDIT: It seems we're already using such a library in other places, namely AlphaFS. I will create a pull request using AlphaFS.

[kenkendk]

It is so ridiculous that .Net still uses the old Windows API with the 260 character limit.
I think this attempts to explain why it has not been fixed, but really....:
https://github.com/dotnet/corefx/issues/13294

We can add support for long paths as explained here:
https://blogs.msdn.microsoft.com/jeremykuhne/2016/07/30/net-4-6-2-and-long-paths-on-windows-10/

But it still requires users to tweak their machine settings, so for now the best approach is probably just to avoid all calls directly to System.IO and go through some layer, like AlphaFS.

[TheFatherMind]

Is there a reason that the app is not spitting out the rejected path (or part of it) into the log so we can see where the failure is happening? I have a computer with this problem and I cannot for the life of me figure out what file it is. Also the hotfix mentioned has been taken down by Microsoft so that people will get windows 10. That annoys me to no end.

Update: So I ended up having to delete the entire backup and start over. Whatever it was, it was not coming from the folders being backed up.