Strange behavior of Directory.CreateDirectory

[FDUdannychen]

I found strange behaviors in Directory.CreateDirectory in my .net core2.0 app. My test code looks like:

var length255 = new string(Enumerable.Repeat('a', byte.MaxValue).ToArray());

var lastLength = 225;
var last = new string(Enumerable.Repeat('a', lastLength).ToArray());

var path = Path.Combine(@"D:\", Path.Combine(Enumerable.Repeat(length255, 127).ToArray()), last);            
            
try
{
    Console.WriteLine(path.Length);
    Directory.CreateDirectory(path);
}
catch (Exception e)
{
    Console.WriteLine(e.GetType());
    Console.WriteLine(Directory.Exists(path));
}

I change the value of lastLength a few times, here is the question:
Q1: lastLength = 224, path.Length = 32739 everything is working fine. What's the magic of 32739?

Q2: 225 <= lastLength <= 247, 32740 <= path.Length <= 32762 throws System.IO.PathTooLongException, meanwhile Directory.Exists prints False. However I can see the first few level directories are created(not all directories), which means the creation is partially done with an exception. Why?

Q3: 248 <= lastLength <= 251, 32763 <= path.Length <= 32766 cases are almost the same as those in Q2, except the exception is System.IO.DirectoryNotFoundException, why?

lastLength > = 252, path.Length >= 32767 fails with System.IO.PathTooLongException, no directory is created, which is as expected.

[EDIT] Add C# syntax highlight by @karelz

[danmoseley]

@JeremyKuhne can doubtless explain ...

[JeremyKuhne]

You have to know quite a bit about how Windows handles paths to fully understand what is happening. I blogged about path handling in great detail starting with https://blogs.msdn.microsoft.com/jeremykuhne/2016/04/21/path-format-overview/.

The bottom line is that there are no "real" path limits in most file systems (NTFS, FAT, ext4, etc.). Limits in Windows APIs are from a few sources:

1. Win32 MAXPATH define, which is 260 characters. A lot of code used this define to make fixed size buffers, including code in Windows itself.
2. The max size of UNICODE_STRING, which is the "native" string that Windows uses. The size field in that struct (which is in bytes) is a ushort. Unicode characters (which in Windows are UTF-16) take two bytes, which effectively gives you up to ushort.MaxValue / 2- e.g. 32767.
3. Win32 APIs don't take a size for incoming paths, which requires you to null terminate the passed in string. You effectively lose a character in capacity to the null.

The lower level Windows APIs (the NT apis) all use UNICODE_STRING. Win32 APIs convert the incoming string into a UNICODE_STRING, and historically would kick back paths that were greater than MAX_PATH. There was always an ability to skip this check using the \\?\ prefix, which .NET now adds for you implicitly. (Newer versions of Windows 10 now optionally support skipping the MAX_PATH check via opting in with a manifest. My blog also discusses this.)

While you might think adding the \\?\ prefix would take away available characters, this isn't the case. NT requires paths to be prefixed with \??\ internally. Our \\?\ gets changed to \??\, so it has no impact. If we didn't have the prefix, it would be added internally.

\??\C: is actually a symbolic link (alias) to the final NT path, which is something in the \device\ namespace. As the path is parsed in the object manager and the io manager in Windows it will change to the final path. This is dependent on a lot of internal details that are specific to how your machine is actually configured.

Bottom line is that there is no way to tell exactly what the limit will be. Eventually somewhere deep in Windows the code will either see that a path it wants to create won't fit and return the path too long error or it will end up truncating a segment of the path and find that foo doesn't exist when you asked for foobar (hence the directory not found).

When .NET creates directories we can't know ahead of time that a given path will be too long, even if it is longer than 32767 characters to start with, due to normalization and parsing that happens in Windows. Rather than block a fixed path length and block some legitimate paths we just always make the attempt and let Windows define/report the error. We just go through and create every subdirectory that doesn't exist. Trying to clean up would be problematic given the async nature of IO and how corner these cases are.

Fyi: As a general rule we don't try to second guess the errors the OS will give. Being over helpful blocks real customer scenarios as we won't always get it right (and will likely not get it right as the OS is always changing).

[danmoseley]

This seems answered.