maxNumberOfServerInstances of 1 implies FILE_FLAG_FIRST_PIPE_INSTANCE on Windows but not on Unix

[MV10]

Description

NamedPipeServerStream constructor's first-instance semantics differ by platform.

On Windows, passing maxNumberOfServerInstances: 1 causes FILE_FLAG_FIRST_PIPE_INSTANCE to be set automatically, so a second server requesting the same pipe name fails.

On Unix, that flag is derived only from PipeOptions.FirstPipeInstance, and maxNumberOfServerInstances is ignored for the purpose, so a second server unlinks the existing socket and rebinds, silently hijacking the pipe from the first server.

Reproduction Steps

Set maxNumberOfServerInstances: 1, and no PipeOptions.FirstPipeInstance:

using System.IO.Pipes;

var label = args[0];
try
{
    using var server = new NamedPipeServerStream("dupe-test", PipeDirection.InOut, 1, PipeTransmissionMode.Byte, PipeOptions.Asynchronous);
    Console.WriteLine($"{label}: server created");
    await Task.Delay(int.Parse(args[1]));
}
catch (Exception ex)
{
    Console.WriteLine($"{label}: {ex.GetType().Name}: {ex.Message}");
}

Run it twice:

./test "process A" 4000 &
sleep 1
./test "process B" 200

Note it must be two separate processes. If a pipe with the same name is created twice within the same process, the internal s_servers cache throws IOException "All pipe instances are busy".

Expected behavior

Consistent behavior on all platforms.

On Windows the second process fails, but on Linux the second succeeds.

Actual behavior

process A: server created
process B: server created

Regression?

Unknown.

Known Workarounds

Explicitly set PipeOptions.FirstPipeInstance ... if you happen to test with two copies of the same program and figure out why the first one stops working.

Configuration

• .NET SDK 10.0.301, runtime 10.0.9
• Debian Linux 6.12.74 (Devuan), x64
• Runtime source code from 10.0

Other information

For Windows, NamedPipeServerStream.Windows.cs sets the flag from the instance count, independent of PipeOptions:

int openMode = ((int)direction) |
               (maxNumberOfServerInstances == 1 ? 
               Interop.Kernel32.FileOperations.FILE_FLAG_FIRST_PIPE_INSTANCE : 0) |
               (int)options |
               (int)additionalAccessRights;

For Unix, NamedPipeServerStream.Unix.cs derives it only from the option:

bool isFirstPipeInstance = (pipeOptions & PipeOptions.FirstPipeInstance) != 0;

and when it is false, the SharedServer constructor unlinks whatever is at the path before binding:

if (!isFirstPipeInstance)
{
    Interop.Sys.Unlink(path); // ignore any failures
}

(Possibly worth noting I just filed a different bug report about the Unix SharedServer ctor here.)

PipeOptions.FirstPipeInstance is 0x00080000, the same value as Win32's FILE_FLAG_FIRST_PIPE_INSTANCE. So on Windows the explicit option is redundant with maxNumberOfServerInstances: 1, while on Unix it is the only thing that works. Callers who set the instance count but not the option (which I would argue is the natural reading of the API) only get protection on Windows.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-io
See info in area-owners.md if you want to be subscribed.