Make Process.Start have a option to change handle inheritance

[pdelvo]

Currently if you call Process.Start internally CreateProcess is called with bInheritHandles = true (hard coded). It would be great to make it possible to change this behavior, e.g. by adding a Property to ProcessStartInfo.

Currently there is no way I know of to change this other then reimplementing System.Diagnostics.Process.

Example

If you run this application twice without exiting the first notepad instance the second instance will not be able to open the tcp port, because notepad is still running. This can be a problem for server applications that are starting child processes themself and crash, or are killed by the user before the socket can be closed.

using System.Diagnostics;
using System.Net;
using System.Net.Sockets;

class Program
{
    static void Main()
    {
        TcpListener listener = new TcpListener(IPAddress.Any, 4567);
        listener.Start();

        Process.Start(new ProcessStartInfo("notepad.exe") { UseShellExecute = false });
        //Simulate application crash without freeing resources
    }
}

Design proposal

The easiest way to make this possible is to add a new Property to ProcessStartInfo and use this in the Call to CreateProcess

namespace System.Diagnostics
{
    public partial class ProcessStartInfo
    {
        public bool InheritHandles
        {
            get;  // defaults to true
            [MinimumOSPlatform("windows7.0")]
            set;
        }
    }
}

Questions

• Is there a very important reason why this was hardcoded like this in the first place?

[krwq]

I agree we should add this API. IMO we should also make TcpListener have option to change HandleInheritability and change the default behavior to not inherit handles if possible (at least when targeting newer version of the framework).

In most cases we should not change InheritHandles on Process and should rather do it whenever we create handles but we should still have an option to disable this.

[pdelvo]

I think adding options to TcpListener/TcpClient/Socket would be a good idea. Are there other places where handle inheritance can be a problem? I was not able to create a problem like the one mentioned above by exclusivly opening a file

[terrajobst]

This seems quite reasonable. As @krwq we should probably do a pass and look into similar classes that perform handle inheritability. This helps in understanding what API we could use.

[krwq]

@pdelvo, We were able to repro this with FileStream although that one creates non-inheritable handles by default so had to override that. I believe that every library should have non-inheritable handles by default but that might be hard to change at this point because of compat reasons

[ellismg]

One open question I have is what this would mean when we try to support this feature cross platform. What would it mean to say that handles are not inherited in Unix? Does "handles" here mean open file descriptors? If they are not to be inherited, does the framework have to find and close them in the child process? How does it interact with file descriptors marked FD_CLOEXEC?

[stephentoub]

I expect there are going to be a fair number of features in System.Diagnostics.Process that result in PlatformNotSupportedException; this might be one of them, if e.g. explicitly closing all fds above 2 doesn't work out or is prohibitive for some reason.

[terrajobst]

We've reviewed this proposal and don't believe it's ready yet. Please take a look at the notes.

[danmoseley]

Fixed link: https://github.com/dotnet/apireviews/tree/master/2015/01-14-misc#306-make-processstart-have-a-option-to-change-handle-inheritance

[Priya91]

@pdelvo @whoisj Would one of you be interested in working on the api proposal for this, responding to the last api review in @danmosemsft's link..

[whoisj]

@Priya91 yeah I could do something, but likely not until after VS 2017 ships (it is an all consuming effort). If @pdelvo wants to start on something, I'd be happy to collaborate as well.

What are the requirements for CoreFx API changes? Does the API need x-plat support, can there be Windows specific elements? Etc.

As a side note, I'd prefer to see a method added to System.Diagnostics.ProcessStartInfo like public void AddInhertiableHandles(IEnumerable<SafeHandle> handles), public void AddInhertiableHandles(IEnumerable<IntPtr> handles) than a misleading property like public bool InheritHandles { get; set; } because any time there's a redirection of pipes, inheritance needs to be enabled but consumers of the API do not likely mean to inherit every handle the parent process has.

[Priya91]

What are the requirements for CoreFx API changes? Does the API need x-plat support

We have the api addition process documented here. It also elaborates on the design principles. Yeah we do need x-plat support, as .NET Core supports Unix platforms as well.

can there be Windows specific elements? Etc.

Are you asking in terms of exposing an API only on Windows? We can't do that, although we have some windows specific APIs in .NET Core and throw PNSE on other platforms.

For now the focus should be more on the API design as you suggested in using a method over property etc, which will take shape once we understand the scenarios and requirements on all platforms.

[whoisj]

@Priya91 👍 and thanks.

[danmoseley]

@pdelvo do you want to update per the feedback and retry review? that would be welcome.

[karelz]

Next steps: We need to decide how to introduce Windows-only APIs. Ideally BCL is mostly platform agnostic. It is something we want in BCL.
Likely not good up-for-grabs candidate.

[wtgodbe]

@agocke had some interest in this issue with https://github.com/dotnet/corefx/issues/26711, anybody know what is the latest?

[ghost]

In node.js they use the concept from Win32 API DETACHED_PROCESS and CREATE_NEW_PROCESS_GROUP https://github.com/nodejs/node/blob/3a19122/deps/uv/src/win/process.c#L1089

See process creation flags: https://msdn.microsoft.com/en-us/library/windows/desktop/ms684863(v=vs.85).aspx

Then in Unix, they also use detached process from the options and then call setsid() to detach the process. https://github.com/nodejs/node/blob/3a19122/deps/uv/src/unix/process.c#L289

Maybe instead of calling it InheritHandles, use the term Detach?

cc @wfurt

[whoisj]

@kasper3 that's fine for NodeJs which is launch processes to behave like threads. Often, developers want to share the console's standard in/out/err pipes with the child. Additionally, neither DETACHED_PROCESS nor CREATE_NEW_PROCESS_GROUP prevent handle inheritance (to the best of my knowledge).

@karelz why not just have the Process.StartInfo take an IEnumerable<SafeHandle> (or similar) which then causes the handle exclusion logic used by each platform to whittle down the inherited handles to only the set specified by the enumerable (and maybe include standard handles as necessary).

[madelson]

According to this thread (brettwooldridge/NuProcess#13 (comment)), it sounds like Java is able to solve this issue in it's Process implementation:

The process implementation in Java uses JNI to first fork the JVM process, then the child JVM process performs a close of all file descriptors except for the pipes to the parent, and then executes the user-specified process, which replaces the child process in-situ (never to return) but inherits the file descriptors (now only the pipes to the parent).

Probably .NET could do something similar?

[whoisj]

@madelson on Linux? sure. On Windows? not so much.

[madelson]

@whoisj The OP calls out that today we call CreateProcess with bInheritHandles = true (hard coded). So on Windows I assumed the solution would be to just pass false for this parameter. From the discussion I thought the main issue was a concern that this behavior couldn't easily be replicated across OS's (e. g. @Priya91's comment about this API being Windows only).

[whoisj]

@madelson passing false for bInheritHandles doesn't work if the caller needs stream redirection.

Instead, I would think (for Windows) we'd want to focus on InitializeProcThreadAttributeList.

[jhudsoncedaron]

I know exactly why this bug exists. The Process.Start() code was written originally to run on Windows 9x where you literally couldn't fix this. The API for actually fixing this on Windows was added on Windows Vista, but nobody ever revisited this code. And when the port to Linux was done it was done directly, emulating the bug, probably without thinking about it.

Anybody using blind inherit handles from any thread other than the main thread is bugged and doesn't know it yet.

[karelz]

@jhudsoncedaron do you have specific changes in mind?
.NET Core 2.1+ supports only Win7+: https://github.com/dotnet/core/blob/master/release-notes/2.1/2.1-supported-os.md#windows so using Vista API should be reasonable.

[jhudsoncedaron]

Yup!

How to fix (Windows): https://blogs.msdn.microsoft.com/oldnewthing/20111216-00/?p=8873
More problems caused by this: https://blogs.msdn.microsoft.com/oldnewthing/20131018-00/?p=2893

To fix on Unix, the child process needs to enumerate over /proc/self/fd and close any integers > 2 except for the handle being used to enumerate the directory itself.

[stephentoub]

This is not a "bug", which suggests you could just change the implementation... doing that would break an absolute ton of code. If we want to add an option to the API (and work through any associated issues), that's fine, but we must not change the existing behavior of the existing API. And we absolutely thought about the behavior when implementing this for Unix.

[jhudsoncedaron]

@stethentoub: If you create two child process from two separate threads without a lock on the unix platform, your code can hang. I'm not certain but I think the build server code in dotnet is affected. It's difficult to come up with a case on the Windows platform where someone could be depending on random handles being inherited because only socket handles are inherited.

[stephentoub]

It's difficult to come up with a case on the Windows platform where someone could be depending on random handles being inherited

e.g. anonymous pipes

If you create two child process from two separate threads without a lock on the unix platform, your code can hang.

Code can hang for lots of reasons. My point is, the implementation of the API is not buggy; it's doing exactly what it's defined to do. You may not like the definition, but that's different than saying there's a bug in the implementation.

[jkotas]

Can this API be implemented on Windows?

I do not think this is implementable on Windows. Windows only lets you specify the subset of inheritable handles to inherit for given process (look for PROC_THREAD_ATTRIBUTE_HANDLE_LIST).

[tmds]

Thank you for the pointer @jkotas.

I took a look at the docs. From what I read, on Windows we can't control what file descriptor the handle becomes in the child process. We can drop that requirement.

The rest should be doable by setting hStdInput, hStdOutput, hStdError based on the first three items in Handles, and passing remaining handles via PROC_THREAD_ATTRIBUTE_HANDLE_LIST.

The approved InheritHandles is meaningful on its own.

I'll write up a separate issue about the Handles use-case.

[danmoseley]

Going through stale issues -- this is well understood and can be up for grabs, right? Anyone interested in offering a PR?

[deeprobin]

@danmoseley I think this issue is important for quite a lot of users (see 👍🏼-Reactions of the top post).
Can we schedule this for .NET 7? Or is it already too late for that?

[tmds]

@deeprobin fyi, because .NET opens handles with CLOEXEC, this is less of a problem now on Linux than when the issue was first created.

Future milestone means Microsoft doesn't have it planned for .NET 7. Community can still make the contribution.

[wfurt]

Is it even feasible @tmds? It feels like we would need to change all existing open handles, right? (e.g. undo CLOEXEC)

I'm wondering if we should focus on API to pass specific handle to child process. (and close this)

[jkotas]

Future milestone means Microsoft doesn't have it planned for .NET 7. Community can still make the contribution.

@deeprobin Feel free to grab this issue if you are interested.

Is it even feasible @tmds?

It is not reasonably possible to implement this proposal on non-Windows. It is why the property is marked as Windows-specific. It is addressing a specific real problem on Windows.

I'm wondering if we should focus on API to pass specific handle to child process. (and close this)

It is very hard to design a uniform API set around this. Windows and Unix treat handles very differently during create process.

[jhudsoncedaron]

@jkotas : The proposed API from ages ago that you give a list of handles to inherit can be implemented on both. You can undo CLOEXEC in the fork child with fcntl calls on Unix, and you can use the CreateProcess thing that gives the list of handles to inherit on Windows.

[tmds]

Is it even feasible @tmds? It feels like we would need to change all existing open handles, right? (e.g. undo CLOEXEC)

I think the main use-case for this API is to prevent handles from leaking into the child by setting InheritHandles = false.

So in case of InheritHandles = true we can keep doing what we're currently doing (which is good for backwards compat).

And in case of InheritHandles = false we'd close all fds > 2.

I'm wondering if we should focus on API to pass specific handle to child process. (and close this)
It is very hard to design a uniform API set around this. Windows and Unix treat handles very differently during create process.

On Unix, the child process expects the caller to pass files as specific fds (by using dup).

On Windows, the handle value stays the same. And it is communicated to the child process for example via an argument as shown in the documentation of AnonymousPipeServerStream: https://docs.microsoft.com/en-us/dotnet/api/system.io.pipes.anonymouspipeserverstream?view=net-6.0.

See #13943 (comment) and next comments about a potential Handles API.

[deeprobin]

@deeprobin Feel free to grab this issue if you are interested.

I can take a look at the Issue. I think the Windows implementation will not be that difficult.

@tmds is the whole thing possible on Unix to the extent of this API?

[joshudson]

@deeprobin : I can implement on Linux even on very old kernels. Need somebody else for Mac OSX though. It is not allowed to call malloc() and therefore opendir() from inside a fork() child when the parent process is multithread because of deadlocks; thus making the implementation system-specific.

[deeprobin]

I run into a small problem in the Windows implementation.

Kernel32.CreateProcess has the possibility to pass an argument for InheritHandles, but this is not possible if the process should be created for a certain user (Advapi32.CreateProcessWithLogonW).
Is there a possibility to tell Windows that handles should be inherited?

[deeprobin]

Also, I'm not sure how to proceed if the property UseShellExecute = true as well as InheritHandles = true, which technically is not possible through the Windows API.
Throw an exception? Ignore?

/cc @danmoseley @terrajobst

[jhudsoncedaron]

@deeprobin : I'd expect a throw if both UseShellExecute and InheritHandles are set.

CreateProcessWithLogonW is bugged in Windows. What I end up having to do is give it an executable that I control and fix stuff up on the other side.

[whoisj]

CreateProcessWithLogonW is bugged in Windows.

that is an incredibly bold statement, and I am not sure it is correct. in fact, I highly doubt that it is correct. an API as fundamental as CreateProcessWithLogonW is very unlikely to be "bugged". It is far more likely that it just doesn't behave in a manner that you might expect or prefer.

You might notice that I was asked to propose an API for this, but that never happened because I left MSFT shortly thereafter for my current role. Even so, the best method to achieve what people are looking for here is to use InitializeProcThreadAttributeList and mostly ignore bInheritHandles because the later conflicts with redirected input/output.

[jhudsoncedaron]

@whoisj : The time has long passed to ship a service that provides new API endpoints for login management. I have been faced with too many arbitrary limitations because no API at all does what is necessary and everybody hates it when you reach for NtCreateToken because there is no other.

One scenario: My code has admin rights and needs to start a process as a user that it has username and password for, but the user only has privileges to log in as a service. I have STDIN, STDOUT, and STDERR redirects that need to be placed. I ought to be able to do this, but I can't because of arbitrary restrictions.

[whoisj]

@jhudsoncedaron if you're on Windows there are advapi32!SaferCreateLevel and advapi32!CreateProcessAsUserW which do most of what you want. You'll likely have to use kernel32!CreateNamedPipeW to create proper stdin, stdout, and stderr pipes to make it work ; but it CAN work - I've done it. That how Visuals Studio executes git.exe when you use Team Explorer.

[jhudsoncedaron]

@whoisj : CreateProcessAsUserW requires SE_ASSIGNPRIMARYTOKEN_NAME; which admins don't have. I tried that already. I was able to get some parts of it working by spawning a service, but ultimately process ends up in the wrong session if I do that.

[whoisj]

when user run Visual Studio elevated (many do), VS runs git not elevated, and it uses CreateProcessAsUserW to do so.

these are just regular users using the "Run as administrator" option - as you know, Windows does have needless "root" accounts - so they're just running as an elevated version of their normal user account.

This works just fine for Visual Studio. There's no magic "secret sauce" or "hidden API" involved, it's all public stuff.

[jhudsoncedaron]

@whoisj: Oh. You missed "user only has privileges to log in as a service" out of the scenario.

[whoisj]

ah my mistake. not sure how to resolve that issue as I've never tried. sorry.

[kyle-rader]

To beat a dead horse, this was the root cause of a tricky bug in a CLI framework we use internally that handles automatic telemetry collection. To avoid blocking your shell while waiting to send that telemetry, the framework spawns an instance of the CLI with event data so it can go send it in the background and not wait (fire and forget). This works great at a shell/terminal. Shaves off almost 500ms of runtime. It also works great when calling these tools from other languages where handles are not being inherited automatically. But if another Dotnet process, shells out to one of these apps, it ends up getting blocked on both the main CLI process and any children telemetry processes. We've mitigated this by moving to UseShellExecute = true, but now we can't stream event data over stdin :(.

[krwq]

Since this API is approved I'd suggest if anyone is interested to contribute. Even throwing PNSE on platforms you currently don't plan to support would improve the state. Most people don't need this and it's only useful when you actually encounter bugs like mentioned above. Make sure to look at the API notes and let us know if you plan to pick this up so that no one duplicates work. For now I can set milestone to 9.0 so this issue is considered for 9.0 (given number of upvotes, weird issues affected people can see, how long this has been open and how relatively cheap it is to add support at least on Windows - haven't investigated Linux/macOS) - note this doesn't guarantee we will actually do anything in 9.0 but we can at least consider this - we'll look at the area holistically and pick which issues are most important.