Add the ability to connect to extra stdIO streams/pipes

[kblok]

Context

I'm trying to connect to Chromium using the flag --remote-debugging-pipe. When that flag is used, Chromium will listen and write to pipes using the stdIO/File Descriptors 3 and 4 (https://github.com/chromium/chromium/blob/a5f23780e29ab3d8f58269bd69bdc297beeed2f9/content/browser/devtools/devtools_pipe_handler.cc#L32).

Problem

I found no way to connect to those pipes using the Process class. So I went to the Console Class to see how the Standard Output is being opened.
The problem is that, from that point, going deeper, all the classes and methods are internal.
I can't use Interop.Sys.Dup to open the FD myself. I can't use SafeFileHandleHelper to open streams like Console does.

So my questions would be:

• Am I looking at the wrong place? I took at AnonymousPipeServerStream, but I found no way to connect to stdIO 3 and 4 from there.
• Is this something I should be able to do using .NET?
• Is there any chance to expose these classes to the final user?

[stephentoub]

Stream s = new FileStream(new SafeFileHandle((IntPtr)3, ownsHandle: false), FileAccess.ReadWrite);

?

[kblok]

@stephentoub

var sRead = new FileStream(new SafeFileHandle((IntPtr)3, ownsHandle: false), FileAccess.ReadWrite);
...
var buffer = new byte[2048];
sRead.Read(buffer, 0, 2048);

I'm getting an IOException Is a directory.

[stephentoub]

Are you sure your process has file descriptor 3 as you expect it? When your process is running, what do you see as output from doing:

ls -al /proc/yourprocid/fd/

where yourprocid is the id of the process in which this code is running?

It sounds like file descriptor 3 is something else.

(I'm assuming you're running on Linux... the above won't work if you're on mac.)

[stephentoub]

Oh, maybe I misunderstood the original question. Are you saying that some other process has file descriptors 3 and 4 open and you want to somehow connect to those? You would need to know what interprocess mechanism that other process is using on those file descriptors. In your process, that other process' fds 3 and 4 have no meaning at all; they're just the numbers that process is using. You would need to use whatever mechanism it's expecting to connect.

[kblok]

I'm on a mac.
In NodeJS I just need to register 2 more pipes in the stdio (https://nodejs.org/api/child_process.html#child_process_options_stdio).

There from the code it's just about declaring 2 pipes and that's all https://github.com/GoogleChrome/puppeteer/blob/master/lib/Launcher.js#L115

I'm trying to mimic that.

[kblok]

Maybe I am the one who needs to create those FDs.

[stephentoub]

As far as I can tell, that's all using NodeJS-specific mechanisms / implementation / patterns / helpers, with node being used in both the parent and child process, and thus it using its own agreed upon handshake for how to set up these pipes. You would need to fully understand the implementation in order to simulate it.

[kblok]

@stephentoub well, Chromium is not a NodeJS app.
I think I should be able to read/create those FDs from .NET.

[stephentoub]

Let me rephrase my questions then:

• Where is this protocol documented? Or is this just you reverse engineering it? What is the protocol?
• Inferring from your comments, it sounds like the child process expects that its file descriptors 3 and 4 have already been configured by the parent process as anonymous pipes, with the parent process owning the other ends?
• And it's your process that's then spawning Chromium as the child process using the Process class?

[kblok]

That's what I'm trying to understand. I don't play with this kind of stuff every day, and I need to set the boundaries between "NodeJS idioms" and standard behaviors.

That's why I tried to make this issue as generic as possible, because these guys are using standard File Descriptors, which I think it's the main question: "How can I create/read/write some other FDs besides stdIn stdOut and stdError".

I first need to be able to connect these two FD before start digging into the protocol itself.

In NodeJS that looks pretty transparent: "Those pipes are streams like StandardInput and StandardOutput".
Then I went to Chromium itself (C code) and the code also looked straight forward (https://github.com/chromium/chromium/blob/a5f23780e29ab3d8f58269bd69bdc297beeed2f9/content/browser/devtools/devtools_pipe_handler.cc#L176). It "just" writes and reads a FD.

[stephentoub]

these guys are using standard File Descriptors

I think this is the part that's confusing the issue, or at least confusing me :) There are only three "standard" file descriptors: stdin (0), stdout (1), stderr (2). And all that means is that apps should expect those file descriptor numbers to map to those concepts, so that when a parent process launches a child process and configures the child process, it can set up 0/1/2 in the child process appropriately and the child process can similarly expect that 0/1/2 mean what it thinks it means.

It "just" writes and reads a FD.

Ok, so that code in the child process is expecting that the parent process has configured its file descriptors 3 and 4.

I first need to be able to connect these two FD

When a process starts a child process, it does so by fork'ing it. That creates an almost complete copy of the parent process. That child process then detects that it's the forked version, does whatever configuration it needs to do, and calls exec to invoke an executable that overwrites what's currently running, but file descriptors remain open (unless they were marked as O_CLOEXEC).

So, for example, if a parent process wants to set up the child process' stdin, stdout, and stderr to be pipes that the parent process can read/write, for each it would call pipe to create pipes, giving the parent process file descriptors for both ends of the pipe, and then once the child process has been created via fork and inherited those file descriptors, it dups them onto the desired target file descriptor number, e.g. the child end of the pipe created by the parent for stdout will be dupd onto fd 1. Then when the actual target executable starts running and refers to stdout, it's referring to the file descriptor dup'd onto 1. You can see all of the logic for this as performed by the Process class in the native code here:
https://github.com/dotnet/corefx/blob/24b9efcd1344949f4bbbf66b9b41e38d4c5271a2/src/Native/Unix/System.Native/pal_process.c#L142
This is done in native code because it's not safe to use fork from managed code.

Thus, what I believe you're asking for is some way to augment the Process class, Process.Start, and the above SystemNative_ForkAndExecProcess logic to say "please create an additional N pipes and dup the child ends onto these specific additional fd numbers I provide, then return to me Streams from the Process returned from Process.Start that wraps the parent ends of those pipes".

Have I understood correctly? If so, that's a tall ask. In particular, I don't know what that would mean in a Windows world where things operate very differently, so such functionality would likely end up being Unix-specific on a type that's meant to work as equivalently as possible everywhere. It also seems to be super-specific to this one scenario, though admittedly if NodeJS added support for this, maybe they had additional known scenarios in mind. But, never say never. If you have a specific API in mind for accomplishing this, please share what you would want it to look like.

Given all that, to answer your three original questions:

Am I looking at the wrong place?

Yes. This isn't related to how Console opens the current process' stdin/stdout/stderr. It's about configuring a spawned child process' file descriptors after fork'ing and prior to exec'ing the child.

Is this something I should be able to do using .NET?

No, I don't know of any way to do what you want reliably today.

Is there any chance to expose these classes to the final user?

Those classes aren't what you want and wouldn't help you here. As noted above, the code that spawns processes would need to be modified to support creating and dup'ing additional file descriptors after the fork / before the exec, APIs would need to be added to specify how many additional pipes are needed and what file descriptors in the child process they should be mapped to, and APIs would need to be added to get back Streams for the resulting parent ends of those pipes.

[kblok]

Thus, what I believe you're asking for is some way to augment the Process class, Process.Start, and the above SystemNative_ForkAndExecProcess logic to say "please create an additional N pipes and dup the child ends onto these specific additional fd numbers I provide, then return to me Streams from the Process returned from Process.Start that wraps the parent ends of those pipes".

Exactly. That would be the feature I need. But it's also important for me to know if this is something I can accomplish now or not.

Thanks for taking the time to go through this issue.

[wtgodbe]

@kblok I'm closing this as abandoned, please re-open if you still have interest in this issue.