Officially support stdin #321

Closed
Chuongv opened this Issue Dec 8, 2015 · 9 comments

Comments

Projects
None yet
6 participants
@Chuongv

Chuongv commented Dec 8, 2015

Opening an issue as per your request: It would be nice if mio could support stdin

@dpc

This comment has been minimized.

Show comment
Hide comment
@dpc

dpc Dec 9, 2015

Contributor

It is possible to handle stdin in mio, just in case that's a problem. I have a project that does just that: https://github.com/dpc/colerr/blob/master/src/iomuxer.rs#L26

But I guess some wrappers to support Windows etc. would be nice, yes. Obviously stderr and stdout should be supported too.

Contributor

dpc commented Dec 9, 2015

It is possible to handle stdin in mio, just in case that's a problem. I have a project that does just that: https://github.com/dpc/colerr/blob/master/src/iomuxer.rs#L26

But I guess some wrappers to support Windows etc. would be nice, yes. Obviously stderr and stdout should be supported too.

@carllerche

This comment has been minimized.

Show comment
Hide comment
@carllerche

carllerche Dec 14, 2015

Owner

This is going to require more research into how STDIN & co works w/ IOCP. I will tentatively assign this to the 1.0 milestone, but will potentially have to punt if it is tricky.

Owner

carllerche commented Dec 14, 2015

This is going to require more research into how STDIN & co works w/ IOCP. I will tentatively assign this to the 1.0 milestone, but will potentially have to punt if it is tricky.

@carllerche carllerche added this to the v1.0 milestone Dec 14, 2015

@alexchandel

This comment has been minimized.

Show comment
Hide comment
@alexchandel

alexchandel Feb 2, 2016

It would be nice if mio added both stdin and file operations. It's justifiable since mio is already a good interface to kqueue/epoll/iocp. Pipes and files are treatable by kqueue and epoll similar to network operations. And files (and named pipes) work with iocp just fine.

Unfortunately standard handles are the exception, and require extra treatment. For some horrible Windows reason, "console" handles and anonymous pipes don't work with IOCP. For console input, there is GetNumberOfConsoleInputEvents, PeekConsoleInput, or WaitForMultipleObjects, and for anonymous pipe input, there is PeekNamedPipe or WaitForMultipleObjects.

alexchandel commented Feb 2, 2016

It would be nice if mio added both stdin and file operations. It's justifiable since mio is already a good interface to kqueue/epoll/iocp. Pipes and files are treatable by kqueue and epoll similar to network operations. And files (and named pipes) work with iocp just fine.

Unfortunately standard handles are the exception, and require extra treatment. For some horrible Windows reason, "console" handles and anonymous pipes don't work with IOCP. For console input, there is GetNumberOfConsoleInputEvents, PeekConsoleInput, or WaitForMultipleObjects, and for anonymous pipe input, there is PeekNamedPipe or WaitForMultipleObjects.

@raphlinus

This comment has been minimized.

Show comment
Hide comment
@raphlinus

raphlinus Aug 11, 2016

I'll put in a vote for this. I'm currently using a thread-heavy design in xi-editor, which is the source of more overhead than I'd like. Now that futures-rs is out, basing my RPC mechanism on that is very appealing, but I also very much don't want to have separate code paths on Windows.

A small amount of discussion in this reddit thread.

I'll put in a vote for this. I'm currently using a thread-heavy design in xi-editor, which is the source of more overhead than I'd like. Now that futures-rs is out, basing my RPC mechanism on that is very appealing, but I also very much don't want to have separate code paths on Windows.

A small amount of discussion in this reddit thread.

@alexcrichton alexcrichton referenced this issue in rust-lang-nursery/futures-rs Aug 12, 2016

Closed

Using futures::stream::Sender in a loop #49

@alexchandel

This comment has been minimized.

Show comment
Hide comment
@alexchandel

alexchandel Sep 5, 2016

stdin probably has to be treated separately, because the Windows implementation must be different for stdin. But it is possible to read stdin non-blocking on Windows. I believe it must be polled with WaitForMultipleObjects.

stdin probably has to be treated separately, because the Windows implementation must be different for stdin. But it is possible to read stdin non-blocking on Windows. I believe it must be polled with WaitForMultipleObjects.

@alexcrichton alexcrichton referenced this issue in alexcrichton/tokio-process Dec 29, 2016

Closed

tokio-stdio - what do you think of this? #7

@timClicks

This comment has been minimized.

Show comment
Hide comment
@timClicks

timClicks Jan 12, 2017

Decided to look into this and take a look at how a few other projects have proceeded. Ended up with a 1,700 word blog post on IOCP and stdio.

@alexchandel I didn't encounter much use of WaitForMultipleObjects, but then again I am not very familiar with the Win32 API. If you could can anything, feel free.

Decided to look into this and take a look at how a few other projects have proceeded. Ended up with a 1,700 word blog post on IOCP and stdio.

@alexchandel I didn't encounter much use of WaitForMultipleObjects, but then again I am not very familiar with the Win32 API. If you could can anything, feel free.

@alexchandel

This comment has been minimized.

Show comment
Hide comment
@alexchandel

alexchandel Jan 13, 2017

@timClicks

Your tldr was about my conclusion as well.

The main problem is that keys don't arrive to the "console stdin" of a process as a byte stream, like they do on normal OSs. Instead, when you type into a "console window" on Windows, they arrive in some Win32 input event queue of the process, along with mouse events, menu events, context-change events, and other garbage. This queue receives them in the form of key-up and key-down events, and even receives them for backspace and the arrow-keys.

When you call getc, fread, std::getline, ReadFile, ReadConsole, or anything else that should read characters from stdin, those call into a Win32 function that processes the input queue, which includes handling arrow-keys, backspaces, command history, and line discipline, and which eventually returns a character or line (depending on what you asked for) or blocks until it can.

Note that cmd.exe itself doesn't handle any key input. Note that there is no Win32 function that will perform this input event processing but NOT block if no character/line is available. Note that not even conhost.exe, the process that draws the "console" window in the first place (cmd.exe is just a batchfile language interpreter really), can be told to parse keyboard events and send them as a byte stream.

This horrible, atrocious design is why cmd.exe cannot ever have command history, why lines don't appear when you type them unless the process is blocking reading, and why you cannot do IOCP on "console stdin". You need a thread to process the events & do line discipline.

What you can do with WaitForMultipleObjects is poll the input queue, and when events are available, filter out the keyboard events and use those as character inputs in lieu of getc. Since you're just taking the key-ups (or key-downs) directly, that means character-echoing, arrow-keys, and backspace won't work, unless you explicitly implement your own input buffer and echo it back to the console. So it's a nasty hack, and probably useful only for console/terminal games like nethack or dwarf fortress, which obviously don't need line discipline, line history, arrow-keys, key-echoing, or anything else.

In cases where you can't poll, or where filtering key events is a pain or too costly to do on your thread, creating another thread that reads stdin blockingly and posts it in on a named pipe/tcp port/IOCP-compatible thing seems like the only way.

The Twisted solution in your blog post doesn't work in the general case, since you need to change your parent/input source as well, which is generally impossible (e.g. for conhost.exe, the most important case).

alexchandel commented Jan 13, 2017

@timClicks

Your tldr was about my conclusion as well.

The main problem is that keys don't arrive to the "console stdin" of a process as a byte stream, like they do on normal OSs. Instead, when you type into a "console window" on Windows, they arrive in some Win32 input event queue of the process, along with mouse events, menu events, context-change events, and other garbage. This queue receives them in the form of key-up and key-down events, and even receives them for backspace and the arrow-keys.

When you call getc, fread, std::getline, ReadFile, ReadConsole, or anything else that should read characters from stdin, those call into a Win32 function that processes the input queue, which includes handling arrow-keys, backspaces, command history, and line discipline, and which eventually returns a character or line (depending on what you asked for) or blocks until it can.

Note that cmd.exe itself doesn't handle any key input. Note that there is no Win32 function that will perform this input event processing but NOT block if no character/line is available. Note that not even conhost.exe, the process that draws the "console" window in the first place (cmd.exe is just a batchfile language interpreter really), can be told to parse keyboard events and send them as a byte stream.

This horrible, atrocious design is why cmd.exe cannot ever have command history, why lines don't appear when you type them unless the process is blocking reading, and why you cannot do IOCP on "console stdin". You need a thread to process the events & do line discipline.

What you can do with WaitForMultipleObjects is poll the input queue, and when events are available, filter out the keyboard events and use those as character inputs in lieu of getc. Since you're just taking the key-ups (or key-downs) directly, that means character-echoing, arrow-keys, and backspace won't work, unless you explicitly implement your own input buffer and echo it back to the console. So it's a nasty hack, and probably useful only for console/terminal games like nethack or dwarf fortress, which obviously don't need line discipline, line history, arrow-keys, key-echoing, or anything else.

In cases where you can't poll, or where filtering key events is a pain or too costly to do on your thread, creating another thread that reads stdin blockingly and posts it in on a named pipe/tcp port/IOCP-compatible thing seems like the only way.

The Twisted solution in your blog post doesn't work in the general case, since you need to change your parent/input source as well, which is generally impossible (e.g. for conhost.exe, the most important case).

@alexchandel

This comment has been minimized.

Show comment
Hide comment
@alexchandel

alexchandel Jan 13, 2017

With regard to mio/tokio, the best approach is probably for mio to create a "stdin thread" on Windows if the input is a console input handle or an anonymous pipe, and feed that input into an IOCP, since mio already uses IOCP internally on Windows.

The issue is that environments with line discipline (e.g. the default state of a terminal emulator, and I believe of Win32 when reading a console handle) correspond to calling readline or ReadConsole on a Win32 console handle, whereas environments without line discipline (e.g. terminal emulators that have been configured to provide immediate input, an anonymous pipes, a named pipe, etc) correspond to calling ReadFile or maybe getc on a Win32 console handle (theoretically one could poll WaitForMultipleObjects here, but mio doesn't poll and polling is ugly). I believe the "console mode" of ReadFile is to read until a carriage return on Windows.

With regard to mio/tokio, the best approach is probably for mio to create a "stdin thread" on Windows if the input is a console input handle or an anonymous pipe, and feed that input into an IOCP, since mio already uses IOCP internally on Windows.

The issue is that environments with line discipline (e.g. the default state of a terminal emulator, and I believe of Win32 when reading a console handle) correspond to calling readline or ReadConsole on a Win32 console handle, whereas environments without line discipline (e.g. terminal emulators that have been configured to provide immediate input, an anonymous pipes, a named pipe, etc) correspond to calling ReadFile or maybe getc on a Win32 console handle (theoretically one could poll WaitForMultipleObjects here, but mio doesn't poll and polling is ugly). I believe the "console mode" of ReadFile is to read until a carriage return on Windows.

@carllerche

This comment has been minimized.

Show comment
Hide comment
@carllerche

carllerche Nov 7, 2017

Owner

Thanks all! The answer is going to be "out of scope for mio".

Mio provides all the necessary hooks to add stdin support out of crate. This lets others make decisions that work for them and Mio doesn't have to take a stance.

Owner

carllerche commented Nov 7, 2017

Thanks all! The answer is going to be "out of scope for mio".

Mio provides all the necessary hooks to add stdin support out of crate. This lets others make decisions that work for them and Mio doesn't have to take a stance.

@carllerche carllerche closed this Nov 7, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment