Improve support for native console programs

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Start the Win32 Python (not the Cygwin port) inside mintty

What is the expected output? What do you see instead?
The interactive Python prompt should be displayed. Instead, nothing happens
and the shell remains suspended until I press Ctrl-C.

What version of the product are you using? On what operating system?
0.3.5-1

Please provide any additional information below.
Win32 Python is works fine under Console2
(http://sourceforge.net/projects/console/), which is a different Windows
terminal emulator.

Original issue reported on code.google.com by sami.kyo...@gmail.com on 17 Feb 2009 at 3:07

[GoogleCodeExporter]

This is due to MinTTY being based on Cygwin pty's, which do not play well with
Windows console apps. I'm not sure I entirely understand the issue, but 
basically
it's because the pty emulation is based on pipes, which means that a Win32
application running in MinTTY sees a pipe rather than a console as its input, 
so it
behaves differently.

You'll see the same issue in rxvt and xterm, and you can find plenty of posts 
on this
on the Cygwin mailing list.

Console2 takes a different approach: it doesn't do its own terminal emulation;
instead it runs a hidden Windows console window, grabs the screen contents from 
that,
and puts it into a nicer window. That means, however, that you're still getting 
many
of the limitations of that such as low speed and the somewhat non-standard 
terminal
emulation of the console-based Cygwin terminal.

It's not clear that it is possible to fix this. The Cygwin guys would probably 
have
done it if there was.

Original comment by andy.koppe on 17 Feb 2009 at 5:47

• Changed state: Accepted
• Added labels: Difficulty-Insane, Priority-None

[GoogleCodeExporter]

Thanks for your explanation. It does seem like it would be non-trivial to 
support
apps such as these inside minTTY.

I wonder would it be possible to somehow identify these problematic 
applications when
they are launched and just run them in a separate Windows console window? You
probably would not get proper IO redirection this way, but at least interactive
applications would remain usable.

Original comment by sami.kyo...@gmail.com on 19 Feb 2009 at 12:09

[GoogleCodeExporter]

That might indeed be possible, by inspecting the executable using objdump or 
some
such, but that would be a job for the shell, not the terminal.

Original comment by andy.koppe on 3 Mar 2009 at 2:24

• Changed title: Interactive non-Cygwin programs don't work

[GoogleCodeExporter]

I'm wondering, how hard would it be to make MinTTY work with MSYS instead of 
Cygwin?
Would this amount to the same work as making MinTTY work with any Windows 
console
program, or does MSYS also have ptys, to porting from Cygwin to MSYS would be a
doable task?

Original comment by sschuberth on 28 Mar 2009 at 7:29

[GoogleCodeExporter]

I'm not sure. Apparently MSYS is a cut-down fork of Cygwin 1.3.3, but I don't 
know
whether they removed ptys. I suspect not though, because there is an rxvt for 
MSYS. I
did have a brief attempt at compiling mintty on msys, but didn't get anywhere 
because
for some reason the 'winres' resource compiler would just hang on my system. 
Let me
know how you get on if you're having a go at it.

Original comment by andy.koppe on 28 Mar 2009 at 7:44

[GoogleCodeExporter]

Have you used MSYS / MinGW from http://www.mingw.org/, or did you install the
gcc-mingw packages in Cygwin? Maybe the latter works better for you, and using
"-mno-cygwin" you'll be able to compile programs under Cygwin that do not 
depend on
"cygwin1.dll".

Original comment by sschuberth on 8 Apr 2009 at 10:24

[GoogleCodeExporter]

I'd installed the MSYS/MingW. Compiling with -mno-cygwin doesn't work, because 
then
all sorts of POSIX facilities are missing:

child.h:4:25: sys/termios.h: No such file or directory
child.c:10:17: pwd.h: No such file or directory
child.c:11:17: pty.h: No such file or directory
child.c:13:23: sys/ioctl.h: No such file or directory
child.c:14:22: sys/wait.h: No such file or directory
child.c:15:18: argz.h: No such file or directory
child.c:16:18: utmp.h: No such file or directory
child.c:18:21: pthread.h: No such file or directory

To have any chance of this working, MinTTY would needs to be built for MSYS. The
'About MSYS' page at http://mingw.org/node/18 has this to say on the topic:

"To build an application for MSYS (as opposed to using MSYS), users will need to
install msysDVLPR. It contains the headers and libraries to for MSYS along with 
an
old version of GCC and Binutils. Resulting programs will only run under MSYS.
Msysdvlpr should never be treated as a targeted platform. It should also be 
noted
that msysdvlpr is unlikely to be updated in the near future."

Original comment by andy.koppe on 26 Apr 2009 at 12:28

[GoogleCodeExporter]

My apologies if this info is obvious, or stupid, or whatever.  I'm not a Win32 
programmer so it's hard for me to judge.  I'm just hoping that maybe the 
difficulty 
of the bug can reduced a little from "insane".  If this issue could be 
resolved, it 
would certainly distinguish mintty from other console apps under cygwin.

Regarding the approach of intercepting/overriding the console API calls:
There is an open source Win32 API Monitoring application available at:
    http://jacquelin.potier.free.fr/winapioverride32/
This can be used to determine the minimum calls that would have to be 
overridden in 
order to get a specific application (eg. the Win32 python console) working.
The site contains quite extensive documentation (in addition to the source), 
including a description of how they implement the API interception:
    http://jacquelin.potier.free.fr/winapioverride32/doc/dev.htm

Though that technique might not be the best for mintty, as it sets up the 
override 
for calls from a specific designated application only.

I read that you should be able to override the console functions merely by 
placing a 
replacement kernel32.dll in the dll search path, ahead of the system dll.  
Apparently 
a registry key "HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session 
Manager\KnownDLLs" indicates DLL files that cannot be overridden in this way.  
kernel32.dll is listed there and must be removed for this technique to work.  I 
personally tried this on WinXP with kernel32.dll, but I wasn't able to get any 
proof 
my substitute dll was called.  No doubt I goofed somewhere.
The KnownDlls registry key is discussed on these pages:
    http://blogs.msdn.com/larryosterman/archive/2004/07/19/187752.aspx
    http://support.microsoft.com/kb/q164501/

I gather that the executable PATH is also used to search for DLLs on windows.  
So I 
thought mintty could add the path to a substitute kernel32.dll to the PATH 
environment var, and all child applications (shells, console apps) would 
inherit it 
and call the console functions in the substitute kernel32.dll instead of the 
real 
one.  Then the substitute console functions could either use standard file 
descriptors (0,1,2) or use additional environment vars to identify the correct 
information to use to communicate with mintty.  The replacement console 
functions 
could translate console API requests into terminal control sequences on the 
communication channels.

The initial goal would only be to provide just enough functionality that 
console 
applications realise they're talking to a console window of some kind and set 
their 
buffering accordingly.  But better support for the windows console API could be 
added 
over time to fix less common problems.

I guess non-console functions would need to be setup to forward the call to the 
real 
kernel32.dll.

Original comment by barry.do...@gmail.com on 1 May 2009 at 8:08

[GoogleCodeExporter]

Thank you very much Barry for researching this and providing those links. 
Interesting
stuff. This is roughly what I had in mind though when I rated this as "Insane". 
;)

Messing with KnownDLLs and overriding kernel32.dll is rather risky and
security-critical stuff. That would need a great big health warning and 
multiple "are
you sure?" checks. And it would likely turn out to be quite fragile due to
kernel32.dll changes between Windows releases.

Another possible point of attack I've seen mentioned is the client-server 
protocol
that apparently exists between console apps and 'winsrv.dll', which implements 
the
console window, but I haven't found any details on it. (In Windows 7 the server 
seems
to have been spun out into 'conhost.exe'.)

Two more approaches are explained here, along with their limitations:
http://homepages.tesco.net/~J.deBoynePollard/FGA/capture-console-win32.html

Finally, getting hold of the console input/output, difficult as it is, isn't
necessarily all that's needed. While it should be fine for simple interactive
programs such as the Python interpreter, there are likely differences
incompatibilities in terminal emulation which would cause fancier programs to 
fail. I
guess that's not worth worrying about at this point though.

Original comment by andy.koppe on 1 May 2009 at 7:11

[GoogleCodeExporter]

I also didn't find any significant info on winsrv.dll (for now, anyway).

It's clear that even if it could eventually be made reliable and stable, an 
attempt 
at fixing this by overriding console API functions would initially be quite 
dodgy.  
It might work with some applications but fail with others, and might break when 
certain OS upgrades are installed or be incompatible with certain versions of 
windows.  Not to mention the scary installation procedure (all the "are you 
sure?" 
stuff)...

Since mintty is a simple, fast, reliable tool (in my mind anyway), dodginess 
just 
won't fit well with it.
I'd like to suggest the possibility of a separate tool to provide the interface 
between programs using the windows console API, and a cygwin pty-based console. 
 It 
wouldn't necessarily have to be limited to work with mintty.
I'm too occupied with other projects, and probably too lacking in relevant 
skills, to 
consider taking this on myself at this time.  But I'll certainly grant kudos to 
anyone who can produce a working implementation which allows a few basic 
console API 
programs (like the Win32 python console) to work with mintty.

Original comment by barry.do...@gmail.com on 7 May 2009 at 3:30

[GoogleCodeExporter]

Thanks Barry. I agree with the idea of trying this as a separate tool, and I 
actually
suggested the same thing in a separate discussion on the Cygwin mailing list:

http://article.gmane.org/gmane.os.cygwin/106409

There, Barry Kelly outlined how the console API could be overridden using debug
functions:

> You could intercept the calls to the console APIs by finding out where
> the imports resolve to using GetProcAddress etc. (i.e. somewhere inside
> where kernel32.dll is mapped to in the process), then adjusting the page
> permissions (VirtualProtect etc.) and rewriting the entrypoint code
> (save original 5 bytes of code, insert JMP <your-stub-here>, and at
> <your-stub-here> save the parameters, update your stuff, execute the
> missing code (the code you skipped - may need adjustment owing to
> code-relative addressing modes in e.g. Jcc instructions), then JMP back.
>
> That's all assuming you have access to the innards of the process, most
> likely with debugging APIs like CreateRemoteThread, DebugActiveProcess /
> CreateProcess w/ debug, and WaitForDebugEvent / ContinueDebugEvent
> debugging loop.
>
> Doing all this could be fun, but I reckon it would amount to a stack of
> hacks. It also requires overbearing privileges.

I also have neither the low-level Windows skills nor spare hacking time to do 
this.
So, as the two Barrys were saying, there's a fun and worthwhile project here for
someone to have a go at.

Original comment by andy.koppe on 8 May 2009 at 4:47

[GoogleCodeExporter]

There's actually a very simple workaround for the original problem with Win32 
Python:
invoke it with option -i.

This explicitly tells it to run in interactive mode. The reason it doesn't work
otherwise is that it checks whether stdin is a console. With mintty and other
pty-based terminals the answer is no, because Cygwin uses pipes to emulate 
ptys, so
Python enters non-interactive mode.

The things one learns off Twitter ...

Original comment by andy.koppe on 16 May 2009 at 6:03

[GoogleCodeExporter]

Unfortunately, not all Windows programs have a workaround like -i to force them 
to act interactively.

My two cents:

The real solution is to use Cygwin programs under Cygwin terminals (mintty, 
rxvt, xterm, PuTTYcyg) and use Windows programs under Windows consoles (the 
built-in 
console, console.sf.net, etc).  So, if you want to use an interactive Python 
interface on Cygwin, use the Cygwin Python instead.

It would be great to have the "one true terminal/console for Windows", 
simultaneously presenting a Windows console device and a Cygwin pty depending 
on the 
"foreground process".  I'm betting that this is not possible to build, or if it 
is, that not possible to make it behave correctly under all circumstances.  
That said, I 
promise not to stop anyone from trying!  Read the console.sf.net source to see 
how they implement a Windows console device, and then read 
http://www.linusakesson.net/programming/tty/index.php and Cygwin's 
implementation of pty's on top of Windows IO primitives.

Original comment by medgar123 on 17 Sep 2009 at 2:27

[GoogleCodeExporter]

Thanks Mark, good points.

The fundamental problem here is that Windows doesn't have  something like a 
"pseudo
console" interface that would allow any program to impersonate a console. (Mind 
you,
it would still be a lot of work to actually implement that interface. For 
example,
Windows does command line history in the console rather than the shell.)

Switching between showing the terminal's own screen buffer and that of a hidden
console depending on foreground process is an interesting idea. However, 
there'd need
to be a way to ensure that Cygwin programs are wired up to the pty while Windows
programs are connected to the console.

Hmmm, I think this could actually be done as a standalone program, and without
requiring the black magic discussed further up this thread. Like Console2, it 
would
create a hidden console and grab the screen contents from that, but instead of
painting it directly into a Win32 window, it would paint it using terminal 
control
sequences. Users would need to invoke it as a wrapper around the Windows 
program, e.g.:

$ console edit foo.txt

I think in principle this approach should work with any of the pty-based Cygwin
terminals. The main challenge would be in keeping the overhead of polling and
painting the console buffer to an acceptable level.

Any volunteers? :)

Original comment by andy.koppe on 19 Sep 2009 at 3:54

[GoogleCodeExporter]

My knowledge of this issue is limited, but maybe the following idea can help. I 
believe the problem is that the _isatty() function of MSVCRT (and possibly 
whatever 
is meant to replace it in more recent versions) only returns true when 
stdin/out are 
connected to "character devices". The only "character device" besides "real" 
console 
windows I could find were serial ports. A while ago I experimented with using 
emulated loop-back COM ports to this end (my goal was to port some *nix 
terminal 
emulator to Windows for use with MSYS), for instance using com0com (http://
com0com.sourceforge.net/). As I recall, _isatty() returned true when 
redirecting 
stdin/out to these devices. My theory was that then the other end of the 
loop-back 
could be connected to a terminal emulator, but I lost motivation before getting 
this 
far. This is essentially the same as using pseudo terminals on *nix, so I don't 
think it's too ugly a solution (there is no limit to the number of emulated 
devices 
and they can be named and accessed through UNC paths). I haven't tested this 
thoroughly, and I can't say if it solves the problem in every case (or if it 
solves 
the problem at all, since I only used dummy programs to check the output of 
_isatty
()), but the idea might be worth investigating.

Original comment by om...@elowar.com on 4 Oct 2009 at 1:10

[GoogleCodeExporter]

Interesting stuff, thanks! This would address the problem with any programs 
similar
to python but without something like the -i option for overriding the _isatty()
detection. It wouldn't, however, help with programs that use the low-level 
console
API. Also, shame that a kernel-mode driver is needed to implement it, 
especially with
the driver signature requirements on newer Windows.

Original comment by andy.koppe on 4 Oct 2009 at 5:26

[GoogleCodeExporter]

Original comment by andy.koppe on 3 Nov 2009 at 8:37

• Changed title: Interactive non-Cygwin programs often don't work

[GoogleCodeExporter]

Issue 146 has been merged into this issue.

Original comment by andy.koppe on 6 Nov 2009 at 7:18

[GoogleCodeExporter]

Issue 153 has been merged into this issue.

Original comment by andy.koppe on 13 Nov 2009 at 9:14

[GoogleCodeExporter]

The console WinEvents API allows keeping track of changes to a console's screen
buffer without polling: 
http://msdn.microsoft.com/en-us/library/ms682102%28VS.85%29.aspx

Original comment by andy.koppe on 15 Nov 2009 at 8:19

[GoogleCodeExporter]

I've put together a little utility called 'conin' that allows programs 
depending on
stdin being a console to be used in mintty. More details at
http://groups.google.com/group/mintty-discuss/browse_thread/thread/1f9cf480117b8
a0b

Original comment by andy.koppe on 12 Dec 2009 at 2:38

[GoogleCodeExporter]

Issue 158 has been merged into this issue.

Original comment by andy.koppe on 20 Jan 2010 at 7:26

[GoogleCodeExporter]

An updated version of 'conin' with added locale/charset support can be found at
http://mintty.googlecode.com/files/conin-0.0.2.zip.

Original comment by andy.koppe on 20 Jan 2010 at 7:47

[GoogleCodeExporter]

same trick of -i for python can be used for mingw bash
just put mintty + cygwin dll in mingw bin directory
then launch it
with mintty bash --login -i

Original comment by sher...@gmail.com on 8 Mar 2010 at 3:36

[GoogleCodeExporter]

peraphs cursors keys and tabs are working in a strange way

Original comment by sher...@gmail.com on 8 Mar 2010 at 3:41

[GoogleCodeExporter]

[deleted comment]

[be5invis]

Maybe you can take a look at ConPTY now. That's the missing part of Windows' console system recently filled.

[mintty]

There is some info @ https://blogs.msdn.microsoft.com/commandline/2018/08/02/windows-command-line-introducing-the-windows-pseudo-console-conpty/
but as usual with Windows APIs, this stuff looks more complex than expected (what's that fuss with Threads before actually involving the ConPTY API in the example?).
Anyway, someone ambitious about the issue might craft a solution within a few days, however, the solution should go into cygwin, not mintty.

[Biswa96]

Cygwin now has Windows ConPTY support. See this commit. Released in Cygwin 3.1.0-0.3 test build.

[eximius313]

Is there any hope for solving this issue?

[mintty]

Yes, you can expect it to be solved with the next cygwin release which supports the new Windows ConPTY API.

[Biswa96]

@eximius313 You can also download and use the cygwin test builds now by selecting test radio button in cygwin setup. But as they are test builds random crashes may occur.

[eximius313]

I'm using it via GitBash, so it's not that easy ;)

[mintty]

Cygwin 3.1.0 has been released with ConPTY support.