Support terminal use on Windows to be optional with a new default attach for the new --console= option.

[shenjackyuanjie]

#2668

https://github.com/shenjackyuanjie/wrapper-rs
in this project

1. I use AttachConsole to attach a subsystem=windows program to it's father console (if exist) and use some other windows api to check if it's started from console or explorer
   Nuitka can ues this to improve --onefile building, and create a program can use STDOUT/STDERR/STDIN when started from a console, and not creating a cmd window when staring from explorer

2. I use process create flag to create a none conhost process when needed, and the host program also dose not contain a conhost, so you can get a clean program

[kayhayen]

The AttachConsole use is a major finding, and we need to use that in the C code of Nuitka. I am very happy about that, as it means GUI programs will be able to properly be debugged without forcing the output elsewhere. This applies to standalone and onefile, their interaction might make it a bit more difficult, but I believe onefile might always be running when the Python program checks its parent, if there is an issue with that, it would be bad. With no cleanup needed, onefile might quit too soon, that would have to be looked at.

The second point, I don't get what flag you are talking about and why it would be needed.

[kayhayen]

I want to make this work for standalone too, and not just onefile, where there is of course the extra step to launch a new process, such that the Python DLL never sees it, occurs. I was not immediately successful with it though.

[kayhayen]

So, I made this to work, the change has interesting effects in case of an assertion, etc. this now raises GUI dialogs potentially, which is not a bad thing though, since they offer to debug inside of Visual Studio, which can make things a bit easier at times maybe.

The code attaches to the console, and re-opens CONOUT$ and CONIN$ for stdout/stderr/stdin if they were not re-directed, I need however to check if the later works beyond stdout right now. For the command line interface this will mean big changes, with --console=force|attach|disable where old --enable-console is force and disable is something I actually need to implement still, and the new default is attach which uses console if available.

[kayhayen]

So, implement the --disable-console (--console=disable) and --enable-console (--console=force) as to just not attach to the console, and forcing it simply doesn't use the subsystem on Windows, that is enough to deal with it. That should make it perfect.

Not sure yet, how this relates to onefile. Currently accelerated and standalone are working nicely. I guess, the onefile should maybe always attach itself, so it can forward it, I shall try it out.

[kayhayen]

So, I found that os.system and subprocess in general work very badly in this context. I noticed no outputs are given, and it seems the file handles that Nuitka creates are invalid for CONIN$ and CONOUT$ for most anything.

When I noticed that the subprocess call didn't output anything, I removed the check to see if all of stdin, stdout, and stderr are None, which they were. I noticed that if one of them is given, the subprocess will crash because it is unable to, e.g., duplicate stdin.

When I looked at os.system, I saw that it calls a function _wsystem and not much else, so Win32 is not finding the standard handles as something it wants to pass. In the onefile bootstrap, I noticed that I had to make the handles inherited by the child process to pass on the attached console handles.

So I am either missing something fundamental or this might well be very unusable. I can, of course, patch up subprocess somehow to use pipes that throw away or get output delayed, but it seems this might not be generally doable. I am struggling to find out what outside of attaching is necessary to make the console handles inherited more automatically, which probably makes them possible to duplicate.

It's looking grim though.