System.InvalidOperationException with Python's `os.execvp` command

[daviewales]

Prerequisites

• Write a descriptive title.
• Make sure you are able to repro it on the latest released version
• Search the existing issues.
• Refer to the FAQ.
• Refer to Differences between Windows PowerShell 5.1 and PowerShell.

Steps to reproduce

I'm trying to use one of Python's os.exec* functions, specifically os.execvp().

On Linux, one can use this to replace the current process with a new one.
On Windows in PowerShell 7.4.6, it kind of works, but has some strange behaviour.

Minimal example

Here is a minimal Python example which attempts to replace the current process with a continuous ping command:

Linux/ZSH:

import os
os.execvp("ping", ['/usr/bin/ping', '1.1.1.1'])

Behaviour: The current Python process is completely replaced by the ping process. When you press Ctrl+C, you are returned to the shell.

Windows/PowerShell:

import os
os.execvp("ping", ['C:\\Windows\\System32\\ping.EXE', '-t', '1.1.1.1'])

Behaviour:

• Two lines of output from ping.
• Mysteriously, the PowerShell prompt is printed on the third line.
• ping keeps printing lines after this.
• If you press Ctrl+C, PowerShell prints the prompt, but ping is not stopped, and keeps printing output.
• If you rapidly repeat Ctrl+C, eventually the message will get through, and ping will stop.

More complicated (but useful) example

The real reason I'm using execvp is to create an SSH wrapper script for az ssh vm:
https://github.com/swsphn/az-ssh-wrapper

I'm using something like the following (which works perfectly in Linux!):

import os
# Insert your own Azure VM IP address
os.execvp('az', ['"C:\\Program Files\\Microsoft SDKs\\Azure\\CLI2\\wbin\\az.CMD"', 'ssh', 'vm', '--ip', '1.2.3.4'])

Behaviour: This starts out the same way as the ping example above:

• PowerShell prints a prompt (unexpected!)
• The output from SSH appears.
• We successfully see the shell prompt on the VM.
• If I type any character, the input triggers my local PowerShell history completion, which causes chaos on my screen.
• If I press the Ctrl key, it triggers the following PowerShell exception:

  An error has occurred that was not properly handled. Additional information is shown below. The PowerShell process will exit.
  Unhandled exception. System.InvalidOperationException: Cannot read keys when either application does not have a console or when console input has been redirected. Try Console.Read.
     at System.ConsolePal.ReadKey(Boolean intercept)
     at Microsoft.PowerShell.Internal.VirtualTerminal.ReadKey()
     at Microsoft.PowerShell.PSConsoleReadLine.ReadOneOrMoreKeys()
     at Microsoft.PowerShell.PSConsoleReadLine.ReadKeyThreadProc()
     at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
  --- End of stack trace from previous location ---
     at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
  

• After the above exception triggers, PowerShell is dead, but I still have a console connected to the remote shell on my VM, so I can use it as expected.
• When I exit the remote shell on the VM, I am returned to a dead PowerShell and the following message:

  [process exited with code 3762504530 (0xe0434352)]
  You can now close this terminal with Ctrl+D, or press Enter to restart.
  

Alternative example which doesn't require an Azure VM

The more complicated example above appears to reproduce with WSL as well. For example:

import os
os.execvp('wsl', ['C:\\Windows\\System32\\wsl.EXE'])

Expected behavior

I expect to be able to replace the current process with another process without PowerShell glitching or crashing.

Actual behavior

PowerShell seems to be trying to control the terminal at the same time that the replacement process is trying to control it, which causes glitches and crashes.

Error details

See above.

Environment data

Name                           Value
----                           -----
PSVersion                      7.4.6
PSEdition                      Core
GitCommitId                    7.4.6
OS                             Microsoft Windows 10.0.22631
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Visuals

Error in PowerShell on Windows

powershell-error.mp4

No error in ZSH on Linux (WSL)

linux-bash-no-error.mp4

[daviewales]

Is this related to: dotnet/runtime#59059?

[237dmitry]

Linux/pwsh, it works as expected

[kasini3000]

Hi, take a look at my PowerShell batch project: Kasini3000
1 The main control computer supports Windows and Linux.
2 Supports Winrm and SSH.Hybrid cloud and devops.
3 supports all PowerShell features, such as remote object passing when ssh-remoteing .
4 Multi process batch, multi-threaded batch, base SQLite DB
5 Other functions: sound and image alarm, multiple SSH key files, etc.

PowerShell is the best engine, and I use it to make the best car.
I hope it can be better than Python and ansible.

[rhubarb-geek-nz]

According to os — Miscellaneous operating system interfaces the replace current process with new program only works on UNIX

On Unix, the new executable is loaded into the current process, and will have the same process id as the caller.

The Windows programming model has never provided fork/exec. It was available as part of Services For Unix subsystem (SFU) and also within Windows Services for Linux (WSL)

One explanation is that on Windows you now have two processes trying to read from stdin.

PowerShell running reading stdin for input
Python then runs reading stdin input
On UNIX the exec replaces the Python program so PowerShell still waits for it to complete
On Windows the exec creates a new process and the original Python process exits. This makes PowerShell think that the Python program has ended (because it has) and resumes reading the stdin
The program launched by execvp also wants to read stdin

It would explain why ping and the prompt appear in parallel, both are running at the same time.

It is only a theory but it matches the observations

[daviewales]

The docs state:

Availability: Unix, Windows

And the function exists and runs on Windows. The documentation for both Unix and Windows states:

These functions all execute a new program, replacing the current process; they do not return.

This certainly appears to be true on both Windows and Unix. But it notes that:

On Unix, the new executable is loaded into the current process, and will have the same process id as the caller.

In contrast, it appears that on Windows, the current executable stops, and a new one starts, with a different process ID to the caller. Could PowerShell handle this situation better? Why doesn't it hand control of stdin to the newly created process?

[rhubarb-geek-nz]

Could PowerShell handle this situation better?

I don't see how. PowerShell knows nothing of Python's child process, all that it sees is that the Python program has ended. Because PowerShell is in an REPL loop it goes back to reading stdin.

Why doesn't it hand control of stdin to the newly created process?

The newly created process does read stdin, it is when PowerShell starts to read that it throws an exception. The only way to prevent this is for the Python program to wait until the child has terminated before it exits.

[rhubarb-geek-nz]

If it is any help, PowerShell also supports exec on UNIX using Switch-Process

[MatejKafka]

I believe there are two separate issues:

1. Console.ReadKey throws in unexpected circumstances dotnet/runtime#59059
2. Python is partially lying to you in an attempt to keep compatibility between platforms:
   • POSIX-compliant platforms (such as Linux and Mac) give you an exec function, which replaces the current process with a new one, but for an external observer, it still looks as the same process. When you run a command in pwsh, it spawns the executable and then waits until it exits before printing the prompt and waiting for a new command to run. With exec, that works because pwsh spawns the python process, which replaces itself with ping using exec, but it's still that same process, so pwsh keeps waiting for it.
   • Windows do not provide anything like exec through the Win32 API, for various reasons (some of them pretty good). What python does instead is to just spawn the new process and exit. When the Python process exits, pwsh sees that the spawned command exited and prompts you for another command. However, the new process spawned by Python is still running and it's reading from the console at the same time as pwsh. If both processes are reading stdin byte-by-byte, each process randomly receives some of the keystrokes. Since pwsh does not know (and can't easily know) about the spawned child process from Python, there's not much it can do in this situation. You'll see the same issue in cmd.exe and probably any other shell on Windows.

[daviewales]

So... wrapper scripts on Windows need to stay running until the child process exits? There's no other option on Windows?

[rhubarb-geek-nz]

The classic mechanism for command line wrapper scripts on Windows is batch files which generally take the form

@pwsh.exe -File "%~dpn0.ps1" %*

or

@java.exe -jar "%~dpn0.jar" %*

A good example are the wrappers for Java programs like maven or gradle on Windows which essentially look at the environment then call java.exe with the program jar file and the original arguments.

So cmd.exe waits around until the java.exe completes.

It is important to remember that Windows is not a POSIX operating system and does not do fork/exec. The fundamental API in Windows, CreateProcess effectively combines the effect of fork/exec by simply allocating a new process in memory and loading a program into it. Now compare the complexity of CreateProcess with the simplicity of fork/exec, it is because they come from completely different architectures. In the POSIX world the setting up of the current directory and redirecting file descriptors would be done in the child process between the fork and exec calls.

On Windows a true wrapper script needs to be at least one of the types enumerated by the PATHEXE variable

PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC

As mentioned, the standard way is either a bat or cmd. Running the python interpreter to run a wrapper script is not very windows-like.

[daviewales]

Thanks for the explanation. It sounds like I can't do what I want by design, so I'll close this.

[microsoft-github-policy-service]

📣 Hey @daviewales, how did we do? We would love to hear your feedback with the link below! 🗣️

🔗 https://aka.ms/PSRepoFeedback

Microsoft Forms