Start-Process should support redirecting to variable names or stream objects

[Jaykul]

Because there are many executables (mostly of linux origin) which use stderr to write status messages (in addition to errors), and rely purely on exit code to indicate whether there actually is an error or not ....

As a PowerShell scripter, I need to be able to run an executable and both stream the output for sake of status updates and capture the (stderr) output so I can report on it properly if there is an actual error.

Start-Process doesn't support this -- it only captures to file, and if it does, it doesn't output. Additionally, since output stream redirection is limited to redirecting to stdout or file, I cannot redirect error to verbose on CLI apps that consistently behave this way...

My current company has contrived more than one complicated wrapper around ProcessStartInfo and Diagnostics.Process to handle the output events and so on. In fact, it seems that every company I work with above a certain level of PowerShell maturity has their own custom Invoke-Process or other replacement for Start-Process to try and handle this.

Start-Process should allow capturing and also outputting

Start-Process should allow capturing (and reading input from) variables instead of just files

[SteveL-MSFT]

Personally, I think we should not enhance start-process and instead provide ways to solve interacting with processes natively. It seems that if we make the Variable: provider work like a file, this should solve some of your problems?

somecommand 2> variable:stderr 1> variable:stdout

Tee-Object doesn't work with redirection, so need to think how to redirect while still outputting...

We should also allow redirecting one stream to another stream beyond stdout like:

somecommand 2>&3

[powercode]

@SteveL-MSFT For long running commands, this would make it look like powershell has hung.

I think we need a layered approach where the lowest level converts output and errors to a tagged union of output * string and error * string. On top of that it is quite easy to layer redirection to different streams, or to capture in variables.

It also opens a path to handling more complex commands where there is mixed progress and verbose output in the different streams, and makes it quite easy to make a parser that interprets the output and direct it do the appropriate powershell stream.

[powercode]

public class ProcessOutput {
   private string _value;
   public ProcessOutputKind Kind {get;}
   public bool IsOutput => Kind == ProcessOutputKind.Output;
   public string Error => Kind == ProcessOutputKind.Error ? _value : throw new InvalidOperationException();
   public string Output => Kind == ProcessOutputKind.Output ? _value : throw new InvalidOperationException();
}

class ProcessEx {
   public IEnumerable<ProcessOutput> Run(..., CancelationToken cancel){
         ...
         foreach(processOutput item in someConcurrentQueueWithOutputAndErrors.Enumerate(cancel)) {
              yield return processOutput;
         }
   }

   private void OnError(...){
         someConcurrentQueueWithOutputAndErrors.Add(new ProcessOutput(ProcessOutputKind.Error, theErrorString);
   }
   private void OnOutput(...){
         someConcurrentQueueWithOutputAndErrors.Add(new ProcessOutput(ProcessOutputKind.Output, theErrorString);
   }
}

Something similar would make it easy to use from the PowerShell side.

foreach($po in ProcessEx.Run(...)){
   if ($po.IsOutput){
         # parse output and split into Output and Progress
   }
   else {
        # parse error and split into for example Verbose, Warning and Error
   }
}

[mklement0]

Start-Process is of limited usefulness particularly on Unix, so I think @SteveL-MSFT's suggestion of using direct invocation with enhanced redirections is the more promising idea to pursue, especially if you're interested in showing ongoing output as well as capturing, which only makes sense with synchronous execution (at least on Unix, where running in a new terminal window is not an option).

If we had the ability to capture in a variable while optionally also passing output through, then I think no lower-level approach is needed.

I had previously suggested syntax >&targetVarName, but I like the idea of using namespace notation (> variable:targetVarName)

However, additional syntax is needed for:

• optional pass-through-as-well behavior, as suggested by @SteveL-MSFT (>|?)

• optional append-to-preexisting-variable-contents behavior (a + prefix might do, as with -OutVariable for instance).

These two behaviors would bring the redirection operators on par with the -OutVariable and -ErrorVariable common parameters.

[mklement0]

P.S: Asking for the ability to capture stderr lines in a variable is the subject of #4332

[Jaykul]

It's too bad this relatively simple request got set aside for a more elegant but unachievable better idea ...
Given it's been 6 years, do you think we could just do the original request?

[SteveL-MSFT]

I've queued this up for WG to discuss

[SteveL-MSFT]

@PowerShell/wg-powershell-cmdlets discussed this, since stderr gets wrapped as an ErrorRecord, we propose a small change in Tee-Object to have a new -ErrorRecordVariable that stores streamed input that is of type [System.Management.Automation.ErrorRecord] and store it to that variable. Existing behavior to write to the host would continue. So an example usage would look like:

ls *foo* README.txt 2>&1 | Tee-Object -Variable stdout_variable -ErrorRecordVariable stderr_variable -ErrorVariable teeObjectErrors

We would enhance Tee-Object docs to show how to use this.