CommandLineRunner/csi should emit errors & diagnostics to STDERR, not STDOUT

[atifaziz]

Version Used: 1.1.0.51109

Steps to Reproduce:

See #9791

Open a Command Prompt on 64-bit Windows and enter and execute the following line on the prompt:

echo throw new Exception("BOOM!");> test.csx && "%ProgramFiles(x86)%\MSBuild\14.0\bin\csi.exe" test.csx 2>log.txt || echo NOK

Expected Behavior:

That only NOK should be displayed on STDOUT and the exception text is written to log.txt since STDERR has been redirected to that file.

Actual Behavior:

At the end of execution log.txt is empty, and both NOK and the exception text are written to STDOUT:

System.AggregateException: One or more errors occurred. ---> System.Exception: BOOM!
   at Submission#0.<<Initialize>>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.CodeAnalysis.Scripting.ScriptExecutionState.<RunSubmissionsAsync>d__9`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.CodeAnalysis.Scripting.Script`1.<RunSubmissionsAsync>d__19.MoveNext()
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at Microsoft.CodeAnalysis.Scripting.Hosting.CommandLineRunner.RunScript(ScriptOptions options, String code, ErrorLogger errorLogger, CancellationToken cancellationToken)
   at Microsoft.CodeAnalysis.Scripting.Hosting.CommandLineRunner.RunInteractiveCore(ErrorLogger errorLogger)
   at Microsoft.CodeAnalysis.Scripting.Hosting.CommandLineRunner.RunInteractive()
   at Microsoft.CodeAnalysis.CSharp.Scripting.Hosting.Csi.Main(String[] args)
---> (Inner Exception #0) System.Exception: BOOM!
   at Submission#0.<<Initialize>>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.CodeAnalysis.Scripting.ScriptExecutionState.<RunSubmissionsAsync>d__9`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.CodeAnalysis.Scripting.Script`1.<RunSubmissionsAsync>d__19.MoveNext()<---

NOK

For more background, see #9791 from which this issue was created on request of @tmat.

The fix in PR #9791 only addresses part of the issue.

[atifaziz]

Then of course tests need to be updated.

@tmat TestConsoleIO adds color markers in the output that are then tested in CommandLineRunnerTests as part of the output (e.g. see line 183). I was thinking of adding «StdErr» and «StdOut» markers in similar vein. Would that be a reasonable approach?

[tmat]

Do we need to change the tests at all? We can leave the existing tests using the same stream for both Error and Out. That way we are testing the common user experience on the console.

Then we can have just one or two tests that separate the streams and verify content of each stream separately.