[Windows Terminal] Console.KeyAvailable causes the next Unicode character input 'EN DASH' to be skipped in console

[daxian-dbw]

Description

When pasting text with Unicode character in it to console on Windows (conhost) using right click, the Unicode character, such as the 'EN DASH' character in the following example

New-StoragePool –FriendlyName

will somehow be skipped and missing from the Console.ReadKey(true) calls. More investigation shows the call to Console.KeyAvailable could be the culprit -- when Console.KeyAvailable is not in use, Console.ReadKey(true) is able to receive the 'EN DASH' character just fine.

Repro code

using System;

namespace console
{
    public class Program
    {
        static void Main(string[] args)
        {
            do {
                if (Console.KeyAvailable)
                {
                    var key = Console.ReadKey(true);
                    Print(key);
                }
                else
                {
                    var key = Console.ReadKey(true);
                    Print(key);
                }
            }
            while(true);
        }

        // Print the decimal value of `KeyChar`, the `ConsoleKey` and `Modifiers` values.
        private static void Print(ConsoleKeyInfo key)
        {
            Console.Write('[');
            Console.Write((int)key.KeyChar);
            Console.Write($" {key.Key} {key.Modifiers}");
            Console.Write(']');
        }
    }
}

Copy the text "l –F", and then right click to paste the text, you will see that the 'EN DASH' character is missing:

Then, copy the text "–F" with the 'EN DASH' character being the first character, so Console.KeyAvailable will not run before accepting that character, then Console.Readkey(true) is able to receive that Unicode character ():

Additional information

Please note that, the same issue happens in .NET 5 preview as well.

If I change the Main method above to be the following, then Console.Readkey can always receive the Unicode character.
But I have to use Console.KeyAvailable in my project together with Console.ReadKey, and hence cannot receive the Unicode when user is pasting by right click in console.

        static void Main(string[] args)
        {
            do
            {
                var key = Console.ReadKey(true);
                Print(key);
            }
            while(true);
        }

[ghost]

Tagging subscribers to this area: @eiriktsarpalis
Notify danmosemsft if you want to be subscribed.

[eiriktsarpalis]

Hi @daxian-dbw

I tried your reproduction locally, but couldn't reproduce the issue. This is what I get on the powershell terminal:

And this is what I see using powershell core on Windows terminal:

This is using .NET Core 3.1.302, but the behaviour is similar in .NET 5 preview.

[daxian-dbw]

@eiriktsarpalis Thanks for looking into this!
Please try out the repro from cmd.exe. I don't know why it works in PowerShell, but it repros consistently when running it from cmd.exe. Below is a screen gif of it when I paste "l –F" with right clicking. The input/output encoding's are the defaults:

OutputEncoding: utf-8
InputEncoding: Codepage - 437

[eiriktsarpalis]

I'm still not reproducing :/

@daxian-dbw are you able to reproduce on a second machine?
@adamsitnik when you get back, would you be able to try and reproduce locally?

[daxian-dbw]

@eiriktsarpalis I found the OutputEncoding may plays a role here. I added Console.OutputEncoding = Encoding.UTF8 below to the original repro code, and now I can reproduce the issue in both cmd.exe and powershell core.

using System;
using System.Text;

namespace console
{
    public class Program
    {
        static void Main(string[] args)
        {
            Console.OutputEncoding = Encoding.UTF8;
            Console.WriteLine("OutputEncoding: {0}", Console.OutputEncoding.WebName);
            Console.WriteLine("InputEncoding: {0}", Console.InputEncoding.WebName);

            do {
                if (Console.KeyAvailable)
                {
                    var key = Console.ReadKey(true);
                    Print(key);
                }
                else
                {
                    var key = Console.ReadKey(true);
                    Print(key);
                }
            }
            while(true);
        }

        private static void Print(ConsoleKeyInfo key)
        {
            Console.Write('[');
            Console.Write((int)key.KeyChar);
            Console.Write($" {key.Key} {key.Modifiers}");
            Console.Write(']');
        }
    }
}

Run from cmd.exe

paste "l –F" by right clicking:

paste "–F" with the 'EN DASH' character being the first character by right clicking:

Run from powershell core

paste "l –F" by right clicking:

paste "–F" with the 'EN DASH' character being the first character by right clicking:

[daxian-dbw]

@eiriktsarpalis Additional data point: I found this only reproduce in Windows Terminal. If I do the same in the conhost, then the 'EN DASH' character can always be captured as [45 OemMinus 0], no matter launching console.exe from cmd.exe, or PowerShell core, or Windows PowerShell. Also, I tried on a second machine and it's the same behavior.

So a side question: why the 'EN DASH' character results in different ConsoleKeyInfo in Windows Terminal vs. conhost?

[eiriktsarpalis]

I'm reproducing the same behaviour. Basically the character is omitted on Windows terminal only (reproducing with either cmd, PowerShell or Git Bash), but not on any other terminal emulators that I tested. It namely works for Visual Studio Console, Git bash, Powershell classic and ConEmu. I'll update the issue description accordingly.

[eiriktsarpalis]

Also, the issue does not reproduce when using Windows terminal on WSL.

[daxian-dbw]

It would be very helpful if @DHowett can share some insight, especially on this question:

So a side question: why the 'EN DASH' character results in different ConsoleKeyInfo in Windows Terminal vs. conhost?

Thanks in advance @DHowett!

[msftrncs]

This looks like a potential problem area.

runtime/src/libraries/System.Console/src/System/ConsolePal.Windows.cs

Lines 285 to 296 in d1c0fa8

	// Skip non key-down && mod key events.
	if (!IsKeyDownEvent(ir) || IsModKey(ir))
	{
	r = Interop.Kernel32.ReadConsoleInput(InputHandle, out ir, 1, out numEventsRead);
	if (!r)
	throw Win32Marshal.GetExceptionForWin32Error(Marshal.GetLastWin32Error());
	}
	else
	{
	return true;
	}

Could it be possible that these characters are not triggering IsKeyDownEvent and this loop is eating them?

[shibayan]

I've noticed the same problem. Similarly, if you call Console.KeyAvailable, certain characters will be skipped. This occurred when I pasted a string containing 'KATAKANA MIDDLE DOT' into the console as shown below.

A・B

Oddly enough, it didn't happen in the Windows Terminal, but seemed to happen in the command prompt.

Repro code

using System;

namespace ConsoleApp28
{
    class Program
    {
        static void Main(string[] args)
        {
            do
            {
                var keyInfo = Console.ReadKey(true);

                Console.WriteLine($"KeyChar = {keyInfo.KeyChar}, Key = {keyInfo.Key}, Modifiers = {keyInfo.Modifiers}");

            } while (Console.KeyAvailable/*true*/);
        }
    }
}

Use Console.KeyAvailable

paste "A・B" by right clicking:

Not use

paste "A・B" by right clicking:

[adamsitnik]

This looks like a potential problem area.

I've marked this issue with help wanted label. The person who is willing to work on it should take one of the provided repro apps and debug the code pointed by @msftrncs in #38966 (comment) If that is the source of the problem, please send a PR with a fix.