Console.ReadKey returns incomplete information for some key sequences on Linux

[lzybkr]

AB#1115513
Using PowerShell 6.0.0-alpha.10 on Ubuntu 14.04, I run

PS> [Console]::ReadKey()

and type Ctrl-@. I get back

KeyChar Key Modifiers
------- --- ---------
        0   0

I see somewhat similar results with other key presses like Ctrl-! or Ctrl-%. In some cases, KeyChar has the correct key, but Modifiers is still 0, resulting in incorrectly assuming Ctrl was not down and e.g. inserting a ! instead of doing whatever action Ctrl+! was bound to.

This issue affects some of the key bindings in PSReadline.

[stephentoub]

In general we don't have a good way to do better than this. They're is no API similar to what Windows has for getting detailed information on the keys that were typed; we simply do our best to infer it from the escape sequences sent to the terminal.

[lzybkr]

There are key bindings that work fine in bash but not with PSReadline, e.g. C-] is character search (a little like F3 on Windows). C-@ is also used in bash to set a mark, then C-x,C-x to exchange the point and mark.

GNU readline understands these escape sequences, so presumably .Net core can as well.

[stephentoub]

GNU readline understands these escape sequences, so presumably .Net core can as well.

If you can make it work, we'd welcome a PR. This is a known limitation we shipped with (mono has a similar limitation), and this is not something we plan to revisit soon.

[stephentoub]

ps I should clarify. I haven't checked the exact combinations you're talking about. If those combinations result in a unique sequence of escape codes we can reliably recognize, then fixing it should be relatively easy. My concern is they don't, and an entirely different mechanism would be needed.

[lzybkr]

Assuming I can attempt to see the sequences easily via Console.In.Read, the keys I'm concerned about don't actually generate anything, so some investigation is necessary. I'm not sure if it's something stty controls.

At any rate, I'll try and investigate further at some point, I expect some PowerShell users will want undo to work properly on the command line on Linux.

[karelz]

@lzybkr were you able to make any progress?
cc: @ianhays

[lzybkr]

The affected key bindings aren't too critical, so I haven't found time to take a look yet.

[lzybkr]

A quick update - showkeys -a shows me that there are distinct sequences for some keys that we care about - below I typed a character followed by Alt and that character.

jason@jason:~$ showkey -a

Press any keys - Ctrl-D will terminate this program

,        44 0054 0x2c
^[,      27 0033 0x1b
         44 0054 0x2c
<        60 0074 0x3c
^[<      27 0033 0x1b
         60 0074 0x3c
.        46 0056 0x2e
^[.      27 0033 0x1b
         46 0056 0x2e
>        62 0076 0x3e
^[>      27 0033 0x1b
         62 0076 0x3e
/        47 0057 0x2f
^[/      27 0033 0x1b
         47 0057 0x2f
?        63 0077 0x3f
^[?      27 0033 0x1b
         63 0077 0x3f

After reviewing the code, I think I believe the bug is related to this method - it doesn't handle most of these characters, it returns default(ConsoleKey), which means the return from here is false, and then here we fall through to the !alt case and return just the Escape character.

It's not that useful to expect a valid ConsoleKey for these characters because it depends on the keyboard layout. On Windows, I'm forced to use the ConsoleKey because the KeyChar is sometimes '\0', but it doesn't have to be this way, so I'm open to returning a ConsoleKeyInfo with the ConsoleKey set to default(ConsoleKey).

Edit by @carlossanlop: since we moved to dotnet/runtime, the StdInReader.cs method mentioned above, returning default(ConsoleKey), is located here.

[mklement0]

Below is a summary of the existing shortcomings and how they relate to how things work in the Unix world.

Please note:

• I haven't looked at the source code, so some of this is speculative. Do tell me where I'm wrong.

Context:

The fundamental restrictions regarding Control-based chords and Alt-based chords - which come from most Unix-like OSs running xterm-emulation terminal programs are:

• Control-based chords are in effect limited to the 33 well-defined chords of caret notation - plus a few effective (but presumable accidental) aliases (see bottom); the ^ symbol is known as the caret and represents the Control key; for instance, ^A represents the chord Control+A (strictly speaking, it is the uppercase A, but at least for letters it doesn't matter whether Shift is held down or not):

  • In short, these well-defined chords map onto the 32 C0 ASCII-range control characters starting with (0x0) NUL / ^A through (0x1f) US) / ^_, plus (0x7F) DEL / ^?) and programs running inside that terminal see only the resulting control character.

• There are no Alt-based chords at all, but most modern terminal programs translate Alt+{char}-based chords (with some terminals on an opt-in basis, as on macOS) into Esc, {char} sequences, and programs running inside that terminal see only the resulting sequence.
  That's why, in the bash world, for instance, readline definitions are expressed in terms of the latter.

• (Additionally, a given terminal program itself may have key bindings for Control-based and/or Alt-based chords, which may preempt any attempts to use these chords by programs running inside these terminals.)

Shortcomings of the existing [Console]::ReadKey() implementation on Unix-like platforms:

^{Note: I'm using lowercase letters in the chord representations below to indicate chords not involving the Shift key.}

On Unix-like platforms, instead of letting programs see the translation of key chords the same way as described above - which may be a single ASCII control character or an Esc, {char} sequence - CoreFX selectively retranslates that into a chord, as you would see it on Windows_:

Control-based chords

• Sensibly, Control-based chords that translate into control characters that have dedicated keys are reflected as such in the .Key property:

  • Control+i a.k.a ^I, which is control char. HT, corresponds to the Tab key

  • Control+h a.k.a ^H, which is control char. BS, corresponds to the Backspace key

  • Control+[ a.k.a ^[, which is control char. ESC, corresponds to the Escape key.

  • Unfortunately, Control+j a.k.a ^J, which is control char. LF (0xA) is mistakenly conflated with Control+m a.k.a ^M, which is control char. CR (0xD): both result in the .KeyChar property containing `n (i.e., LF), and .Key containing Enter, which is incorrect:

    • Only ^M corresponds to key Enter, and the .KeyChar property should contain `r (i.e., CR), not `n.
    • ^J has the correct .KeyChar value (`n), but its .Key property should be J, and its .Modifier property should be Control.
      * Somewhat ironically, this distinction is correctly observed on Windows.

• Most letter-based Control-based chords (e.g., Control+a) are passed through the same way they are on Windows (rather than being translated into ASCII control characters).

  • However, letter-based chords with uppercase letters, i.e. with Shift held down (e.g., Control+Shift+a) are currently quietly ignored.

  • Additionally, curiously, Control+z, even in its "shift-less" (lowercase) variant is quietly ignored as well; note that Control+z has special meaning in traditional shells (not the terminal emulators themselves): in cmd.exe, it signals EOF during interactive input (copy con ...); in bash, it suspends the currently executing program and sends it to the background.

  • For Control+c to be recognized as a keypress (as opposed to being treated as the signal to terminate a command), [console]::TreatControlCAsInput = $true must be executed first; without that, the terminating effect is curiously delayed: nothing appears to happen at first, but the very next next keypress then terminates the call, without returning anything (only the character typed is printed to the terminal).

• Another quietly ignored chord is Control+\

• The remaining punctuation-based chords (other than Control+[ a.k.a. Esc) pass the control character correctly in the .KeyChar property, but fail to populate the .Key and .Modifier properties:

  • Control+]
  • Control+^
  • Control+_

Alt-based chords

CoreFX tries to translate Esc, {char} sequences , as reported by the terminal emulator, back into Alt+{char}-based chords, but not consistently:

• Letter-based Alt-based chords, including uppercase variants, cords are correctly translated, and behave as on Windows.

• By contrast, symbol-/punctuation-based Alt chords (e.g., Alt+') do appear to arrive as their raw Esc, {char} sequences, which [Console]::ReadKey() and therefore PSReadLine cannot handle: [Console]::ReadKey() only reports the Esc keypress

---

To predict if a given chord will currently work on Unix-like platforms, run the following from PowerShell:

# Press one key chord at a time to see its interpretation as a [ConsoleKeyInfo] instance,
# followed by the hex. representation of the resulting character (of the `.KeyChar` property).
# To exit, press Ctrl+c, then Enter.
while ($true) { $ck = [console]::readkey(); $ck | out-string; 'KeyChar code point: {0} (0x{0:x})' -f [int] [char] $ck.KeyChar }

To see what key chords [Console]::ReadKey() and therefore PSReadline should be able to handle, as @lzybkr has stated, run showkey -a in a terminal on a Linux platform (you may have to install the kbd package first).

• Column 1 is the resulting character(s).

  • For most Control- and Alt-based chords the character is represented in caret notation; and, as stated, since Alt+{char}-based chords are translated into Esc, then {char} sequences, you'll see the two resulting characters, the first of which is ^[, i.e., ESC (27 / 0x1b) in caret notation.

• Columns 2-4 are the resulting character's code point in decimal, octal, and hexadecimal notation

---

The full list of caret-notation (Control-based) key chords and their - inconsistently supported - aliases:

CaretNotation TargetControlChar ShiftAlias DeFactoAliases Comment                             
------------- ----------------- ---------- -------------- -------                             
^@            0x0 (NUL)         ^`         {^<space>, ^2}                                     
^A            0x1 (SOH)         ^a                                                            
^B            0x2 (STX)         ^b                                                            
^C            0x3 (ETX)         ^c                                                            
^D            0x4 (EOT)         ^d                                                            
^E            0x5 (ENQ)         ^e                                                            
^F            0x6 (ACK)         ^f                                                            
^G            0x7 (BEL)         ^g                                                            
^H            0x8 (BS)          ^h                        NEEDED FOR NORMAL TERMINAL OPERATION
^I            0x9 (HT)          ^i                        NEEDED FOR NORMAL TERMINAL OPERATION
^J            0xa (LF)          ^j                                                            
^K            0xb (VT)          ^k                                                            
^L            0xc (FF)          ^l                                                            
^M            0xd (CR)          ^m                        NEEDED FOR NORMAL TERMINAL OPERATION
^N            0xe (SO)          ^n                                                            
^O            0xf (SI)          ^o                                                            
^P            0x10 (DLE)        ^p                                                            
^Q            0x11 (DC1)        ^q                                                            
^R            0x12 (DC2)        ^r                                                            
^S            0x13 (DC3)        ^s                                                            
^T            0x14 (DC4)        ^t                                                            
^U            0x15 (NAK)        ^u                                                            
^V            0x16 (SYN)        ^v                                                            
^W            0x17 (ETB)        ^w                                                            
^X            0x18 (CAN)        ^x                                                            
^Y            0x19 (EM)         ^y                                                            
^Z            0x1a (SUB)        ^z                                                            
^[            0x1b (ESC)        ^{                        NEEDED FOR NORMAL TERMINAL OPERATION
^\            0x1c (FS)         ^|         ^5                                                 
^]            0x1d (GS)         ^}                                                            
^^            0x1e (RS)         ^~                                                            
^_            0x1f (US)                    {^7, ^/}                                           
^?            0x7f (DEL)                   ^8                                                 

Caveats re aliases:

• Alias chords ultimately map to the same, single control character, so they are effectively indistinguishable from the canonical caret-notation chord.

  • This fact needs to be documented, so that users understand why different chords can trigger the same PSReadLine key binding.

• To avoid confusion - and given that the aliases function inconsistently across terminal emulators - PSReadLine keybindings should only be defined in terms of the caret-notation chords.

---

• Column ShiftAliases refers to, loosely speaking, aliases that are the "+ Shift" / uppercase equivalents, calculated by adding 0x20 to a caret-notation character's code point.

  • With letter-based chords that means that whether you hold down Shift or not doesn't matter - this should apply to all Latin-alphabet-based keyboard layouts.

  • With punctuation-based chords, whether you hold down Shift or not also doesn't matter with an US-English keyboard layout, except for _ and ? (the "shifted" ^_ and ^? actually produce each other, i.e. produce a different control char.)

    • On macOS, curiously, ^` for ^@ works neither in the default terminal (Terminal.app) nor in popular replacement iTerm.app (the former ignores ^` , the latter treats it as plain `); if you run an X11 xterm emulator, however, it does work.

• Column DeFactoAliases lists additional aliases:

• It is unclear to me why these aliases work and whether they do so accidentally. Their functioning may depend on the language-specific keyboard layout.

• ^space is a de-facto alias for ^@ that works in all terminals I've looked at.

• By contrast, all the other ones - discovered experimentally with an US-English keyboard layout - seem to ONLY work in Linux xterm-256color terminals (and also in an X11 xterm emulator on macOS).

[carlossanlop]

Triage:
@lzybkr would you be interested in working on a PR for this?

[ericstj]

cc @carlossanlop @eiriktsarpalis

[carlossanlop]

Quick update:

I debugged Console.ReadKey on Ubuntu. The problem seems to be coming from the native method Interop.Sys.ReadStdin, which we are calling here:

runtime/src/libraries/System.Console/src/System/IO/StdInReader.cs

Line 76 in cf1390c

int result = Interop.CheckIo(Interop.Sys.ReadStdin(buffer, bufferSize));

If I type a key combination like Control + Left Arrow, it gets detected correctly by the native call we make here:

runtime/src/libraries/System.Console/src/System/IO/StdInReader.cs

Lines 318 to 328 in cf1390c

	if (ConsolePal.TryGetSpecialConsoleKey(_unprocessedBufferToBeRead, _startIndex, _endIndex, out keyInfo, out keyLength))
	{
	key = keyInfo.Key;
	isShift = (keyInfo.Modifiers & ConsoleModifiers.Shift) != 0;
	isAlt = (keyInfo.Modifiers & ConsoleModifiers.Alt) != 0;
	isCtrl = (keyInfo.Modifiers & ConsoleModifiers.Control) != 0;
	ch = ((keyLength == 1) ? _unprocessedBufferToBeRead[_startIndex] : '\0'); // ignore keyInfo.KeyChar
	_startIndex += keyLength;
	return true;
	}

Going through the above successful code path means that we do not go into the long switch case section mentioned in this comment, which belongs to the GetKeyFromCharValue method:

runtime/src/libraries/System.Console/src/System/IO/StdInReader.cs

Line 241 in cf1390c

internal ConsoleKey GetKeyFromCharValue(char x, out bool isShift, out bool isCtrl)

But if I try one of @lzybkr 's key combinations, Interop.Sys.ReadStdin fails to detect them.

For example, Control + ! with my keyboard layout (US), forces me to use Control + Shift + 1, but does not return anything. Interestingly, running the command showkey -a does catch the key combination Control + ! successfully.

Same with Control + @: My keyboard layout forces me to use Control + Shift + 2 and does not get detected.

Next step: Understand what we are doing in Interop.Sys.ReadStdin that is preventing us from catching such key combinations. cc: @eiriktsarpalis

[eiriktsarpalis]

Fundamentally, the issue seems to stem from the fact that the GetKeyFromCharValue method, which does not provide mappings for many common key characters such as `, ,, ., [, ], \ and =. I'm assuming this is intentional, as doing otherwise would be making assumptions about the keyboard layout.

However, this seems to have the side-effect that combining Alt with any of the above keys results in the keyChar value being replaced by the escape key, so such chords are effectibly unusable. This strikes me as being overly conservative, however it's not clear how we could improve it in a reliable way.

[iSazonov]

however it's not clear how we could improve it in a reliable way.

@eiriktsarpalis Do you say that it is impossible to use P/Invokes for keyboard layout mapping in ReadKey()?

[eiriktsarpalis]

@eiriktsarpalis Do you say that it is impossible to use P/Invokes for keyboard layout mapping in ReadKey()?

I'm saying I don't know how we could improve it. Can you recommend particular APIs we could make use of?

[iSazonov]

Maybe https://linux.die.net/man/3/getch - it uses TermInfo too or/and return two-character strings like ^A for Ctrl-A and M-A for Alt-M.

[eiriktsarpalis]

it uses TermInfo too or/and return two-character strings like ^A for Ctrl-A and M-A for Alt-M.

Unless I'm misunderstanding something, it is already the case that key combinations are surfaced as character strings compatible with what getch and friends do with keypad support enabled. The problem is mapping those back to the actual key combinations, i.o.w. how can we reliably map the sequence '\027', '=' to Alt+OemPlus that is valid for my particular keyboard layout?

[waf]

I don't see this specifically mentioned, but shift+enter is also incomplete on Linux (I'm testing on WSL2 Ubuntu). Using the following program returns different results on Windows vs Linux:

class Program
{
    static void Main(string[] args)
    {
        Console.WriteLine("Press a key:");
        var key = Console.ReadKey();
        Console.WriteLine("Key: " + key.Key);
        Console.WriteLine("KeyChar: " + key.KeyChar);
        Console.WriteLine("Modifiers: " + key.Modifiers);
    }
}

On Windows, when we press shift+enter we get:

Key: Enter
KeyChar:
Modifiers: Shift

On Linux, we get:

Key: Enter
KeyChar:
Modifiers: 0

[BDisp]

On Linux the Shift+Home and Shift+End isn't caught by the Console.ReadKey.

[jeffhandley]

This issue is likely to move to .NET 7.0, where we're anticipating a collection of Console-related work will be done. Leaving it in 6.0 for the time-being though.

[BDisp]

@waf that's normal: In Windows the System.Console modifiers keys are detected but on Linux it only detects the Enter key and the Shift key is wrapped with Escape Sequences that needs to being decoded.

[Alex-K-O-R]

Please, correct me if I'm wrong: Linux (NET.Core) doesn't have convenient ConsoleKeyInfo modifier describer because there are problems with particular combinations detection (20-30% of all) in OS.

Is that correct?

[BDisp]

The ConsoleKeyInfo modifiers in Linux (NET.Core) does not caught without decode the Ansi Escape Sequence. Only Windows OS caught the modifiers keys. Unfortunately, System.Console don't uses the Console Virtual Terminal Sequences on Windows, but only on Linux. Also, some keys aren't caught by Windows or Linux because these keys are used by the OS.

[jeffhandley]

We are going to move this out to .NET 7 with the intention of grouping this with other Console-related efforts as being discussed in #52374.

[danmoseley]

https://dev.azure.com/devdiv/DevDiv/_workitems/edit/1486859

[BDisp]

On Linux the System.Console.ReadKey is sending Shift+PageDown on a Shift+CursorDown and Shift+PageUp on a Shift+CursorUp. Also it doesn't respond on pressing Shift+End, Shift+Home, Shift+PageDown and Shift+PageUp.