New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Arrows ignored by bash when it started in ConEmu #111

Closed
Maximus5 opened this Issue Apr 10, 2016 · 76 comments

Comments

Projects
None yet
@Maximus5

I'm the author of ConEmu - alternative terminal for Windows. My users have created the issue Maximus5/ConEmu#629. When bash is started in ConEmu console, it totally ignores all extended keys (Arrows, Fn, etc.), so users can't access command line history with Up/Down arrows.

What is the problem with bash? Up/Down arrows properly cycle cmd.exe history, when it's started in ConEmu, but bash ignores them.

ConEmu is GUI application, so it get keyboard events via WM_KEYUP/WM_KEYDOWN and forward them to the created (usually hidden) real console using WriteConsoleInput.

@Maximus5

This comment has been minimized.

Show comment
Hide comment
@Maximus5

Maximus5 Apr 12, 2016

I've created workaround in ConEmu build 160411. However, it would be nice to know, is it a bug or intended behavior.

I've created workaround in ConEmu build 160411. However, it would be nice to know, is it a bug or intended behavior.

@Maximus5

This comment has been minimized.

Show comment
Hide comment
@Maximus5

Maximus5 Apr 13, 2016

Unfortunately, arrow keys still do not work in vim.

Unfortunately, arrow keys still do not work in vim.

@sunilmut

This comment has been minimized.

Show comment
Hide comment
@sunilmut

sunilmut Apr 13, 2016

Member

bash.exe has its own logic to read from console directly (this is as per the design today and subject to change in future releases). I am not a console expert, but I wonder if ConEmu interacting with the console directly is interfering with bash.exe's ability to process keyboard events?

Member

sunilmut commented Apr 13, 2016

bash.exe has its own logic to read from console directly (this is as per the design today and subject to change in future releases). I am not a console expert, but I wonder if ConEmu interacting with the console directly is interfering with bash.exe's ability to process keyboard events?

@Maximus5

This comment has been minimized.

Show comment
Hide comment
@Maximus5

Maximus5 Apr 13, 2016

ConEmu use official console API WriteConsoleInput to post keypressed into the conhost input buffer. That worked perfectly until "Bash on Ubuntu" appears.

ConEmu use official console API WriteConsoleInput to post keypressed into the conhost input buffer. That worked perfectly until "Bash on Ubuntu" appears.

@sunilmut

This comment has been minimized.

Show comment
Hide comment
@sunilmut

sunilmut Apr 13, 2016

Member

Ok. I am not sure I fully understand the problem here. My understanding of this issue is that bash is not able to process arrow keys when launched from within ConEmu? Can you help clarify.

Member

sunilmut commented Apr 13, 2016

Ok. I am not sure I fully understand the problem here. My understanding of this issue is that bash is not able to process arrow keys when launched from within ConEmu? Can you help clarify.

@Maximus5

This comment has been minimized.

Show comment
Hide comment
@Maximus5

Maximus5 Apr 13, 2016

Short answer: Yes.

Many words below.
Let's take for example UpArrow key.

When user press and release UpArrow, normal console application
recieves two console key events:

  1. bKeyDown = TRUE; wVirtualKeyCode = VK_UP;
  2. bKeyDown = FALSE; wVirtualKeyCode = VK_UP;
    References: PeekConsoleInput, ReadConsoleInput, INPUT_RECORD, KEY_EVENT_RECORD.

But Unix console applications expect different key sequences.
For UpArrow keypress either ESC [ A or ESC O A sequences are expected
depending on requested ‘DECCKM’ mode.
References: ANSI sequence ESC [ ? 1 h and ESC [ ? 1 l.

The bash.exe is Windows console application, therefore it's expected
that this application would work properly with standard Windows console API
keypresses (KEY_EVENT_RECORD, VK_UP). But, I'm not sure who, the translation
is done when user presses UpArrow. And bash.exe recieves ESC [ A instead.

Well, that is rude workaround, but if bash.exe is started in ConEmu with
-cur_console:p1 switch, ConEmu does this translation and arrows magically
start working.

That is really rude, because ConEmu does not recieve ANSI sequences from
linux processes. And, when user starts vim from bash prompt, ‘DECCKM’ is not changed.
Therefore, ‘Arrows are not working in vim under ConEmu’.
There are absolutely no way to determine that vim was started from bash,
or vim has requested to change ‘DECCKM’ mode.
Seems like linux processes (init, bash, vim) somehow write their console output
directly to conhost, bypassing bash.exe. That's bad, because ConEmu has absolutely
no control over linux processes (started from service), only bash.exe may be hooked
to catch it's output and ‘fix’ input.

If you would like to check, download latest ConEmu build (alpha 160411 ATM),
and run from Win+R:

ConEmu.exe -basic -cmd bash.exe -cur_console:p1

Short answer: Yes.

Many words below.
Let's take for example UpArrow key.

When user press and release UpArrow, normal console application
recieves two console key events:

  1. bKeyDown = TRUE; wVirtualKeyCode = VK_UP;
  2. bKeyDown = FALSE; wVirtualKeyCode = VK_UP;
    References: PeekConsoleInput, ReadConsoleInput, INPUT_RECORD, KEY_EVENT_RECORD.

But Unix console applications expect different key sequences.
For UpArrow keypress either ESC [ A or ESC O A sequences are expected
depending on requested ‘DECCKM’ mode.
References: ANSI sequence ESC [ ? 1 h and ESC [ ? 1 l.

The bash.exe is Windows console application, therefore it's expected
that this application would work properly with standard Windows console API
keypresses (KEY_EVENT_RECORD, VK_UP). But, I'm not sure who, the translation
is done when user presses UpArrow. And bash.exe recieves ESC [ A instead.

Well, that is rude workaround, but if bash.exe is started in ConEmu with
-cur_console:p1 switch, ConEmu does this translation and arrows magically
start working.

That is really rude, because ConEmu does not recieve ANSI sequences from
linux processes. And, when user starts vim from bash prompt, ‘DECCKM’ is not changed.
Therefore, ‘Arrows are not working in vim under ConEmu’.
There are absolutely no way to determine that vim was started from bash,
or vim has requested to change ‘DECCKM’ mode.
Seems like linux processes (init, bash, vim) somehow write their console output
directly to conhost, bypassing bash.exe. That's bad, because ConEmu has absolutely
no control over linux processes (started from service), only bash.exe may be hooked
to catch it's output and ‘fix’ input.

If you would like to check, download latest ConEmu build (alpha 160411 ATM),
and run from Win+R:

ConEmu.exe -basic -cmd bash.exe -cur_console:p1

@zadjii-msft zadjii-msft added the console label Apr 28, 2016

@Maximus5

This comment has been minimized.

Show comment
Hide comment
@Maximus5

Maximus5 May 2, 2016

Sure, it is. Monday, 02 May 2016, 07:09PM +03:00 from Daniel Gordon < notifications@github.com> :

So Just to be clear here - you guys are talking about bash.exe - that is or is not the same thing as the new Ubuntu Subsystem in the most recent Windows 10 dev build?

You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub

Maximus5 commented May 2, 2016

Sure, it is. Monday, 02 May 2016, 07:09PM +03:00 from Daniel Gordon < notifications@github.com> :

So Just to be clear here - you guys are talking about bash.exe - that is or is not the same thing as the new Ubuntu Subsystem in the most recent Windows 10 dev build?

You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub

@DanielGGordon

This comment has been minimized.

Show comment
Hide comment
@DanielGGordon

DanielGGordon May 2, 2016

@Maximus5 I deleted my comment because I figured it out on my own by looking at the top of this page 10 seconds after I posted my comment... Sorry for the confusion.

@Maximus5 I deleted my comment because I figured it out on my own by looking at the top of this page 10 seconds after I posted my comment... Sorry for the confusion.

@bigonazzi

This comment has been minimized.

Show comment
Hide comment
@bigonazzi

bigonazzi May 14, 2016

Hi, console2 and consoleZ, two other terminal emulators, have the same problem with arrows not working on bash.

Hi, console2 and consoleZ, two other terminal emulators, have the same problem with arrows not working on bash.

@Maximus5

This comment has been minimized.

Show comment
Hide comment
@Maximus5

Maximus5 May 14, 2016

@bigonazzi Of course they are. Only ConEmu is able to send xterm-ish keyboard sequences, which LX kernel expects.

@bigonazzi Of course they are. Only ConEmu is able to send xterm-ish keyboard sequences, which LX kernel expects.

@xilun

This comment has been minimized.

Show comment
Hide comment
@xilun

xilun May 14, 2016

Just would like to add that IMO this is quite important to have WSL working properly with ConEmu, because some serious console users (myself included) are not going to be satisfied with the basic Windows Console (which is now quite better than the legacy one, but still far away from any random XWindow Console we are used to)

For now I'm doing some dev under WSL on the Windows Console (for now coping with the international keyboard bugs it has by switching to a US layout) but I will be seriously disappointed if I can't use ConEmu for a vast majority of WSL software, and that obviously includes vim...

@Maximus5 thanks for your excellent ConEmu btw :)

xilun commented May 14, 2016

Just would like to add that IMO this is quite important to have WSL working properly with ConEmu, because some serious console users (myself included) are not going to be satisfied with the basic Windows Console (which is now quite better than the legacy one, but still far away from any random XWindow Console we are used to)

For now I'm doing some dev under WSL on the Windows Console (for now coping with the international keyboard bugs it has by switching to a US layout) but I will be seriously disappointed if I can't use ConEmu for a vast majority of WSL software, and that obviously includes vim...

@Maximus5 thanks for your excellent ConEmu btw :)

@bigonazzi

This comment has been minimized.

Show comment
Hide comment
@bigonazzi

bigonazzi May 14, 2016

@xilun +1

@Maximus5 I like conemu, all I'm saying is that the issue is bigger than just one program and it should be addressed with high priority.

@xilun +1

@Maximus5 I like conemu, all I'm saying is that the issue is bigger than just one program and it should be addressed with high priority.

@Darktex

This comment has been minimized.

Show comment
Hide comment
@Darktex

Darktex May 14, 2016

@xilun +1

The Windows default terminals don't have nearly enough features for any power user

Darktex commented May 14, 2016

@xilun +1

The Windows default terminals don't have nearly enough features for any power user

@djmott

This comment has been minimized.

Show comment
Hide comment
@djmott

djmott May 28, 2016

@Maximus5 thank you for ConEmu. It's a huge productivity improvement. MS should ditch the POS default console, give you a few million USD and ship it with every install of Windows.

djmott commented May 28, 2016

@Maximus5 thank you for ConEmu. It's a huge productivity improvement. MS should ditch the POS default console, give you a few million USD and ship it with every install of Windows.

@sunilmut

This comment has been minimized.

Show comment
Hide comment
@sunilmut

sunilmut May 28, 2016

Member

@Maximus5 - Given the popularity of ConEmu, you could also convert it to a UWP using “Project Centennial." and submit it to Windows Store for even a wider audience.

Member

sunilmut commented May 28, 2016

@Maximus5 - Given the popularity of ConEmu, you could also convert it to a UWP using “Project Centennial." and submit it to Windows Store for even a wider audience.

@nigurr

This comment has been minimized.

Show comment
Hide comment
@nigurr

nigurr Jul 13, 2016

+1 Please fix this

nigurr commented Jul 13, 2016

+1 Please fix this

@miniksa

This comment has been minimized.

Show comment
Hide comment
@miniksa

miniksa Jul 14, 2016

Member

Sorry, this is my bad. @zadjii-msft and I overlooked this portion of the design when we implemented the virtual terminal input handling in the console host such that it doesn't present itself properly for ConEmu and friends. Unfortunately because it's a design flaw, it's a little bit more work than a quick bug fix.

I have it tracked internally as 7640020. I can't commit to a date it'll be fixed, but know that it is in our tracker, it is understood, and it is being ranked/prioritized.

Member

miniksa commented Jul 14, 2016

Sorry, this is my bad. @zadjii-msft and I overlooked this portion of the design when we implemented the virtual terminal input handling in the console host such that it doesn't present itself properly for ConEmu and friends. Unfortunately because it's a design flaw, it's a little bit more work than a quick bug fix.

I have it tracked internally as 7640020. I can't commit to a date it'll be fixed, but know that it is in our tracker, it is understood, and it is being ranked/prioritized.

@bigonazzi

This comment has been minimized.

Show comment
Hide comment
@bigonazzi

bigonazzi Jul 14, 2016

@miniksa good to hear that you're listening to user feedback!

@miniksa good to hear that you're listening to user feedback!

@Novynn

This comment has been minimized.

Show comment
Hide comment
@Novynn

Novynn Jul 22, 2016

Ah, just came across this issue. I hope we'll see some progress soon!

Novynn commented Jul 22, 2016

Ah, just came across this issue. I hope we'll see some progress soon!

@Maximus5

This comment has been minimized.

Show comment
Hide comment
@Maximus5

Maximus5 Jul 22, 2016

@Novynn What progress do you want to see? ConEmu had implemented workaround and arrows started working months ago.

@Novynn What progress do you want to see? ConEmu had implemented workaround and arrows started working months ago.

@Novynn

This comment has been minimized.

Show comment
Hide comment
@Novynn

Novynn Jul 22, 2016

@Maximus5 I was meaning more towards on Microsoft's end, I wasn't aware of a workaround!

I've just started using Cmder today, which reports it's ConEmu as being 160710 stable (I see this version is the latest release on GitHub) and I am unable to use arrow keys.

I'm using cmd.exe in Cmder to start Bash on Windows to ssh into a screen session on a Linux machine. This is on the Windows 10 Insider Preview 14393.

Novynn commented Jul 22, 2016

@Maximus5 I was meaning more towards on Microsoft's end, I wasn't aware of a workaround!

I've just started using Cmder today, which reports it's ConEmu as being 160710 stable (I see this version is the latest release on GitHub) and I am unable to use arrow keys.

I'm using cmd.exe in Cmder to start Bash on Windows to ssh into a screen session on a Linux machine. This is on the Windows 10 Insider Preview 14393.

@Maximus5

This comment has been minimized.

Show comment
Hide comment
@Maximus5

Maximus5 Jul 22, 2016

ConEmu.exe -basic -run {bash}
ConEmu.exe -basic -run {bash}
@Novynn

This comment has been minimized.

Show comment
Hide comment
@Novynn

Novynn Jul 22, 2016

Ah, that makes a lot more sense! Works perfectly thanks.

I still hope that Microsoft resolves this issue so no workarounds are required on your end.

Novynn commented Jul 22, 2016

Ah, that makes a lot more sense! Works perfectly thanks.

I still hope that Microsoft resolves this issue so no workarounds are required on your end.

@pbecotte

This comment has been minimized.

Show comment
Hide comment
@pbecotte

pbecotte Jul 26, 2016

I have this issue trying to use the new bash terminal in Pycharm. They could probably implement the workaround listed here, but doesn't that still leave the terminal not working for things like vim?

I have this issue trying to use the new bash terminal in Pycharm. They could probably implement the workaround listed here, but doesn't that still leave the terminal not working for things like vim?

@DanielGGordon

This comment has been minimized.

Show comment
Hide comment
@DanielGGordon

DanielGGordon Jul 26, 2016

Last time I tried it out, arrows worked on the command line, but not in
VIM. Has this changed with a recent update?

On Tue, Jul 26, 2016 at 11:35 AM, Paul Becotte notifications@github.com
wrote:

I have this issue trying to use the new bash terminal in Pycharm. They
could probably implement the workaround listed here, but doesn't that still
leave the terminal not working for things like vim?


You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#111 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ALRu8N6Xetakj1XAO9zQffA6NUfMrT7Nks5qZjdtgaJpZM4ID8Gx
.

Last time I tried it out, arrows worked on the command line, but not in
VIM. Has this changed with a recent update?

On Tue, Jul 26, 2016 at 11:35 AM, Paul Becotte notifications@github.com
wrote:

I have this issue trying to use the new bash terminal in Pycharm. They
could probably implement the workaround listed here, but doesn't that still
leave the terminal not working for things like vim?


You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#111 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ALRu8N6Xetakj1XAO9zQffA6NUfMrT7Nks5qZjdtgaJpZM4ID8Gx
.

@pbecotte

This comment has been minimized.

Show comment
Hide comment
@pbecotte

pbecotte Jul 26, 2016

Pycharm has the parameters to call the executable with hard-coded in, so I can't just use the workaround out of the box (at least I think that's what's happening :) )

Pycharm has the parameters to call the executable with hard-coded in, so I can't just use the workaround out of the box (at least I think that's what's happening :) )

@gfairchild

This comment has been minimized.

Show comment
Hide comment
@gfairchild

gfairchild Aug 3, 2016

I'm experiencing the same vim issues another few users are reporting. Arrows work on the command line, but not in vim.

I'm experiencing the same vim issues another few users are reporting. Arrows work on the command line, but not in vim.

@Zardoz89

This comment has been minimized.

Show comment
Hide comment
@Zardoz89

Zardoz89 Aug 4, 2016

htop looks that does weird things with the arrow keys. and Midnight Comander (mc), don't get any notice of arrow keys being pressed.

Zardoz89 commented Aug 4, 2016

htop looks that does weird things with the arrow keys. and Midnight Comander (mc), don't get any notice of arrow keys being pressed.

@nergdron

This comment has been minimized.

Show comment
Hide comment
@nergdron

nergdron Aug 4, 2016

I've found that with the workarounds in ConEmu, htop locally works as expected now, but remotely via SSH it still gives the weird broken default behaviour.

nergdron commented Aug 4, 2016

I've found that with the workarounds in ConEmu, htop locally works as expected now, but remotely via SSH it still gives the weird broken default behaviour.

@meta-inf

This comment has been minimized.

Show comment
Hide comment
@meta-inf

meta-inf Aug 5, 2016

Neovim does not receive arrow keys properly with the {bash} profile.

meta-inf commented Aug 5, 2016

Neovim does not receive arrow keys properly with the {bash} profile.

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Aug 6, 2016

Same as nergdron, but also can confirm this is not a ConEmu issue: the exact same thing happens when using ssh under Bash in vanilla conhost.

It appears to be a WSL/conhost design fault that needs fixing if we're going to use ssh on Windows.

ghost commented Aug 6, 2016

Same as nergdron, but also can confirm this is not a ConEmu issue: the exact same thing happens when using ssh under Bash in vanilla conhost.

It appears to be a WSL/conhost design fault that needs fixing if we're going to use ssh on Windows.

@khards

This comment has been minimized.

Show comment
Hide comment
@khards

khards Mar 20, 2017

Weekly bump the bug bump

khards commented Mar 20, 2017

Weekly bump the bug bump

@zadjii-msft

This comment has been minimized.

Show comment
Hide comment
@zadjii-msft

zadjii-msft Mar 20, 2017

Member

Okay so just to clarify, nothing has particularly changed since @miniksa's post from July 14, 2016:

[we] overlooked this portion of the design when we implemented the virtual terminal input handling in the console host such that it doesn't present itself properly for ConEmu and friends. Unfortunately because it's a design flaw, it's a little bit more work than a quick bug fix.
I have it tracked internally as 7640020. I can't commit to a date it'll be fixed, but know that it is in our tracker, it is understood, and it is being ranked/prioritized.

Unfortunately, we have a lot of other things that are a lot higher priority than fixing this. Rest assured, when there's an update on our end, we'll post here. That way we can avoid sending weekly pings to the 27 people on this thread (and quite possibly more watching)

Member

zadjii-msft commented Mar 20, 2017

Okay so just to clarify, nothing has particularly changed since @miniksa's post from July 14, 2016:

[we] overlooked this portion of the design when we implemented the virtual terminal input handling in the console host such that it doesn't present itself properly for ConEmu and friends. Unfortunately because it's a design flaw, it's a little bit more work than a quick bug fix.
I have it tracked internally as 7640020. I can't commit to a date it'll be fixed, but know that it is in our tracker, it is understood, and it is being ranked/prioritized.

Unfortunately, we have a lot of other things that are a lot higher priority than fixing this. Rest assured, when there's an update on our end, we'll post here. That way we can avoid sending weekly pings to the 27 people on this thread (and quite possibly more watching)

@jackchammons jackchammons removed the bug label Apr 11, 2017

@bgshacklett

This comment has been minimized.

Show comment
Hide comment
@bgshacklett

bgshacklett Apr 15, 2017

@jackchammons For those of us not in the know, could you give a quick summary of why this is no-longer considered a bug?

@jackchammons For those of us not in the know, could you give a quick summary of why this is no-longer considered a bug?

@zadjii-msft zadjii-msft added the feature label Apr 17, 2017

@zadjii-msft

This comment has been minimized.

Show comment
Hide comment
@zadjii-msft

zadjii-msft Apr 17, 2017

Member

@bgshacklett Right now it's being tracked internally as work item MSFT:7640020. It's a little bit more than just a simple "bug" fix - this is more "feature" level work. Internally, the distinction makes quite a bit of difference on how we're able to plan work. This should have the feature label now to make sure that this is labeled appropriately.

Member

zadjii-msft commented Apr 17, 2017

@bgshacklett Right now it's being tracked internally as work item MSFT:7640020. It's a little bit more than just a simple "bug" fix - this is more "feature" level work. Internally, the distinction makes quite a bit of difference on how we're able to plan work. This should have the feature label now to make sure that this is labeled appropriately.

@nigurr

This comment has been minimized.

Show comment
Hide comment
@nigurr

nigurr Apr 18, 2017

@zadjii-msft Thanks for the update. it would be great if you can assign the milestone or tentative timeline for shipping this

nigurr commented Apr 18, 2017

@zadjii-msft Thanks for the update. it would be great if you can assign the milestone or tentative timeline for shipping this

@zadjii-msft

This comment has been minimized.

Show comment
Hide comment
@zadjii-msft

zadjii-msft Apr 18, 2017

Member

@nigurr Unfortunately I don't have a tentative timeline for this fix at the moment, sorry.

Member

zadjii-msft commented Apr 18, 2017

@nigurr Unfortunately I don't have a tentative timeline for this fix at the moment, sorry.

@jackchammons

This comment has been minimized.

Show comment
Hide comment
@jackchammons

jackchammons Apr 19, 2017

Member

@bgshacklett Just doing a little house keeping on the GH labels. It's still an issue that we plan on addressing but it's being tracked internally as a feature. Semantics, I know, but it helps us keep things organized.

Member

jackchammons commented Apr 19, 2017

@bgshacklett Just doing a little house keeping on the GH labels. It's still an issue that we plan on addressing but it's being tracked internally as a feature. Semantics, I know, but it helps us keep things organized.

@tobico

This comment has been minimized.

Show comment
Hide comment
@tobico

tobico Aug 11, 2017

Please consider prioritizing the fix for this. Having a wide variety of quality Terminal applications available is very important for WSL to be a useful feature of Windows.

tobico commented Aug 11, 2017

Please consider prioritizing the fix for this. Having a wide variety of quality Terminal applications available is very important for WSL to be a useful feature of Windows.

@DropsOfZut

This comment has been minimized.

Show comment
Hide comment

solved?

@ichenfeng

This comment has been minimized.

Show comment
Hide comment
@ichenfeng

ichenfeng Aug 19, 2017

Please consider fixing this.
In ConsoleZ(another third-party terminator), Arrows doesn't work too. and The reason for using third-party terminator is for the terrible font setting for CMD and PowerShell in Chinese edition.

Please consider fixing this.
In ConsoleZ(another third-party terminator), Arrows doesn't work too. and The reason for using third-party terminator is for the terrible font setting for CMD and PowerShell in Chinese edition.

@i336

This comment has been minimized.

Show comment
Hide comment
@i336

i336 Aug 19, 2017

That's a very good point - the new console subsystem needs to have really, really good Unicode support. Modern programming languages support unicode strings and expect to be able to use it for I/O, and there are lots of prompt hacks using unicode characters as well.

I figured I might as well ride the bit of traffic that's been generated right now to chime in and add that the community remains super interested in a series of blog posts on the architectural updates. An in-depth writeup that adds every last detail that's legally possible would be widely appreciated - both because architectural upgrade trivia is always really fun, and also because it'll clearly explain why everything took so long.

I personally completely understand that this bugfix is the tail end of a lot of planets aligning being strong-armed into place, and that it'll take a little while. The console subsystem probably still has design cruft in it from Windows 3.1 - the NTVDM code doesn't ship in most (64-bit) installations, but is still builds for 32-bit, so it's still in the codebase. Knowing Microsoft's obsession with backward-compatibility, fixing this will not be easy.

For people frustrated about this, I have an open question (I don't have any Windows-capable hardware at the moment, so I unfortunately can't try this out myself): Can urxvt or xterm from Cygwin running on a Windows X server talk to WSL? If it can, that could be a very workable alternative.

For the MS devs working on this issue, when you see this update - I understand that speculating on timelines creates lots of avoidable fallout, so I won't ask for that. What I will ask for is a vague idea of the status of the internal tracking bug. How far have 7640020 and/or its dependencies/blockers progressed since the bug was initially filed (around August last year)?

I'm not trying to extricate an executive "I'm impatient, hurry up" summary. This is not ready yet, not much to do except face that. That's fine. But to know that there has been some movement would be cool.

Ah, if only you could share internal details. Then you could keep us in the loop about "the architecture, the customers, the issues, and the dependencies" :D and what the progress is with each. We wouldn't mind <3

i336 commented Aug 19, 2017

That's a very good point - the new console subsystem needs to have really, really good Unicode support. Modern programming languages support unicode strings and expect to be able to use it for I/O, and there are lots of prompt hacks using unicode characters as well.

I figured I might as well ride the bit of traffic that's been generated right now to chime in and add that the community remains super interested in a series of blog posts on the architectural updates. An in-depth writeup that adds every last detail that's legally possible would be widely appreciated - both because architectural upgrade trivia is always really fun, and also because it'll clearly explain why everything took so long.

I personally completely understand that this bugfix is the tail end of a lot of planets aligning being strong-armed into place, and that it'll take a little while. The console subsystem probably still has design cruft in it from Windows 3.1 - the NTVDM code doesn't ship in most (64-bit) installations, but is still builds for 32-bit, so it's still in the codebase. Knowing Microsoft's obsession with backward-compatibility, fixing this will not be easy.

For people frustrated about this, I have an open question (I don't have any Windows-capable hardware at the moment, so I unfortunately can't try this out myself): Can urxvt or xterm from Cygwin running on a Windows X server talk to WSL? If it can, that could be a very workable alternative.

For the MS devs working on this issue, when you see this update - I understand that speculating on timelines creates lots of avoidable fallout, so I won't ask for that. What I will ask for is a vague idea of the status of the internal tracking bug. How far have 7640020 and/or its dependencies/blockers progressed since the bug was initially filed (around August last year)?

I'm not trying to extricate an executive "I'm impatient, hurry up" summary. This is not ready yet, not much to do except face that. That's fine. But to know that there has been some movement would be cool.

Ah, if only you could share internal details. Then you could keep us in the loop about "the architecture, the customers, the issues, and the dependencies" :D and what the progress is with each. We wouldn't mind <3

@rprichard

This comment has been minimized.

Show comment
Hide comment
@rprichard

rprichard Aug 19, 2017

FWIW, winpty works around this issue by detecting when the console's ENABLE_VIRTUAL_TERMINAL_INPUT flag is enabled and, when it is, switching to an alternate system for generating console input:

Unfortunately, sending a window message doesn't work quite right, because the resulting console input records seem to use modifier state from the keyboard. winpty would like to synthesize the modifier state, but I don't think there's a way to do that using SendMessage. This issue also affects Ctrl-C handling, which is a more noticeable problem with winpty. (See rprichard/winpty#116 and https://github.com/rprichard/winpty/blob/ce9239af5d751195ea6982a41c7d71356ac9201c/src/agent/ConsoleInput.cc#L361.)

FWIW, winpty works around this issue by detecting when the console's ENABLE_VIRTUAL_TERMINAL_INPUT flag is enabled and, when it is, switching to an alternate system for generating console input:

Unfortunately, sending a window message doesn't work quite right, because the resulting console input records seem to use modifier state from the keyboard. winpty would like to synthesize the modifier state, but I don't think there's a way to do that using SendMessage. This issue also affects Ctrl-C handling, which is a more noticeable problem with winpty. (See rprichard/winpty#116 and https://github.com/rprichard/winpty/blob/ce9239af5d751195ea6982a41c7d71356ac9201c/src/agent/ConsoleInput.cc#L361.)

@rprichard

This comment has been minimized.

Show comment
Hide comment
@rprichard

rprichard Aug 19, 2017

AFAIK, other console emulators on Windows (e.g. ConEmu and ConsoleZ) ought to be able to work around this issue the same way that winpty does.

@i336

Can urxvt or xterm from Cygwin running on a Windows X server talk to WSL?

AFAIK, one could run WSL urxvt or xterm connecting to any Windows X server, including Cygwin's. One could also run any Cygwin terminal emulator (including mintty, urxvt, xterm) and connect to WSL using https://github.com/rprichard/wslbridge (a program I wrote). wslbridge is also shipped as part of https://github.com/mintty/wsltty, a standalone package bundling mintty, wslbridge, and a Cygwin runtime DLL.

AFAIK, other console emulators on Windows (e.g. ConEmu and ConsoleZ) ought to be able to work around this issue the same way that winpty does.

@i336

Can urxvt or xterm from Cygwin running on a Windows X server talk to WSL?

AFAIK, one could run WSL urxvt or xterm connecting to any Windows X server, including Cygwin's. One could also run any Cygwin terminal emulator (including mintty, urxvt, xterm) and connect to WSL using https://github.com/rprichard/wslbridge (a program I wrote). wslbridge is also shipped as part of https://github.com/mintty/wsltty, a standalone package bundling mintty, wslbridge, and a Cygwin runtime DLL.

@Maximus5

This comment has been minimized.

Show comment
Hide comment
@Maximus5

Maximus5 Aug 19, 2017

FYI ConEmu has implemented a workaround long long ago

FYI ConEmu has implemented a workaround long long ago

@fpqc

This comment has been minimized.

Show comment
Hide comment
@fpqc

fpqc Aug 20, 2017

@Maximus5 I once got the msys2 bridge + wslbridge working together in tandem (without MinTTY), but it stopped working after like an hour. Have you done any testing with it?

fpqc commented Aug 20, 2017

@Maximus5 I once got the msys2 bridge + wslbridge working together in tandem (without MinTTY), but it stopped working after like an hour. Have you done any testing with it?

@Maximus5

This comment has been minimized.

Show comment
Hide comment
@Maximus5

Maximus5 Aug 20, 2017

Of course. It's working. Just use latest ConEmu version.

Of course. It's working. Just use latest ConEmu version.

@zadjii-msft

This comment has been minimized.

Show comment
Hide comment
@zadjii-msft

zadjii-msft Sep 8, 2017

Member

Hey everyone,

I finally have an update on this bug! I have a potential fix in code review right now.

To summarize the problem: We were only translating KEY_EVENTs into appropriate VT sequences when the events arrived from the window message loop. I moved the translation layer around, and now the INPUT_RECORDs written to the input buffer from the API (WriteConsoleInput) will also get translated appropriately.

It's not going to be available in the Fall Creator's Update, but it should be in the release that follows.

Thanks for your patience!

Member

zadjii-msft commented Sep 8, 2017

Hey everyone,

I finally have an update on this bug! I have a potential fix in code review right now.

To summarize the problem: We were only translating KEY_EVENTs into appropriate VT sequences when the events arrived from the window message loop. I moved the translation layer around, and now the INPUT_RECORDs written to the input buffer from the API (WriteConsoleInput) will also get translated appropriately.

It's not going to be available in the Fall Creator's Update, but it should be in the release that follows.

Thanks for your patience!

@zadjii-msft zadjii-msft removed the backlog label Sep 8, 2017

@jarun

This comment has been minimized.

Show comment
Hide comment
@jarun

jarun Sep 8, 2017

Good news. 👍 I just wasted an hour figuring out the issue and reached here.

jarun commented Sep 8, 2017

Good news. 👍 I just wasted an hour figuring out the issue and reached here.

@rprichard

This comment has been minimized.

Show comment
Hide comment
@rprichard

rprichard Sep 8, 2017

@zadjii-msft Good to hear. A few questions:

  1. Does this also affect recognition of Ctrl-C or Ctrl-Break? That is, when ENABLE_PROCESSED_INPUT is on, will WriteConsoleInput send a signal for these key presses? It currently doesn't, and I'm guessing that won't change. (If it did, I might be able to fix rprichard/winpty#116.)

  2. Is mouse input affected? There's an analogous situation where mouse input should produce VT escape sequences, and the encoding is controlled by mouse mode flags (e.g. 1006 for the "SGR mouse encoding"). I haven't gotten around to testing the console yet, so maybe WriteConsoleInput calls already do this conversion.

  3. Do you have suggestions on how programs can detect this behavior change (i.e. so that a workaround can be disabled)? For my situation (winpty), I think it'd be worth it to temporarily enable the VT input mode and test whether input is converted (by calling WriteConsoleInput then ReadConsoleInput). I'm guessing there isn't a better approach, though I suppose checking the OS version could be faster.

@zadjii-msft Good to hear. A few questions:

  1. Does this also affect recognition of Ctrl-C or Ctrl-Break? That is, when ENABLE_PROCESSED_INPUT is on, will WriteConsoleInput send a signal for these key presses? It currently doesn't, and I'm guessing that won't change. (If it did, I might be able to fix rprichard/winpty#116.)

  2. Is mouse input affected? There's an analogous situation where mouse input should produce VT escape sequences, and the encoding is controlled by mouse mode flags (e.g. 1006 for the "SGR mouse encoding"). I haven't gotten around to testing the console yet, so maybe WriteConsoleInput calls already do this conversion.

  3. Do you have suggestions on how programs can detect this behavior change (i.e. so that a workaround can be disabled)? For my situation (winpty), I think it'd be worth it to temporarily enable the VT input mode and test whether input is converted (by calling WriteConsoleInput then ReadConsoleInput). I'm guessing there isn't a better approach, though I suppose checking the OS version could be faster.

@ericblade

This comment has been minimized.

Show comment
Hide comment
@ericblade

ericblade Sep 11, 2017

That is great news (not so great that it won't make Fall.. oh well :-) ) .. just curious if this means that it ended up being significantly less work than initially expected? :-D

That is great news (not so great that it won't make Fall.. oh well :-) ) .. just curious if this means that it ended up being significantly less work than initially expected? :-D

@zadjii-msft

This comment has been minimized.

Show comment
Hide comment
@zadjii-msft

zadjii-msft Sep 11, 2017

Member

@rprichard

  1. I don't believe it will. The change is pretty limited in scope to just the VT translator, so I don't believe any of the rest of the input is modified.
  2. Mouse is unfortunately not fixed by this change. The keyboard events were easier to move than the mouse events unfortunately. I have a work item to address that issue before the end of the next release.
  3. I don't have any good recommendations on that topic, sorry. Both the mechanisms you describe sound like they should work, though I have no idea which would be faster.

It was a bit easier than anticipated, but like I mentioned, mouse input is actually harder than anticipated unfortunately.

Member

zadjii-msft commented Sep 11, 2017

@rprichard

  1. I don't believe it will. The change is pretty limited in scope to just the VT translator, so I don't believe any of the rest of the input is modified.
  2. Mouse is unfortunately not fixed by this change. The keyboard events were easier to move than the mouse events unfortunately. I have a work item to address that issue before the end of the next release.
  3. I don't have any good recommendations on that topic, sorry. Both the mechanisms you describe sound like they should work, though I have no idea which would be faster.

It was a bit easier than anticipated, but like I mentioned, mouse input is actually harder than anticipated unfortunately.

@bgshacklett

This comment has been minimized.

Show comment
Hide comment
@bgshacklett

bgshacklett Sep 11, 2017

Sorry, guys, I know you're being bombarded, but can you tell us how being on the Windows Insider fast track might affect the speed at which we see some of these changes?

Sorry, guys, I know you're being bombarded, but can you tell us how being on the Windows Insider fast track might affect the speed at which we see some of these changes?

@zadjii-msft

This comment has been minimized.

Show comment
Hide comment
@zadjii-msft

zadjii-msft Sep 11, 2017

Member

@bgshacklett If you're on the fast ring, you'll see these changes pretty immediately after they hit winmain, which is usually a month or so after I commit the code on my end. Otherwise, if you're on just a non-insider's build, you'll have to wait for the entire release to be finished, which is about twice a year (based on previous Windows 10 releases)

Member

zadjii-msft commented Sep 11, 2017

@bgshacklett If you're on the fast ring, you'll see these changes pretty immediately after they hit winmain, which is usually a month or so after I commit the code on my end. Otherwise, if you're on just a non-insider's build, you'll have to wait for the entire release to be finished, which is about twice a year (based on previous Windows 10 releases)

@bitcrazed

This comment has been minimized.

Show comment
Hide comment
@bitcrazed

bitcrazed May 22, 2018

Collaborator

Thanks for the discussion.

Closing this issue since:

  1. This issue is now fixed
  2. This is the WSL issues repo, but this is an issue in Console which has its own Console GitHub Repo
  3. GitHub doesn't allow issues to be moved between repos, preserving posters' identity :(

If you have further asks/issues, please file new issues on our Console GitHub Repo.

Collaborator

bitcrazed commented May 22, 2018

Thanks for the discussion.

Closing this issue since:

  1. This issue is now fixed
  2. This is the WSL issues repo, but this is an issue in Console which has its own Console GitHub Repo
  3. GitHub doesn't allow issues to be moved between repos, preserving posters' identity :(

If you have further asks/issues, please file new issues on our Console GitHub Repo.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment