Differences between vt100/xterm and windows console

[samiam95124]

Windows Terminal version (or Windows build number)

Windows 10

Other Software

No response

Steps to reproduce

Windows console does not implement many vt100/xterm controls. Basically, for fun I did a ssh to a linux computer and noticed that the console implementation in Windows tries, but does not quite succeed, in emulating terminal sequences for Xterm. This was noted by running a program on the linux machine written for xterm.
First, its great that windows console has a terminal emulation mode. This used to be an add-on, so it is a big improvement. I mainly wanted to find out if there was interest in taking the emulation to another, more complete level.
The current xterm emulation fails on several features, for example it does not implement text colors. However it also fails on several other basic screen controls. For the clearest example, contrast this with teraterm/ssh, which works perfectly (as I am sure is true of other xterm emulators).
Thanks,
Scott Franco
samiam@moorecad.com

Expected Behavior

See previous.

Actual Behavior

See previous.

[DHowett]

Thanks for filing this! I read it and was quite confused, though:

The current xterm emulation fails on several features, for example it does not implement text colors.

What SSH client are you using?

Terminal is fairly well compliant with xterm+truecolor... MC works fine, even in full color mode, and even complicated things like screen regions and margins and DECOM et al work. This is true even outside of Terminal, and has been roughly true for the past fiveish years.

The problem with the Windows Console is that many applications think they know better than us and don't send us those control sequences.

[DHowett]

We even track our support against xterm and vt52-vt520 across the code changes in our repository! Here's a snapshot of the latest support.

We're pretty serious about this stuff, so we would love to get any additional information on the software you're using that might be getting in the way. 😄

[samiam95124]

Hi, good to know you are interested in
high fidelity emulation. some answers:

Fails to turn off cursor:

This test is supposed to present large X's in the corners. Its a test if we can turn off automatic scrolling:

Character attributes, lots of attributes missing:

There are other test issues, but I suspect they are consequences of the above issues.

Now lets do it with teraterm (still on same windows 10 machine):

Cursor can be turned off, this is good:

Scrolling control works:

Colors and attributes:

Now this I didn't expect, the mouse works!

Xterm can send mouse positions back to the remote program by special character sequences. You have to turn it on with a special character sequence
in your program.

[samiam95124]

Ok, so details.

Here is the project, its open source:

https://sourceforge.net/projects/petitami/

Now, I HATE when people just point me to their program and basically say "go fish", good luck finding the problem. I am NOT that guy. I can ship you a binary, but it will likely pull in a bunch of .sos, so, (like most Linux projects) its actually easier to get the source tree. It needs to be run on a linux machine, Ubuntu 20.04 at the moment (VM would work), and you ssh to that. It has a windows version of the test program in it, but it uses the windows console API, so its not going to help. Once you do basic setup, reading the INSTALL file does the real work, then its just

make terminal_test

Ok, why am I doing any of this. PA or Petit-Ami is a portable tool kit (TK) designed to port programs. The module terminal gives a terminal mode (character mode) API, and there is one in the Linux directory and one in the Windows directory. The linux one uses xterm character sequences for the majority of its work. The Windows one uses the windows console API. The program terminal_test is a pretty complete test of the terminal API, which just happens to be a pretty good test of the underlying Xterm or Windows console API as well. It uses "unusual" xterm features such as (quoting from memory) a buffer flip mode that preserves the original screen and also disables scrolling. After we do that my code does the scrolling itself.

Anyways, I have used the system for a while on linux/xterm, and I always wanted to see how well it worked remotely via SSH, so I tried a couple of combinations. xterm from another linux machine works fine (no surprise there), and so does teraterm from Windows 10. I would not have been too shocked if that didn't work, but it did -- kudos to the teraterm guys.

For your part, I tried that on a regular console window. I don't think ssh (openssh) does much or any manipulation on its character stream. In the way old days, windows console didn't do any terminal character interpretation, you had to add it with another program. Now it appears it does, which is great. xterm emulates VT102 (according to the documentation) and I would not hold it against anyone for not following every oddity of xterm (a lot of which I find annoying as well). But if you are interested in "extra credit" i'd love help in any way I can, just let me know.

[samiam95124]

PS downloaded and tried it with Windows Terminal, it just goes all black. The test program flips the colors to black on white, so I suspect all it did was color the foreground black so it is printing black on black.

Regards,

Scott Franco

[samiam95124]

Just for grins, I tried another terminal emulator:

putty - works complete, does not support the remote mouse.

I looked around, but not much else was free, and I don't feel like setting up cygwin. No offense to cygwiners, but if I want a linux emulator I will run linux.

[DHowett]

Wow, this is comprehensive! Thanks so much.

This is my “before I get to the office” response, so it will be pretty sparse.

Would you be willing to try specifying the path to “c:\windows\system32\openssh\ssh.exe” so that we may avoid the MSYS one?

Earlier versions of the msys runtime implement their own VT parser, because they have to work on Windows 7/8/8.1 where the console infrastructure unfortunately doesn’t have support for anything.

This processing happens in the runtime in their virtual TTY device’s I/O wrapper, so any connected client applications are unaware that it’s happening.

The version of ssh that comes with Windows, in system32, delegates all VT parsing to us (explicitly- I had to request that they do so 😄)

I’ll document a bit more when I’m back in my office.

[samiam95124]

Sure, it just has to wait for tonight since I have a day job.

Cheers,

Scott Franco

PS so interesting story (or not), that code actually came from windows originally. The first version used ANSI/VT100 escapes and ran using an ANSI.SYS TSR on DOS, and later ran in a DOS window under Windows, some time around 1987. With Windows NT about 1993, and subsequently with windows 95 and the Win32 port, I adapted it to use the console API. The start of the code is even older than that, about 1983 on a Wyse 80 serial terminal. The original code laid unused for years, since there was not much demand for ANSI mode nor many serial terminals left in the world, but when I did the XTERM port most of it came back again, since, at heart XTERM is just an enhanced serial terminal disguised as Xwindows program.

[samiam95124]

Cursor off:

Last line test:

Colors and attributes:

Mouse:

Wow I am stunned. It was indeed the ssh program, seems it was ripping out control characters or somesuch. I bow to the masters and apologize for doubting you.

I get better results on Windows terminal now. I did notice that it does not turn the cursor back on:

I also saw one oddity (only in windows terminal) where it painted a black background where it should have been white (the program relies on blank characters to be written with background colors).

The mouse didn't work, but that is an extra credit feature. If you have
any interest in supporting that, this covers it:

https://www.systutorials.com/docs/linux/man/4-console_codes/

Thanks again for the help/support.
Scott Franco
San Jose, CA

[samiam95124]

I checked the openssh man page, it does step on escapes, but (supposedly) the "-e none" option disables that and "makes it fully transparent". Nope, does not work. Lesson learned.

[DHowett]

Thanks for following up! I'm glad it was a simple solution -- rather than "oh no, we have to implement all of the control sequences now" 😁

Funny enough, we do (should?) support mouse mode, but only in the more common X11 ways (not X10):

mode	supported
X10 (DECSET 9)	❌
VT200 (DECSET 1000)	✔️
VT200 highlight event (DECSET 1001)	❌
button event (DECSET 1002)	✔️
any event (DECSET 1003)	✔️
focus event (DECSET 1004)	❌
alternate screen scroll (DECSET 1007)	✔️❌ (works in console, doesn't work in terminal)

encoding	supported
X10	✔️
UTF-8 (DECSET 1005)	✔️ (edited: we DO support this!)
SGR (DECSET 1006)	✔️ (also, best encoding)
URXVT (DECSET 1015)	❌
SGR-Pixel (DECSET 1016)	❌

of note: the version of OpenSSH that shipped with Windows until the recent update (7.7) was built in such a way that it deletes our incoming mouse report sequences 😬 however, they fixed that for 8.1, the current version

We usually use vttest to validate these things, but we'll see if we can't get yours up and running as well. Thanks for the link!

[samiam95124]

So the method I used for mouse support was for xterm is from X10???

So my take is that the linux/terminal package was written for xterm (on linux). I didn't really check if the xterm specifics were also ANSI/VT102, since it targeted XTERM specifically. But then the fact that it means I am using a protocol that happens to be widely implemented changes things a bit.

The most questionable things about XTERM that I implemented are (and keep in mind that I did this a while ago):

1. As I recall, I needed a special command to lock scrolling off that also happens to flip buffers so that the original screen is stored/restored.
2. The mouse (QED).
3. The AIX colors. I had to do this to get the colors I wanted, an example being bright white. It is tied to a define.

If you can point me to an internet source that covers the mouse method windows implements that, it would be appreciated. I would, of course, evaluate it for transparency, meaning if I can test both types of mouse support and determine which one the terminal attached supports.

Regards,

Scott Franco