xterm renderer.

[HexDecimal]

Something I've always wanted in libtcod was to add an xterm renderer and other console renderers. This is useful for those who want an alternative to Curses that works in an actual terminal and those who host roguelikes over SSH servers.

Events will be parsed and sent to SDL's event system, so that SDL's event handling can still be used for all renderers. The old event API reads from SDL too, so it should all fit into place. Old programs could use the new renderer without needing to be recompiled.

• Convert a TCOD_Console into ANSI escape codes.
• Convert terminal keyboard and mouse inputs to SDL events.
• Combine the above into a new context.

Resources on this topic:
https://xn--rpa.cc/irl/term.html
https://en.wikipedia.org/wiki/ANSI_escape_code
https://invisible-island.net/xterm/ctlseqs/ctlseqs.html

[epixinvites]

When will this support be added? I kinda need it pretty soon. Or is there any way that I can help in developing this?

[HexDecimal]

If you're okay with Curses then PDCurses lets you use that API without dropping support for Windows.

The process to actually implement this would be done in 3 steps: Converting a TCOD console into escape codes, converting inputs into SDL events, and putting those together to make a new TCOD context. I might end to up doing these one at a time rather than all at once.

[epixinvites]

Well, my project will be turned into something like a server-side program, so people just SSH into my computer and can start playing the game.

[HexDecimal]

I now have a branch where I'm adding ANSI support.

https://github.com/libtcod/libtcod/tree/xterm

This only supports Windows 10 and doesn't handle events yet.

[theq629]

For events, is there a good place in the context code to hook in to check for input?

[HexDecimal]

For converting into SDL Events you mean? Right now I can't think of anything other than spawning a thread for checking for terminal input from stdin. That'd be the only way for existing code using SDL_WaitEvent to not hang.

[theq629]

Yes, reading from stdin and converting to SDL. I'll give that a try unless you have anything in progress.

[HexDecimal]

I'm not doing anything for this right now. I don't have the best setup to test with.

[theq629]

So my compiling and test setup is pretty awkward, but I've got the main features from my python package working in this branch for unixish platforms: basic keyboard events, window resize events, quit events, recommended console size, requesting console size. The input thread seems to work fine. I haven't got into handling keys like arrows that come in as escape codes yet, though.

[HexDecimal]

I tried my best to make compiling easier by explaining as much as possible in the contributing guide. Right now anything other than VSCode is going to be awkward to use, unless you have some other kind of IDE that integrates well with CMake.

Making a draft pull request to the xterm branch will help with testing your branch. It needs to compile on all platforms so you might need to disable it completely on Windows. There's also MinGW which seems to have all of the console headers but also has _WIN32 defined. You might need to use SDL to access threading in a cross-platform way.

[theq629]

Sure, I've opened a draft PR.

The autotools build from the guide actually worked fine for me, it was just trying to build the samples and my own test code against a local install that was harder. Right now everything I thought was unix-specific is under #ifndef _WIN32. I'll see if I can get set up to test on Windows. (unfortunately I'm not very used to C, C++, or Windows)

[theq629]

Also by the way, is there a reason to call this "xterm" rather than "ansi" or "console" or something?

[HexDecimal]

Because it's to specifically target modern xterm consoles, rather than something far older like VT100. Not everything has support for the 24-bit color escape codes for example.

[HexDecimal]

"console" implies that there's some kind of cross-platform console, which isn't the case. "ansi" doesn't mean anything on its own, terminal-wise.

[theq629]

I've never heard "xterm" being used as a generic term elsewhere, it's a specific terminal emulator. Pretty much all the output here is covered by the ANSI standard or what is usually called ANSI, the only thing I'm aware of that might be xterm-specific is the request terminal size escape that I added.

[HexDecimal]

It's hard to avoid the virtual terminal functions completely, and it's hard to say it's just ANSI escapes then these special functions are required to set it up correctly.

My choice on using xterm was mostly based on how the XTERM environment variable is used on most Linux systems to determine how the terminal is configured. Some programs will also use xterm as a feature set with other options being something like vt100, but that isn't consistent.

[HexDecimal]

There's also https://invisible-island.net/xterm/ctlseqs/ctlseqs.html which is the most complete guide to these control sequences I know of, which just plainly calls them "XTerm Control Sequences" while also being the most clear about which virtual terminal versions support which codes.

[theq629]

I think https://invisible-island.net/xterm/ctlseqs/ctlseqs.html is using that term because it's the xterm spec, and I'm not sure how many people normally run xterm itself any more. Isn't the standard environment variable TERM rather than XTERM? Or I guess you mean TERM=xterm, which is a good point. For me it's confusing because TERM=xterm is a standard modern capabilities setting, but xterm the software is probably the least modern terminal emulator still in use and supports old and different stuff like Tektronix 4010. I guess one extra consideration is that if you are thinking of splitting off windows support as a separate renderer then on unix it might be possible to use terminfo for all the escapes except maybe (for performance) the core present code, and then it should handle the $TERM environment properly.

This is all pretty much bike shedding and doesn't really matter though.

[HexDecimal]

Yeah, TERM=xterm is correct.

And since the XTerm spec is what I was targeting I kind of ended up with the name xterm for the renderer. It doesn't really matter what the actual terminal is as long as it supports what xterm supports. All the other terminal specs have a long and complex history that's difficult to follow. XTERM was the most concise name I could come up with.

I know it's bike shedding. I just want to be sure that XTERM isn't a really stupid name for the renderer.

[theq629]

To me it's quite a strange name just because it's not a usage I've seen before. The danger in presenting it as "as long as it supports what xterm supports" is that xterm the software supports basically everything, and other terminal emulators don't. It doesn't seem really stupid if you mean TERM=xterm (xterm termcap/terminfo capabilities, I guess, and possibly it should be xterm-truecolor) but that isn't what I think of when I see it. (I don't know that much about terminals though.)

What about calling it just "terminal" or "real-terminal" or something like that?

[HexDecimal]

What about calling it just "terminal" or "real-terminal" or something like that?

I've said before that "console" and "terminal" are too generic. They imply some kind of universal cross-platform text terminal which doesn't actually exist. If I have an implementation using ANSI escape codes and another using the Windows Console API then do you think it's clear to call them "terminal" and "console"? You can't seriously say that one is more "real" than the other.

The danger in presenting it as "as long as it supports what xterm supports" is that xterm the software supports basically everything, and other terminal emulators don't.

This is what I was going for actually. I won't mind if older consoles can't support what xterm supports, I only expect these games to be played on the most recent Linux distributions which will have a modern console. I know that things like SSH will have some trouble with mouse support and that's okay.

[theq629]

I was thinking of "real" as in a text terminal instead of a graphical window, so it would have to be like unix-real-terminal which is awkward.

I won't mind if older consoles can't support what xterm supports, I only expect these games to be played on the most recent Linux distributions which will have a modern console.

The thing is that it's the other way around for xterm the software. xterm is 37 years old and supports old terminal standards that essentially no one uses anymore. So there are effectively no older terminal emulators, and "as long as it supports what xterm supports" suggests you could actually be sending eg Tektronix 4010 codes and then the user probably needs xterm itself or a real physical Tektronix 4010.

The TERM=xterm terminfo capabilities are a good modern consensus, so it's a matter of whether people understand that meaning from the name.

[theq629]

Also to be clear, on unix "terminal" is theoretically a hardware text terminal, and "terminal emulator" is local software that pretends to be that. So xterm is a terminal emulator that supports multiple terminal standards, and TERM=xterm effectively defines a kind of terminal that only exists in emulated form, but is just one kind of terminal that xterm supports.

[HexDecimal]

I guess I should mention that I don't plan on supporting a real terminal, at least initially. I'm not really into retro hardware. I'd call the renderer something like VT100 if that was the plan.

[theq629]

Oh yes, I doubt anyone is wanting to play tcod games on a real physical terminal (although I suspect this would work on a physical VT220 except for the colour, since that's apparently what TERM=xterm is extending).

[theq629]

What about calling it "unix-terminal" and just documenting what you are aiming to support? I doubt anyone would expect it to handle older terminals, and if you find some conflict between popular modern terminal emulators you would probably want to support all of them.

Also, I'm still happy to look into using terminfo for most terminal escapes (which would increase compatibily and generally be the best thing to do for modern unix as far as I understand) if that interests you, but it would mean either parsing the terminfo files or depending on something like ncurses that includes terminfo parsing.

[HexDecimal]

What about calling it "unix-terminal" and just documenting what you are aiming to support? I doubt anyone would expect it to handle older terminals, and if you find some conflict between popular modern terminal emulators you would probably want to support all of them.

It has to be a C enum like TCOD_RENDERER_XTERM, no dashes.

I'll say I'm targeting XTerm and typical SSH supporting terminals. Are there other terminals you think should be supported?

Also, I'm still happy to look into using terminfo for most terminal escapes (which would increase compatibily and generally be the best thing to do for modern unix as far as I understand) if that interests you, but it would mean either parsing the terminfo files or depending on something like ncurses that includes terminfo parsing.

I wouldn't want to depend on Curses, but it'd be unreasonable to make an alternative to terminfo. Maybe the terminfo library should be used directly?

[epixinvites]

The reason I switched from NCurses to libtcod for my project was the nightmare experience I've had with terminfo.

[HexDecimal]

I've never used terminfo before so I haven't experienced how bad it is. I've used Curses before and I know that doesn't have the best API.

[theq629]

As far as I can tell there isn't a standard terminfo library, and ncurses might be the most common one (not the regular curses interface though, just the curs_terminfo stuff like in this example). Parsing terminfo is probably doable but annoying.

On the other hand, what we have already did work fine on several terminal emulators except for details which don't matter so much (window resize requests).

@epixinvites You were using terminfo, not the regular curses interface?

[theq629]

I'll say I'm targeting XTerm and typical SSH supporting terminals. Are there other terminals you think should be supported?

Gnome-terminal is the default for gnome and I think that includes ubuntu, so it is likely the most common Linux desktop terminal enumlator. Konsole is default for KDE. For OS X the default terminal doesn't do true colour and I think iterm2 might be the most common one that does. I don't think SSH cares about the type of terminal, does it?

Maybe something like "XTerm and modern terminal emulators with true colour support"? I'm guessing that true colour is going to be the sticking point for any modernish terminal emulator.

[HexDecimal]

Current docs, nothing is set in stone yet:

libtcod/src/libtcod/console_types.h

Lines 387 to 402 in 98b226a

	/***************************************************************************
	@brief A renderer targeting modern XTerm terminals with 24-bit color support.
	This is an experimental renderer with partial support for XTerm and SSH.
	This will work best on those terminals.
	Terminal inputs and events will be passed to SDL's event system.
	There is poor support for ANSI escapes on Windows 10.
	It is not recommended to use this renderer on Windows.
	\rst
	.. versionadded:: 1.20
	\endrst
	*/
	TCOD_RENDERER_XTERM,

For OS X the default terminal doesn't do true colour and I think iterm2 might be the most common one that does.

I really shouldn't be surprised that OS X doesn't have full color support.

I don't think SSH cares about the type of terminal, does it?

SSH is just a way for terminals to log into remote terminals. So the question is more like: what does something like PuTTY support?

[theq629]

What about something like this for the first part?

       @brief A renderer targeting XTerm and modern terminal emulators with 24-bit color support.
  
       This is an experimental renderer which aims to support terminal emulators with 24-bit color
       and common modern capabilities, with XTerm as a reference target. It can run over SSH.

what does something like PuTTY support?

As far as I know PuTTY is a terminal emulator plus SSH client together, so I think it would have the same compatibility concerns as other terminal emulators. (Apprently PuTTY supports true colour, but I'm not totally clear if that's in a release or a patch.)

[clort81]

It's very welcome to see support for avoidance of ncurses.
Termcap and Curses were created to be a compatibility API between incompatible hardware serial terminals. Terminals used differing command sequences for control and curses gave programmers a standardized way to handle the screen.
Since the advent of terminal emulators, those have converged around vt100/ansi support. Nobody runs a WYSE terminal emulator. The need for curses has largely evaporated, and using it binds you to a number of severe limitations, such as palettized foreground/background combinations, which limit the number of color combinations for fg/bg that you can use.

IMO, as an owner and user of original hardware serial terminals like the DEC VT100, VT52, Wyse etc -- xterm is the only terminal emulator. The rest are terminal emulator approximators.

vttest Tests passed:

yes | partial | no | program
488 | 4 | 6 | xterm-new
154 | 6 | 338 | xterm-r6
188 | 5 | 305 | vt220
104 | 0 | 394 | vt102
204 | 3 | 291 | rxvt
219 | 3 | 276 | urxvt
191 | 2 | 305 | putty
170 | 3 | 325 | konsole
184 | 6 | 308 | vte

https://www.invisible-island.net/xterm/xterm.faq.html

'Modern' frequently translates to 'incorrect'.

[HexDecimal]

At this point I've become unhappy with the state of the new renderer. In particular how it handles events and how I'd need to setup parity between SDL and terminal events which seems like a recipe for tedium. Making a new event API to combine them would seem ridiculous after having to throw out libtcod's old event API due to how badly implemented it was.

So what I'm thinking of as an alternative to this renderer is to finish the compile switches so that libtcod can be compiled without linking to SDL, and then suggest using Notcurses as the terminal API. Something like Notcurses is what I would've wanted out of this anyway and I have zero interest in reinventing it. The only feature libtcod would need to add is maybe outputting a tcod console to the Notcurses API.

In general, continuing to work on this requires a rich experience with terminals which I don't have to implement what's already been done better in another library. I'd simply like to applaud the Notcurses devs and defer to them when anyone wants a modern terminal API. I certainly won't be recommending PDCurses anymore. Libtcod can focus on being a terminal emulator that integrates with SDL, which is what everyone else seems to want.