Implement new Erlang shell

[garazdawi]

This PR re-implements the entire tty driver for both Unix
and Windows to use a common nif instead of two seperate drivers.

The Unix implementation works pretty much as it did before only that
a lot more of the terminal logic has been moved from Erlang to C.

The windows implementation now uses Windows Terminal Sequences, i.e.
the same sequences as most Unixes to control the terminal. This means
that werl.exe is no longer needed and erl.exe will have the "newshell"
with all the features normally only found on Unix.

The new implementation also uses dirty I/O threads for all I/O which
means that it can leave the FDs in blocking mode. This fixes problems
when the Erlang tty is interacting with other systems such as bash.

The PR also moves all non-shell I/O to use the new nifs instead of using
the "fd driver" so that the I/O for escripts use dirty IO threads. The only time
when the "fd driver" is used is when "-oldshell" is explicitly given or a shell
is requested in a system that does not support termcap.

This PR is not ready to merge yet, I'm opening it up for people to try it out and
give feedback.

[github-actions]

CT Test Results

       8 files     325 suites   3h 44m 24s ⏱️
5 157 tests 4 913 ✔️ 238 💤 6 ❌
7 234 runs  6 900 ✔️ 328 💤 6 ❌

For more details on these failures, see this check.

Results for commit cf981d3.

♻️ This comment has been updated with latest results.

To speed up review, make sure that you have read Contributing to Erlang/OTP and that all checks pass.

See the TESTING and DEVELOPMENT HowTo guides for details about how to run test locally.

Artifacts

• Complete CT logs (Download Logs)
• HTML Documentation (Download HTML Docs)
• Windows Installer

// Erlang/OTP Github Action Bot

[josevalim]

@garazdawi is there a way to detect if we are running with ANSI support or not? We need to detect this from Elixir in order to know if we should emit ANSI escape sequences or not. I assume rebar3 and others need similar functionality.

On Unix, we use test -t to check it. On Windows, we check if a registry key is set:

https://github.com/elixir-lang/elixir/blob/main/bin/elixir.bat#L170

However, I am assuming that with this patch we will have ANSI enabled on Windows regardless of the registry key? Thank you! ❤️

[essen]

The only time
when the "fd driver" is used is when "-oldshell" is explicitly given or a shell
is requested in a system that does not support termcap.

Out of curiosity which systems don't in ? Wondering if it's even necessary to worry about this.

[garazdawi]

is there a way to detect if we are running with ANSI support or not? We need to detect this from Elixir in order to know if we should emit ANSI escape sequences or not. I assume rebar3 and others need similar functionality.

On Unix, we use test -t to check it.

test -t does not check if you are running with ANSI support or not. It tests if it is a terminal. Most terminals today do support ANSI escape sequences (though varying sub/super-sets of them), so checking if you are running a terminal can be enough.

So to answer your question: No there is no way to check if we are running with ANSI support or not. There is however a way to check if standard output is a terminal, which I think is what you were really asking for.

1> io:getopts(user).
[{expand_fun,#Fun<group.0.63993037>},
 {echo,false},
 {binary,false},
 {encoding,unicode},
 {tty,true}] %% This is set to true when stdout is a isatty

On Windows, we check if a registry key is set:

https://github.com/elixir-lang/elixir/blob/main/bin/elixir.bat#L170

However, I am assuming that with this patch we will have ANSI enabled on Windows regardless of the registry key? Thank you! heart

Windows Terminal Sequences will be enabled when you start an interactive session. At the moment it will not enabled if you do "erl -noshell", or run an escript. Though now that I think about it, maybe that is the wrong approach...

[garazdawi]

Out of curiosity which systems don't in ? Wondering if it's even necessary to worry about this.

Embedded devices that care a lot about the size of the disk of their system.

[josevalim]

There is however a way to check if standard output is a terminal

Perfect, thank you!

At the moment it will not enabled if you do "erl -noshell", or run an escript. Though now that I think about it, maybe that is the wrong approach...

I would say that we also want to enable ANSI for Windows on those cases. For example, Elixir test runner does use ANSI escapes while running tests or other installation scripts, which often run with -noshell. Maybe rebar3 does the same.

Ideally, in terms of ANSI support, everything is the same regardless if the Erlang shell is available. For example, io:getopts(user) should return still return {tty, true} if we have a terminal and -noshell. :)

Out of curiosity which systems don't in ? Wondering if it's even necessary to worry about this.

I also assume the -oldshell/fd is used when redirecting stdout to a file.

[garazdawi]

Ideally, in terms of ANSI support, everything is the same regardless if the Erlang shell is available. For example, io:getopts(user) should return still return {tty, true} if we have a terminal and -noshell. :)

I'm just wondering if there are any cases when you would not want ANSI enabled on windows. We can always add some extra functionality for Windows later if it is a problem.

I also assume the -oldshell/fd is used when redirecting stdout to a file.

No, it is not. The new nifs are used there so that the work can be properly done on dirty I/O schedulers. The only cases when the fd driver is used is if "-oldshell" is passed, or libtinfo is unavailable and you attempt to start a shell.

[josevalim]

I'm just wondering if there are any cases when you would not want ANSI enabled on windows.

Do you mean technical cases? Or cases based on the use case?

For the latter, we always handle it at the application level. For example, some recommend disabling ANSI when NO_COLOR is set, so I am hoping we can change Elixir to do something like this to detect if we should emit ANSI or not:

is_user_opts_tty() and is_no_color_unset()

No, it is not.

Perfect, thanks!

[garazdawi]

I've pushed updates that:

1. Always enables Windows Terminal Sequences (unless running oldshell)
2. shell_slogan and shell_session_slogan STDLIB configuration parameters for controlling the slogans printed by the shell.
3. shell:start_interactive/0,1 for starting the shell subsystem programatically (from an escript, or erl -noinput).
4. Made erl_childsetup reset the terminal when it detects that Erlang has exited in order to fix ERL-282: escript fails when redirecting stdout to less and less terminates #3150

[josevalim]

shell:start_interactive/0,1 for starting the shell subsystem programatically (from an escript, or erl -noinput).

Fantastic!!!!!!!!

---

@garazdawi I have one additional question based on another recurring issue on Windows: what about UTF-8 support? Can we now force the terminal to be in UTF-8? We often have to ask users to set chcp 65001 before starting the terminal, otherwise data either won't be inputted or outputted correctly.

Sorry for the questions. I got a new computer and I am struggling to setup a Windows VM (due to the Apple chip). :(

[garazdawi]

what about UTF-8 support? Can we now force the terminal to be in UTF-8?

Yes it is fixed. We call SetConsoleOutputCP to set the correct output codepage.

[max-au]

One other feature me (and @josevalim I think as well) is really interested about is some way to extend pretty printing in the shell. There is some very basic mechanism for format records (after loading with rr). But otherwise current shell implementation does not allow pretty printing for user-defined types with the only exception of, well, exceptions (EEP-54).
I guess this is caused by lack of RTTI (e.g. one can't easily tell sets v2 from a map), but shell might be technically able to leverage functions specs to store types of bound variables.

[garazdawi]

@max-au I replied in Erlang Forums about something similar. We can discuss it more there if you would like. Let's keep this PR about the contents of this PR.