Add os:isatty/1 as a bif

[andrewjstone]

The posix libc function, isatty(), is a useful function that will tell
you if a file descriptor is attached to a terminal. If, for instance, you
want to write a program that uses ansi color codes, it is not enough to
just check $TERM because the output could be redirected to a file. In
that case you would end up with the escape codes in the file making it
harder to read and parse. By calling os:isatty/1 with stdout as its
parameter the program can ensure that it never writes escape codes to
log files.

This PR adds os:isatty/1 as well as docs and a test of its argument
handling. os:isatty/1 accepts the atoms stdin, stdout and stderr as well
as any 32 bit non_negative_integer() representing a file descriptor. It
simply returns false in case the integer is not an FD, and throws in the
case of a bad argument. The test does not check the return value in the
valid case, because that would depend upon how tests are run.

This code should work on all platforms including Windows, as isatty is a
posix function. Note however that it is deprecated on Windows according
to MSDN in favor of the C++ standard _isatty().

[ghost]

If, for instance, you want to write a program that uses ansi color codes, it is not enough to
just check $TERM because the output could be redirected to a file. In that case you would end up with the escape codes in the file making it harder to read and parse. By calling os:isatty/1 with stdout as its parameter the program can ensure that it never writes escape codes to log files.

Regardless of the usefulness of isatty/1, isn't the right way to solve this to query the terminal's capabilities?

[josevalim]

I just want to mention this function would be extremely useful in Elixir too.

@Tuncer As far as I know, there is no correct way of testing if a terminal accepts ANSI escape codes. Currently what we do in Elixir is to check if STDOUT is a tty during Elixir start scripts on UNIX systems. The downside of this approach though is that we can't do such checks for escripts, which then have coloring disabled by default (even on UNIX). The function added here would fix this bug and help us clean up the existing logic by moving the check to Erlang/Elixir land, which is a very welcome addition.

[nox]

Does it work on Windows out of the box?

[nox]

Not sure I like the names stdout/stderr/stdin given they don't correspond to any builtin group leader.

[ghost]

To be clear, I don't object the patch. It's just that the color problem cannot be solved by checking only isatty/1, as you cannot assume all terminals to properly interpret color codes.

@josevalim wrote:

@Tuncer As far as I know, there is no correct way of testing if a terminal accepts ANSI escape codes. Currently what we do in Elixir is to check if STDOUT is a tty during Elixir start scripts on UNIX systems. The downside of this approach though is that we can't do such checks for escripts, which then have coloring disabled by default (even on UNIX). The function added here would fix this bug and help us clean up the existing logic by moving the check to Erlang/Elixir land, which is a very welcome addition.

Just like io:columns/1 and io:rows/1, can't we also expose tput setab bgcolor, tput setaf fgcolor, tput smul, etc.?

[andrewjstone]

@Tuncer You are correct, you also would need to check $TERM to ensure the terminal supports colors or other codes. This function just ensures that you are indeed talking to a terminal.

@nox Originally I only allowed non_negative_integers(), but the atoms were suggested as a useful addition. Not sure I understand the issue with group leaders.

Also, it looks like this implementation may have issues on windows 8: https://lists.gnu.org/archive/html/bug-gnulib/2013-01/msg00007.html

[garazdawi]

I haven't given this a lot of thought, but instinctually I like @Tuncer's idea of trying to introduce this into the io protocol, i.e. io:isatty(Device). This would delegate the check into sys.c and ttysl_drv.c, where platform dependent decisions could be made. What do you think about this?

random thought: Maybe the direction we want to move in is towards: io:tput(Device, setab, 1) etc? I looked into the ncurses api, and they have things such as has_colors() which could be very useful in figuring out whether the terminal supports colors or not. Though generating fancy color codes that change all the time would be a pain with that api...

[andrewjstone]

@garazdawi I don't have a real problem with moving this code to the io module. The reason I put it in os is that it is a standard posix call, and I want to gauge the OTP team's interest in opening up the os module, or some other module perhaps, to introduce other posix calls such as ulimit.

I also don't really think that the color code stuff has to go in erlang proper. It can live just fine in a library without locking anyone into a specific implementation, as long as the tools are there to allow an erlang program to do the right thing when it wants.

[ghost]

@andrewjstone wrote:

I also don't really think that the color code stuff has to go in erlang proper. It can live just fine in a library without locking anyone into a specific implementation, as long as the tools are there to allow an erlang program to do the right thing when it wants.

It would be a useful extension of the existing tty code like io:columns/1, and as long as it requires native code, it's also much easier for escripts to use that way.

[OTP-Maintainer]

Patch has passed first testings and has been assigned to be reviewed

---

I am a script, I am not human

---

[andrewjstone]

Ahh @Tuncer I didn't even realize io:columns and io:rows did what they did
until just checking. I missed that in your comment earlier. In that
respect, yes maybe this does fit better in the io module, and those other
functions seem useful as well.

On Tue, Sep 23, 2014 at 4:06 PM, OTP Maintainer notifications@github.com
wrote:

Patch has passed first testings and has been assigned to be reviewed

I am a script, I am not human

—
Reply to this email directly or view it on GitHub
#480 (comment).

[garazdawi]

Another good thing about the io API is that it is distributed. So things such as erl -remsh could do proper detection of whether a tty is available. It is however quite a lot more work to do, but doing it "the right way" usually is.

[andrewjstone]

@garazdawi @Tuncer Just an FYI. I'm working on a new patch to use the io protocol instead as you have suggested. I assume I'll need to plug it into all the builtin io servers as well. Doesn't seem too hard. Thanks for all the suggestions, this is a learning experience.

[ghost]

Great, looking forward to it.

[andrewjstone]

@Tuncer @garazdawi @nox I just pushed up a new branch with changes that seem to work on unix systems at least. io:isatty/0 still returns {error, enotsup} though when using remsh. I'm not sure how to fix that. Does this patch look appropriate?. Am I on the right track? Am I missing anything obvious besides docs and tests?

If it looks good I'll clean it up and open a new PR. I just wanted a bit of input first. Thanks for your time.

[josevalim]

Tiny feedback: I am wondering if we should start wrapping those requests in tuples like {tty, isatty}, {tty, get_columns} and so on. This way if you are not a tty you can just match on {tty, _} and return enotsup instead of having to handle each command individually. This will be increasingly important if we plan to add more and more commands to it.

[garazdawi]

Hello,

just letting you know that I've seen the additions and will hopefully get back to you tomorrow.

[garazdawi]

I've been thinking a little more about this, and I'm torn about what to do.

Ideally I do not think there should be any such capabilities check in Erlang at all. Instead we should implement something similar to/exactly like ANSI escape codes into the io protocol. The ttsl driver for windows/unix can then do what it can with that data and use termcap/win_con to draw/detect whatever is possible. We should also add an option which makes it so that Erlang starts without a shell but with the ttysl driver (for usage by escripts for instance), and also of course make it possible to ignore escape sequences (necessary if we are running under run_erl). This would make it possible to implement things such as ^L, color coding etc on all OSs and also io forwarding with colors for things such as -remsh and -slave would just work. We could correctly get colors in the ssh shell, common_test logs etc etc.

However such an implementation is, I believe, quite far away, and the question becomes whether should we implement something small which allows the application writer (i.e. you guys) to make non-portable guesses about how the IO device works. Or if we should wait for a proper implementation?

Or would it be possible to find middle ground where what is implemented is useful both before and after the io protocol is extended with escape codes?

[josevalim]

We should also consider that, in some cases where we do not have the ANSI functionality, we actually enter a different mode and emit output different values. I know we have something similar in Elixir and a classic example is git which won't show the report when cloning the repository if you are piping to a file.

So I would say being able to query the terminal capabilities is a good thing regardless. Then the debate is if we should support something straight from termcap/win_con for now instead of simply checking for isatty.

[garazdawi]

Good point about making other decisions than control sequences based on the capabilities of the terminal. Though I'm not sure if it would be possible to make any good decisions in portable manner. For instance could an Erlang program running in cmd.exe make any intelligent decisions about whether output is redirected to a file or not?

Also what would such a capability api look like? Maybe a subset of what terminfo looks in verbose format? https://gist.github.com/garazdawi/298fed4e69d5b7dc3a34

[andrewjstone]

My use case is exactly as @josevalim describes. I want to output different information if redirecting to a file. Basically, if I get back false or an enotsup I just assume I can't output colors etc... This seems to be a very common thing to do. I'm not sure I have an opinion on the rest of this. I'm not looking to try to parse escape codes at all. In fact the only thing I would do is possibly add them in by ensuring I'm talking to a tty and checking $TERM. That's very easy code to write.

[garazdawi]

Yes, I understand that a much smaller change would solve the problem for you.

The reason why I'm being a little bit difficult on this is because when someone comes and asks why there are no colors in the windows terminal, I don't want to have to answer that we did not think it was important enough. I do not think that we should add a new feature in that does not support an OS that more than 50% of the computers in the world uses, without doing a thorough investigation into if it is possible to add it there as well.

[ghost]

@garazdawi wrote:

Ideally I do not think there should be any such capabilities check in Erlang at all. Instead we should implement something similar to/exactly like ANSI escape codes into the io protocol. The ttsl driver for windows/unix can then do what it can with that data and use termcap/win_con to draw/detect whatever is possible.

Are you saying we should pass down escape codes as strings and/or come up with erlang term equivalents for well known codes?

We should also add an option which makes it so that Erlang starts without a shell but with the ttysl driver (for usage by escripts for instance), and also of course make it possible to ignore escape sequences (necessary if we are running under run_erl). This would make it possible to implement things such as ^L, color coding etc on all OSs and also io forwarding with colors for things such as -remsh and -slave would just work. We could correctly get colors in the ssh shell, common_test logs etc etc.

These are very useful and important, but they shouldn't block solving @andrewjstone and @josevalim's problem the right way.

However such an implementation is, I believe, quite far away, and the question becomes whether should we implement something small which allows the application writer (i.e. you guys) to make non-portable guesses about how the IO device works. Or if we should wait for a proper implementation?

Why not expose a way to query the terminal's color capabilities and use that instead or together with isatty/1?

Or would it be possible to find middle ground where what is implemented is useful both before and after the io protocol is extended with escape codes?

What incompatibility are you think of here?

[josevalim]

Why not expose a way to query the terminal's color capabilities and use that instead or together with isatty/1?

That may be the way to go. We provide a way to query the color capabilities, which may return something like :ansi or :none. The implementation can start somewhat conservative and expand based on known emulators. Eventually, we can either 1) add :ansi support to the windows driver 2) support one of the ansicon or similar tools on windows 3) or return :win and add specific codes to the windows driver (in case we can't emulate the ansi ones).

Once we support windows somehow, libraries should be able to fill in the gap and transparently provide color support across OSes (like colorama in Python).

[andrewjstone]

@garazdawi I think there was a bit of a disconnect. It is possible for me to implement io:isatty/1 using _isatty on windows. However, I did not realize that the windows shell didn't interpret ansi codes which is why I was so confused about why you would want to go through so much effort to get ansi escape code handling built into erlang rather than have a library implement the codes. After doing a bit of reading on MSDN I realize now that windows only allows color/ncurses like functionality via the console API, although shell wrappers like ansi.sys may work sometimes.

So yes, according to the above you would need to do a lot more work inside erlang to provide color capabilities. The thing is though, this patch is simply about providing io:isatty/1 functionality. I only used color codes in my example, but there may be other legitimate uses for this posix call. As long as the call itself is supported on both *nix and windows, and color support isn't directly provided, I can't really see a reason not to allow it in.

I additionally like @Tuncer and @josevalim's idea of an intermediate step that then returns some sort of terminal type capability (we aren't just talking color here). However, for now application developers on both systems can check $TERM and make a decision that way, defaulting to using no escape codes unless they see an acceptable terminal set.

[garazdawi]

@Tuncer

Ideally I do not think there should be any such capabilities check in Erlang at all. Instead we should implement something similar to/exactly like ANSI escape codes into the io protocol. The ttsl driver for windows/unix can then do what it can with that data and use termcap/win_con to draw/detect whatever is possible.

Are you saying we should pass down escape codes as strings and/or come up with erlang term equivalents for well known codes?

My current vision of what I would like is to send escape codes in the strings which conveniently look just like the ANSI escape codes and then send them to the ttsl driver for interpretation using termcap/win_con, or strip them if we are sending them to the old shell driver.

So for unix with curses io:format("\e[31mhello\e[37m") could end up being something like:

tputs(color[RED],0,outc);
write_buf("hello",5);
tputs(color[WHITE],0,outc);

where color[] would have to be initiated using calls to tgetstr("Sf" | "AF",&af) and friends. It would of course be very nice if we could just pass on the escape codes to the printing terminal without working with curses, but I do not think it is possible to make work everywhere? There are so many different varieties of terminals that just doing some cursory reading about this stuff makes my head spin...

[garazdawi]

@josevalim

Why not expose a way to query the terminal's color capabilities and use that instead or together with isatty/1?

That may be the way to go. We provide a way to query the color capabilities, which may return something like :ansi or :none. The implementation can start somewhat conservative and expand based on known emulators. Eventually, we can either 1) add :ansi support to the windows driver 2) support one of the ansicon or similar tools on windows 3) or return :win and add specific codes to the windows driver (in case we can't emulate the ansi ones).

Once we support windows somehow, libraries should be able to fill in the gap and transparently provide color support across OSes (like colorama in Python).

Any ideas on how we detect that something supports ansi? and if so that it supports it in the same manner on different terminal types? For instance one of our SunOS 10 machine report TERM=xterm, isatty = true, but terminfo does not report color support. So to be cross-platform we have to report the color status of terminfo. However \e[1m aka bold does work, so should we report that as well? Very soon we have the entire terminfo string to report. Which might not be a bad thing? We'd have to create such a string for windows+platforms without termcap, which might be doable.

Also does anyone know if the setaf capability always is \E[3%p1%dm? I'm assuming that it can be something else as well?

[garazdawi]

@andrewjstone

So yes, according to the above you would need to do a lot more work inside erlang to provide color capabilities. The thing is though, this patch is simply about providing io:isatty/1 functionality. I only used color codes in my example, but there may be other legitimate uses for this posix call. As long as the call itself is supported on both *nix and windows, and color support isn't directly provided, I can't really see a reason not to allow it in.

My concern is that querying isatty gives you very little useful information, other than are we currently redirecting to a file or not. It tells you nothing about which escape sequences a terminal supports or not. And as I said to @josevalim above, neither does checking $TERM. If we want to allow users to make decisions about what escape sequences work we should try to solve that problem somehow.

This might be by implementing a limited number of escape sequences into the io protocol and then making sure that they work cross platform. Or it could be by allowing the user to query the terminfo capabilities of the various IODevices. I'm quite new to all this io protocol/termals stuff so I don't really know which way is the best to go. Other people here at OTP who know more than me seem to think implementing escape sequences into the io protocol is that way to go, but personally I don't know.

[andrewjstone]

@garazdawi

My concern is that querying isatty gives you very little useful information, other than are we currently redirecting to a file or not. It tells you nothing about which escape sequences a terminal supports or not. And as I said to @josevalim above, neither does checking $TERM. If we want to allow users to make decisions about what escape sequences work we should try to solve that problem somehow.

That's reasonable I suppose. I understand the conservativeness of the OTP team in getting things merged in. For now riak will have to live without colors I guess. It's not a big deal, but I didn't think the original BIF was either ;)

I'm also relatively unfamiliar with the intricacies of terminals, but I have used ansi escape codes to do fun things in the past in other languages and was hoping that Erlang could provide the same in a simple manner. I guess it will have to wait until someone gets around to implementing a full IO protocol. Either way I'm about done here.

[ghost]

@andrewjstone, nothing prevents us from merging a version of your isatty/1 patch, as it's independent of all the terminal capability stuff we talked about. isatty/1 can be used for other things than the imprecise color support check, so I don't see why we should not add it.

Once we have the discussed extended API, you can rewrite your code to correctly check for color support, but it should be sufficient to make the file redirection decision.

@garazdawi maybe the OTP team would be more welcoming if isatty/1 was added as a preliminary (i.e. experimental) function until the other stuff is figured out.

[ghost]

@garazdawi from what I can gather, querying terminfo appears to be the proper way to check capabilities. I'm not sure, but maybe we also have to deal with reported capabilties actually being unsupported.

Also, it seems reasonable to me to provide convenience abstractions for well known codes and let the implementation return {error, enotsup} or optionally ignore/discard if it's unsupported.

[garazdawi]

@andrewjstone I hope I have deterred you too much away from submitting patched to us. Unknowingly you stepped into one of the more complex cross platform areas where Erlang does things quite differently from other programming environments.

isatty/1 can be used for other things than the imprecise color support check, so I don't see why we should not add it.
@garazdawi maybe the OTP team would be more welcoming if isatty/1 was added as a preliminary (i.e. experimental) function until the other stuff is figured out.

@Tuncer
The only check that I can think of that you can do with isatty is checking if the IODevice definitely does not support any escape sequences. Getting true back does (afaik) not give you any details about how you can communicate with it.

So in conclusion I do not think we want to have isatty in Erlang/OTP. It does not make sense in a cross platform environment and only helps you on some systems. If you want to make it just work on in your specific environment I suggest using something like https://github.com/mazenharake/cecho to manage the output.

If someone wants to attempt to implement escape code support into the io protocol, I'd be happy to guide them on the way. The next step should be to write a mini eep describing the changes to the io protocol+other api's and how that would be implemented on windows/termcap/non-termcap systems.

I'm closing this PR now, if you want to continue the discussion please do so on the erlang-questions mailing list.

[andrewjstone]

I don't want an ncurses api right now. Even if I did, it still doesn't
solve the problem of the user redirecting the terminal to a file. I guess
I'd have to patch that library, add a NIF or use a custom build. I'm also
not interested in adding escape support to the i/o protocol. It's a lot of
work for minimal gain IMO.

I must admit that I am slightly deterred from submitting patches to the OTP
team and wonder if our priorities differ in some important areas. However,
I love Erlang and still believe it's the best production environment out
there for writing distributed systems. If I have a patch in the future I'll
gladly submit it as I want everyone to benefit from the usefulness of the
patch. I just hope that we don't conflict on more important and substantive
things.

On Mon, Oct 6, 2014 at 11:35 AM, Lukas Larsson notifications@github.com
wrote:

@andrewjstone https://github.com/andrewjstone I hope I have deterred
you too much away from submitting patched to us. Unknowingly you stepped
into one of the more complex cross platform areas where Erlang does things
quite differently from other programming environments.

isatty/1 can be used for other things than the imprecise color support
check, so I don't see why we should not add it.
@garazdawi https://github.com/garazdawi maybe the OTP team would be
more welcoming if isatty/1 was added as a preliminary (i.e. experimental)
function until the other stuff is figured out.

@Tuncer https://github.com/tuncer
The only check that I can think of that you can do with isatty is checking
if the IODevice definitely does not support any escape sequences. Getting
true back does (afaik) not give you any details about how you can
communicate with it.

So in conclusion I do not think we want to have isatty in Erlang/OTP. It
does not make sense in a cross platform environment and only helps you on
some systems. If you want to make it just work on in your specific
environment I suggest using something like
https://github.com/mazenharake/cecho to manage the output.

If someone wants to attempt to implement escape code support into the io
protocol, I'd be happy to guide them on the way. The next step should be to
write a mini eep describing the changes to the io protocol+other api's and
how that would be implemented on windows/termcap/non-termcap systems.

I'm closing this PR now, if you want to continue the discussion please do
so on the erlang-questions mailing list.

—
Reply to this email directly or view it on GitHub
#480 (comment).