Doesn't work on FreeBSD

[Parakleta]

I cannot get this to work on FreeBSD. I'm not sure what it is expecting to find but I know FreeBSD does use a terminfo database which a comment in searcher.rs says it doesn't support. Is there a way to add support for the terminfo database, or else can I point it to some other files to convince it to work, or could there be some default fallback behaviour (assume VT100?) ?

[Stebalien]

Where does it state this? Regardless, I've pushed a new branch that should work on FreeBSD unless the format is significantly different (it shouldn't be). Does it?

[Parakleta]

The comment is in searcher.rs:13. The issue is that the BSDs use the Berkeley DB variant, rather than the split directories method. I've done some more research and unfortunately while NCurses is able to read both formats, it's only able to write the format is was compiled for. This means there is no easy way on BSD to generate the split directories files.

The options then seem to be either call out to infocmp to extract the textual representation, or else load the NCurses library to decode the information for you, or else implement (or load) a Berkeley DB file loader which is capable of reading version 1.85 files, and extract the binary representation from the terminfo.db file (the format you are currently parsing is stored in the .db file).

Regarding your branch, FreeBSD stores the file in /usr/local/share/misc/terminfo.db (note the added local, and trailing .db) which is correctly set in the TERMINFO environment variable.

[Stebalien]

I see. The man page (https://www.freebsd.org/cgi/man.cgi?query=terminfo&sektion=5) disagrees. Can you point me to a webpage that describes the database?

[Stebalien]

Oh wait, are you're referring to termcap?

[Stebalien]

Could you send me your compiled termcap.db?

[Parakleta]

That man page link is interesting, the man page on my system is for "ncurses version 5.9 (patch 20141213)" (described at the end of the first paragraph). Ah, I've installed the ncurses port, the man page I have been looking at is the ports terminfo. The other relevant man page is ports term, with the section detailing the hashed database.

At that location there are termcap and termcap.db files. The termcap file is also available at /etc/termcap. The .db file is again Berkeley DB format and contains the de-aliased version of the regular termcap file. With the ncurses port installed I have the terminfo.db file which contains the binary format. Bleh, it's all a bit of a mess.

A version of the termcap file is available in freebsd/etc/termcap.small. Mine has more entries, but is otherwise the same format. This format (and the associated .db) seem to be the only files that can be guaranteed to exist (no terminfo without ncurses port).

If you need the binary file I'll need to figure out a method to get it to you, GitHub doesn't seem to like it.

[Stebalien]

Yes, I need the binary. You can email it to me (steven@stebalien.com) or put a base64 encoded version in a comment.

[Parakleta]

Sent as email.

ETA: I just noticed I missed a flag when tar-ing so they're not compressed, but they should still unpack fine.

ETA2: To inspect the .db files I've found the command db_dump185 -p term*.db to be helpful.

[Stebalien]

Unfortunately, my copy of Berkeley db (linux) can't seem to read the database. At this point, I'm going to have to say patches welcome, sorry.

[Stebalien]

Actually, I'm going to leave this open for anyone else to work on.

[Parakleta]

I think maybe the only reliable way to do this cross platform would be to consume each platform's interface to these files. The two options I can see are to either use the termcap library or the tput shell command.

Do you have any thoughts on moving in either of these directions?

The only other options I can think of would be to diverge for each platform's format (i.e. hook in a Berkeley DB 1.85 library/implementation for BSDs), or else target the lowest common denominator and start parsing the /etc/termcap file with associated tc entries. These options seem like too much work to me.

[Stebalien]

Calling another program is out of the question. Personally, I'd diverge. On FreeBSD, I'd use berkeley DB to parse the database. On my distro (arch), termcap isn't even packaged anymore; you're expected to use ncurses. I don't want to depend on ncurses and, if rustc ever updates its version of term, they definitely won't be happy depending on ncurses.

[Parakleta]

Well you say you don't want to depend on ncurses but then you're relying on a data file that is installed as part of ncurses. If you try and use the terminfo file on FreeBSD your program will fail unless the user has installed ncurses. Using term.h is then the closest I think you can get to cross platform.

If you do want to truly break the requirement for ncurses or related, I think you should package the terminfo files yourself so you make no assumptions about what is available on the target system.

ETA: You could then pre-parse this file into whatever format you like.

[Stebalien]

Not all terminfo files are shipped with ncurses (e.g. my terminal, termite) but you raise a good point. However, as-is, it's a dependency of convenience; any project wishing to use term without depending on ncurses can ship the databases separately as you noted. For example, I expect redox will eventually do this if they ever try to get rustc running on redox.

Your original suggestion of implementing a minimal VT100 fallback is sounding better and better (although I'd eventually like to fully support FreeBSD).

[Stebalien]

@alexcrichton, as original maintainer of term, thoughts?

(also, FYI, ubuntu/debian ship the terminfo files in separate ncurses-base, ncurses-term packages).

[alexcrichton]

Ah sorry I never knew much about the design of term or how it was implemented, I was basically just making sure it kept compiling. As a result I may not be much help here :(

[Stebalien]

@pcwalton?

[pcwalton]

I don't think I've ever worked on this crate.

[Stebalien]

Ah, you started term (technically) but not the terminfo crate. Last try... @cmr?

[Stebalien]

So, I've recently read some of the ncurses code (to make sure we were doing some things correctly) and have decided that term will not depend on ncurses. ncurses has suffered from quite a bit of bit rot.

[emberian]

Sorry, must have missed the email. Yes, I'm responsible for this code. We should not link to native libraries or anything else silly like that, we should just implement bdb. I can work on this some.

[Stebalien]

Can you find documentation for bdb 1.85? I can't find anything describing the format.

[emberian]

No, I planned on inspecting the source from the original BSD-licensed release and seeing if the current format is still compatible with that.

[emberian]

In any case, it will be a fair amount of effort to reverse engineer and implement even just a Berkeley db reader. A conditional dependency on ncurses for only FreeBSD (or other platforms) wouldn't be the worst thing ever.

The problem with linking directly against modern versions of db is that they are LGPL, which would contanimate any users of libterm on only those platforms.

[Parakleta]

One solution is to just implement a parser for the termcap textual format file because this is available to everyone. It's a bit of effort to handle the cross references, but surely easier than implementing BDB.

Regarding licensing and linking, FreeBSD core has a BDB implementation which will be BSD or MIT licensed. I think in fact that's the reason their BDB files are stuck at v1.85 which isn't forward compatible and so difficult to read on other platforms - it may be the latest version without licensing issues. Related to this is that linking against modern versions of BDB I don't think will help because I don't think it can actually read the old format.

The dependency on NCurses shouldn't be a problem at least for FreeBSD given that it is included in 'core', if that's the direction you choose to go (so long as you stick to whichever outdated version they're shipping). There's also just the term.h component of Curses which is in the Opengroup standards so should be reasonably cross platform (maybe?).

[emberian]

@Parakleta interesting! I didn't know that the textual termcap files were still distributed (and I never looked). Where is that database maintained/who maintains it?

[Parakleta]

As far as I can tell the NCurses project is the current owner of the files. FreeBSD maintains its own minimal set however, which other systems may also do. The link from this previous comment points to a discussion and some links to what is claimed to be the most current termcap file.

If you do want to truly break the requirement for ncurses or related, I think you should package the terminfo files yourself so you make no assumptions about what is available on the target system.

[Stebalien]

Unfortunately, it's not that simple. Term is a statically linked library so we have literally no control over packaging (although we could compile in the database but that's a terrible idea). Ideally, we could convince the FreeBSD devs to create a standard terminfo port and have all rust programs that depend on term depend on that.

[Parakleta]

I don't know what you mean by "standard". Linux doesn't imply "standard" and if fact often the opposite is true. FreeBSD comes with terminfo and termcap files pre-installed, together with a minimal copy of NCurses. They supply textual terminal definitions, and a hashed database of binary definitions as per the NCurses specification. It just so happens that NCurses accepts two mutually exclusive binary formats, one which is a single database file, and the other which is a bunch of files, one per definition.

There are things in the OpenGroup standards about curses which you could try to use as a baseline which I suggested before, such as term.h. Technically it is flagged for deletion from the standard but I don't know if it has been yet and it should at least be available on every system with some curses variant installed (which is already a dependency for you).

You say compiling it in is a terrible idea but how many options do you currently use from the terminfo database anyway. Why not provide your own binary file with just the options you need, and a utility which takes the output of infocmp to add an entry to your binary file?

I think you're stuck, you're depending on an optional binary file (in that one of two formats can be chosen) in an optional package you are not depending on and that you are not using to access the binary file. This I think is a mistake. If you want to use the binary format you should be using the API in term.h to access it, otherwise you should use the textual format.

You could skip the parsing of macros in the textual termcap/terminfo file if it's a problem by requiring the user to provide the parsed version such as comes from infocmp.

I was just quoting the previous message to provide the link for @cmr, not to re-propose the idea of packaging the files (although I don't think it is that bad an idea).

[Stebalien]

I don't know what you mean by "standard". Linux doesn't imply "standard" and if fact often the opposite is true.

By standard I mean what both Mac OSX and Linux do and what FreeBSD used to do until 2006 (when they switched to termcap and then to a hashed terminfo database, see https://svnweb.freebsd.org/ports/head/devel/ncurses/Makefile?view=log). Even the online FreeBSD ncurses manual pages still reference the multi file database.

You say compiling it in is a terrible idea but how many options do you currently use from the terminfo database anyway. Why not provide your own binary file with just the options you need, and a utility which takes the output of infocmp to add an entry to your binary file?

Because not all terminal descriptions are included in the ncurses database, entries could not be added without recompiling every program, and every program would need to carry this information with it.

I think you're stuck, you're depending on an optional binary file (in that one of two formats can be chosen) in an optional package you are not depending on and that you are not using to access the binary file. This I think is a mistake.

We depend on the terminfo databases being installed; any package using term must depend on some package providing these files. That doesn't need to be ncurses. On Ubuntu, this is ncurses-term, and in MinGW it's called mingw32-terminfo.

Furthermore, I explicitly do not want to depend on the ncurses library itself because it's an unmaintainable mess of legacy code.

If you want to use the binary format you should be using the API in term.h to access it, otherwise you should use the textual format.

On my machine (and I'm guessing that this applies to most non FreeBSD machines), I don't even have the textual version installed so that isn't a global solution. As for term.h, this uses ncurses and again, I would like to steer as far clear of ncurses as possible.

[Parakleta]

term.h isn't strictly part of NCurses, it should be part of any curses implementation as defined at OpenGroup.

On any system that provides the compiled termcap/terminfo database there should be the utility infocmp which can be used to extract the textual format. I had just assumed that the /etc/termcap file was always available, but if this isn't the case on other systems and you don't want to use the tgetent() function or the tput or infocmp utilities I guess the only other option is to support both the directory tree binary format and the ascii format and try and detect the file format and add to the search locations.

I don't know. Anything cross-platform makes my head hurt. My approach is always to either call out to some common interface which I (assume I) can trust or else push down to the lowest common denominator, I'm not a fan of trying to maintain multiple implementations. I had assumed the ascii format was available everywhere and so that would be the lowest common denominator, but without that any solution is going to be messy.

At the very least I think you must add the parser for the ascii format because this will make work-arounds possible. Even without macro support, once that exists I can call infocmp and capture the output to a file and then set my TERMCAP variable to point to it. Clumsy, but working.

[alexreg]

I take it this still hasn't been resolved?

[Stebalien]

@alexreg not yet, sorry. My plan is to include built-in support for ANSI terminals and a whitelist of known ANSI-compatible terminal emulators. However, to do that correctly, term needs to be rearchitected from scratch (something I've started several times but never finished).

[alexreg]

Ah, I see. No worries. Do kindly let me know if you get around to it though.

On 22 Nov 2016, at 16:18, Steven Allen notifications@github.com wrote:

@alexreg https://github.com/alexreg not yet, sorry. My plan is to include built-in support for ANSI terminals and a whitelist of known ANSI-compatible terminal emulators. However, to do that correctly, term needs to be rearchitected from scratch (something I've started several times but never finished).

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub #44 (comment), or mute the thread https://github.com/notifications/unsubscribe-auth/AAEF3DDTPyDuFF0nDWP0LHO9i42OSt26ks5rAxXhgaJpZM4GfMkY.

[alexreg]

I presume you're aware of this crate? https://crates.io/crates/ansi_term

[asomers]

Reading the .db file is easy using dbopen, which is built into FreeBSD's libc. I got that working. The next problem is that termcap.db contains the terminal's textual representation, whereas term expects the compiled representation used by terminfo. So src/terminfo/parser needs to be extended to handle that. I don't know how to read either the compiled or text representation. Would anybody else be able to write the parser if I share my dbopen code?

[asomers]

My WIP is here: https://github.com/asomers/term/tree/termcap . Reading the db file is done, and it works well enough that I can set bold text and reset the terminal. But I get stack underflow errors when trying to expand a command to set the foreground color. There's probably some other difference between the terminfo and termcap formats that I'm not noticing. If anybody wants to give it a shot, the command to run is cargo run --example prterm. If everything is working, that should exercise all of the terminal commands used by rustup.