This reveals two bugs in isql or actualy in the editline library that is included in the Firebird sources:
1. Consuming all available memory. The libary tries to read the terminal's size. It gets some random values - negative or an incredibly huge numbers. The negative numbers are corrected in extern/editline/src/term.c:term_change_size(), however the huge positive numbers are not which leads to an attempt of allocating a huge amount of memory in term_alloc_display(). This way isql hangs and often makes system unresponsive (swap allocation).
2. Segmentation fault. When isql finally eats up all the memory and it cannot get more, in two places return codes are not validated. The library will not detect that the memory was not allocated and will try to dereference a null pointer.
The second problem is easy to fix, however the first one needs to be fixed in Solaris. For Firebird I can think only of a workaround which will detect insane terminal sizes and correct them just as it corrects negative sizes.
I'm far not sure it's worth fixing host-OS bugs if we can easily avoid the, specially when this is client utility, not server. The simplest way is to build with switch --without-editiline for this OS. Or add a notice in readme that isql should not be used in such environment. Or use SUN's suggested workaround - "use the stty command to set the correct number of rows and columns once you login with zlogin".
You've made a valid point, Alex. However that doesn't change the fact that editline has a bug - it allows an abnormal memory consumption that could kill a system. In this case it's a Solaris bug, revealed under certain conditions. But it could happen everywhere, googling I could find many reports of invalid terminal sizes returned by ioctl TIOCSWINSZ. The values are already checked against negatives, so why not check them also against large numbers? The effort required is minimal, changes are simple and there is no chance of breaking anything. My opinion is that OS bugs are no excuse for misbehaving software if you can easily make it work better.