notcurses-info loses information in the scrollback on some terminals

[j4james]

• LANG="C.UTF-8"
• TERM="xterm-256color"
• notcurses version 2.3.17 (built from commit 84b72e1)
• GNOME Terminal 3.38.1 (VTE 0.62.3)

Steps to reproduce:

1. Set the terminal window size to 80x24.
2. Run notcurses-info twice in a row.
3. Scroll up to see the results of the first run which have now scrolled offscreen.
4. Note that part of the output is missing (see screenshot below).

From what I can make out, notcurses-info seems to be using the SU escape sequence for something, which is problematic. While some terminals will pan the viewport down, in the same way as a linefeed at the bottom of the page, some will merely scroll the content of the buffer, which has the same effect as deleting lines from the top of the window (thus losing content you might expect to see in the scrollback).

I'm assuming you want the viewport to pan down, which is probably the most common interpretation in Linux terminals, but both Konsole and VTE have gone with the buffer scrolling interpretation instead. So unless you can at least persuade VTE to change their behaviour, it may be a good idea to avoid scrolling like this.

[dankamongmen]

i'm not able to reproduce this in the exact manner you've specified, but i know this issue, or something close to it, exists. oh wait nevermind, that was kitty. let me get an 80x24 xterm...hrmm, same results. i have all expected output in my scrollback at 80x24 on XTerm 368:

[schwarzgerat](130) $ ./notcurses-info
notcurses 2.3.17 on XTerm 368
24 rows (23px) 80 cols (11px) 552x880 rgb+256 colors
gcc-10.3.0 (LE)
terminfo from ncurses 6.2.20201114 zlib 1.2.11
avformat 58.76.100 avutil 56.70.100 swscale 5.9.100 avcodec 58.134.100
af+ ab+ sum- cup+ vpa+ hpa+ sgr0+ op+ fgop+ bgop+ bce+                          
bold+ ital+ struck+ ucurl- uline+ u7+ ccc+ rgb+ gpm-                            
utf8+ half+ quad- sex- braille+ images+ video+ indn+                            
defbg 0x333333 not considered transparent             🯁🯂🯃https://notcurses.com  
max sixel size: 529x880 colorregs: 256                                          
 ▘▝▀▖▌▞▛▗▚▐▜▄▙▟█⎧ 🬀🬁🬂🬃🬄🬅🬆🬇🬈🬊🬋🬌🬍🬎🬏🬐🬑🬒🬓▌🬔🬕🬖🬗🬘🬙🬚🬛🬜🬝⎫♠♥🯰🯱🯲🯳🯴🯵🯶🯷🯸🯹⅗⅘⅙⅚⅛⎧▕▏⎫┌╥─╥─╥┐🭩⎛⎞
╲╿╱ ◨◧ ◪◩ ◖◗ ⫷⫸ ⎩🬟🬠🬡🬢🬣🬤🬥🬦🬧▐🬨🬩🬪🬫🬬🬭🬮🬯🬰🬱🬲🬳🬴🬵🬶🬷🬸🬹🬺🬻█⎭♦♣¼½¾⅐⅑⅒⅓⅔⅕⅖⅜⅝⅞⅟↉⎪🮇▎⎪├╜╓╫╖╙┤🭫⎜⎟
╾╳╼ ◲◱ ◶◵ 🮣🮠 🮤🮥◜◝ ◿◺ 🮞🮟 ◢◣ ┌┐─  ┏┓━ ╭╮─  ╔╗═ 🭽🭾▁ ⩘▵△▹▷▿▽◃◁⭡⭣⭠⭢⭧⭩⭦⭨⎪🮈▍⎪├─╨╫╨─┤┇⎜⎟
╱╽╲ ◳◰ ◷◴ 🮡🮢 🮦🮧◟◞ ◹◸ 🮝🮜 ◥◤ └┘│  ┗┛┃ ╰╯│  ╚╝║ 🭼🭿🭵 ⩗▴⏶⯅▲▸⏵⯈▶▾⏷⯆▼◂⏴⯇◀⎪▐▌⎪╞═╤╬╤═╡┋⎜⎟
⎡⠀⠁⠈⠉⠂⠃⠊⠋⠐⠑⠘⠙⠒⠓⠚⠛⠄⠅⠌⠍⠆⠇⠎⠏⠔⠕⠜⠝⠖⠗⠞⠟⠠⠡⠨⠩⠢⠣⠪⠫⠰⠱⠸⠹⠲⠳⠺⠻⠤⠥⠬⠭⠦⠧⠮⠯⠴⠵⠼⠽⠶⠷⠾⠿⎤⎨🮉▋⎬╞╕╘╬╛╒╡┊⎜⎟
⎢⡀⡁⡈⡉⡂⡃⡊⡋⡐⡑⡘⡙⡒⡓⡚⡛⡄⡅⡌⡍⡆⡇⡎⡏⡔⡕⡜⡝⡖⡗⡞⡟⡠⡡⡨⡩⡢⡣⡪⡫⡰⡱⡸⡹⡲⡳⡺⡻⡤⡥⡬⡭⡦⡧⡮⡯⡴⡵⡼⡽⡶⡷⡾⡿⎥⎪🮊▊⎪└┴─╨─┴┘╏⎝⎠
⎢⢀⢁⢈⢉⢂⢃⢊⢋⢐⢑⢘⢙⢒⢓⢚⢛⢄⢅⢌⢍⢆⢇⢎⢏⢔⢕⢜⢝⢖⢗⢞⢟⢠⢡⢨⢩⢢⢣⢪⢫⢰⢱⢸⢹⢲⢳⢺⢻⢤⢥⢬⢭⢦⢧⢮⢯⢴⢵⢼⢽⢶⢷⢾⢿⎥⎪🮋▉⎪╭──╮⟬⟭╔╗≶≷
⎣⣀⣁⣈⣉⣂⣃⣊⣋⣐⣑⣘⣙⣒⣓⣚⣛⣄⣅⣌⣍⣆⣇⣎⣏⣔⣕⣜⣝⣖⣗⣞⣟⣠⣡⣨⣩⣢⣣⣪⣫⣰⣱⣸⣹⣲⣳⣺⣻⣤⣥⣬⣭⣦⣧⣮⣯⣴⣵⣼⣽⣶⣷⣾⣿⎦⎪██⎪│╭╮│╔═╝║⊆⊇
 ▔🭶🭷🭸🭹🭺🭻▁ 🭁🭌 🭂🭍 🭃🭎 🭄🭏 🭅🭐 🭆🭑 🭇🬼 🭈🬽 🭉🬾 🭊🬿 🭋🭀 ₀₁₂₃₄₅₆₇₈₉ ⎛ ▁▂▃▄▅▆▇█ ⎞⎪🭨🭪⎪╰╯││║╔═╝⊴⊵
 ▏🭰🭱🭲🭳🭴🭵▕ 🭒🭝 🭓🭞 🭔🭟 🭕🭠 🭖🭡 🭧🭜 🭢🭗 🭣🭘 🭤🭙 🭥🭚 🭦🭛 ⁰¹²³⁴⁵⁶⁷⁸⁹ ⎝ ▔🮂🮃▀🮄🮅🮆█ ⎠⎩🭪🭨⎭⧒⧑╰╯╚╝❨❩⟃⟄
👾🏴🤘🚬🌍🌎🌏🥆💣🗡🔫⚗️⚛️☢️☣️🌿🎱🏧💉💊🕴️📡🤻🦑🇮🇱🇦🇶👩‍  🪤🚱✊  🔬🧬🏴‍ 🤽               
1 render, 55.67µs (55.67µs min, 55.67µs avg, 55.67µs max)
1 raster, 15.75µs (15.75µs min, 15.75µs avg, 15.75µs max)
1 write, 205.49µs (205.49µs min, 205.49µs avg, 205.49µs max)
23.58KiB (23.58KiB min, 23.58KiB avg, 23.58KiB max) 0 inputs
0 failed renders, 0 failed rasters, 0 refreshes, 0 input errors
RGB emits:elides: def 2:78 fg 2:1174 bg 1178:78
Cell emits:elides: 1256:2528 (66.81%) 97.50% 99.83% 6.21%
Bitmap emits:elides: 0:0 (0.00%) 0B (0.00%) SuM: 0 (0.00%)
[schwarzgerat](0) $ ./notcurses-info 
notcurses 2.3.17 on XTerm 368
24 rows (23px) 80 cols (11px) 552x880 rgb+256 colors
gcc-10.3.0 (LE)
terminfo from ncurses 6.2.20201114 zlib 1.2.11
avformat 58.76.100 avutil 56.70.100 swscale 5.9.100 avcodec 58.134.100
af+ ab+ sum- cup+ vpa+ hpa+ sgr0+ op+ fgop+ bgop+ bce+                          
bold+ ital+ struck+ ucurl- uline+ u7+ ccc+ rgb+ gpm-                            
utf8+ half+ quad- sex- braille+ images+ video+ indn+                            
defbg 0x333333 not considered transparent             🯁🯂🯃https://notcurses.com  
max sixel size: 529x880 colorregs: 256                                          
 ▘▝▀▖▌▞▛▗▚▐▜▄▙▟█⎧ 🬀🬁🬂🬃🬄🬅🬆🬇🬈🬊🬋🬌🬍🬎🬏🬐🬑🬒🬓▌🬔🬕🬖🬗🬘🬙🬚🬛🬜🬝⎫♠♥🯰🯱🯲🯳🯴🯵🯶🯷🯸🯹⅗⅘⅙⅚⅛⎧▕▏⎫┌╥─╥─╥┐🭩⎛⎞
╲╿╱ ◨◧ ◪◩ ◖◗ ⫷⫸ ⎩🬟🬠🬡🬢🬣🬤🬥🬦🬧▐🬨🬩🬪🬫🬬🬭🬮🬯🬰🬱🬲🬳🬴🬵🬶🬷🬸🬹🬺🬻█⎭♦♣¼½¾⅐⅑⅒⅓⅔⅕⅖⅜⅝⅞⅟↉⎪🮇▎⎪├╜╓╫╖╙┤🭫⎜⎟
╾╳╼ ◲◱ ◶◵ 🮣🮠 🮤🮥◜◝ ◿◺ 🮞🮟 ◢◣ ┌┐─  ┏┓━ ╭╮─  ╔╗═ 🭽🭾▁ ⩘▵△▹▷▿▽◃◁⭡⭣⭠⭢⭧⭩⭦⭨⎪🮈▍⎪├─╨╫╨─┤┇⎜⎟
╱╽╲ ◳◰ ◷◴ 🮡🮢 🮦🮧◟◞ ◹◸ 🮝🮜 ◥◤ └┘│  ┗┛┃ ╰╯│  ╚╝║ 🭼🭿🭵 ⩗▴⏶⯅▲▸⏵⯈▶▾⏷⯆▼◂⏴⯇◀⎪▐▌⎪╞═╤╬╤═╡┋⎜⎟
⎡⠀⠁⠈⠉⠂⠃⠊⠋⠐⠑⠘⠙⠒⠓⠚⠛⠄⠅⠌⠍⠆⠇⠎⠏⠔⠕⠜⠝⠖⠗⠞⠟⠠⠡⠨⠩⠢⠣⠪⠫⠰⠱⠸⠹⠲⠳⠺⠻⠤⠥⠬⠭⠦⠧⠮⠯⠴⠵⠼⠽⠶⠷⠾⠿⎤⎨🮉▋⎬╞╕╘╬╛╒╡┊⎜⎟
⎢⡀⡁⡈⡉⡂⡃⡊⡋⡐⡑⡘⡙⡒⡓⡚⡛⡄⡅⡌⡍⡆⡇⡎⡏⡔⡕⡜⡝⡖⡗⡞⡟⡠⡡⡨⡩⡢⡣⡪⡫⡰⡱⡸⡹⡲⡳⡺⡻⡤⡥⡬⡭⡦⡧⡮⡯⡴⡵⡼⡽⡶⡷⡾⡿⎥⎪🮊▊⎪└┴─╨─┴┘╏⎝⎠
⎢⢀⢁⢈⢉⢂⢃⢊⢋⢐⢑⢘⢙⢒⢓⢚⢛⢄⢅⢌⢍⢆⢇⢎⢏⢔⢕⢜⢝⢖⢗⢞⢟⢠⢡⢨⢩⢢⢣⢪⢫⢰⢱⢸⢹⢲⢳⢺⢻⢤⢥⢬⢭⢦⢧⢮⢯⢴⢵⢼⢽⢶⢷⢾⢿⎥⎪🮋▉⎪╭──╮⟬⟭╔╗≶≷
⎣⣀⣁⣈⣉⣂⣃⣊⣋⣐⣑⣘⣙⣒⣓⣚⣛⣄⣅⣌⣍⣆⣇⣎⣏⣔⣕⣜⣝⣖⣗⣞⣟⣠⣡⣨⣩⣢⣣⣪⣫⣰⣱⣸⣹⣲⣳⣺⣻⣤⣥⣬⣭⣦⣧⣮⣯⣴⣵⣼⣽⣶⣷⣾⣿⎦⎪██⎪│╭╮│╔═╝║⊆⊇
 ▔🭶🭷🭸🭹🭺🭻▁ 🭁🭌 🭂🭍 🭃🭎 🭄🭏 🭅🭐 🭆🭑 🭇🬼 🭈🬽 🭉🬾 🭊🬿 🭋🭀 ₀₁₂₃₄₅₆₇₈₉ ⎛ ▁▂▃▄▅▆▇█ ⎞⎪🭨🭪⎪╰╯││║╔═╝⊴⊵
 ▏🭰🭱🭲🭳🭴🭵▕ 🭒🭝 🭓🭞 🭔🭟 🭕🭠 🭖🭡 🭧🭜 🭢🭗 🭣🭘 🭤🭙 🭥🭚 🭦🭛 ⁰¹²³⁴⁵⁶⁷⁸⁹ ⎝ ▔🮂🮃▀🮄🮅🮆█ ⎠⎩🭪🭨⎭⧒⧑╰╯╚╝❨❩⟃⟄
👾🏴🤘🚬🌍🌎🌏🥆💣🗡🔫⚗️⚛️☢️☣️🌿🎱🏧💉💊🕴️📡🤻🦑🇮🇱🇦🇶👩‍  🪤🚱✊  🔬🧬🏴‍ 🤽               
1 render, 56.80µs (56.80µs min, 56.80µs avg, 56.80µs max)
1 raster, 15.91µs (15.91µs min, 15.91µs avg, 15.91µs max)
1 write, 8.11ms (8.11ms min, 8.11ms avg, 8.11ms max)
23.59KiB (23.59KiB min, 23.59KiB avg, 23.59KiB max) 0 inputs
0 failed renders, 0 failed rasters, 0 refreshes, 0 input errors
RGB emits:elides: def 3:78 fg 2:1174 bg 1178:78
Cell emits:elides: 1256:2528 (66.81%) 96.30% 99.83% 6.21%
Bitmap emits:elides: 0:0 (0.00%) 0B (0.00%) SuM: 0 (0.00%)
[schwarzgerat](0) $ 

i know about a possibly-related issue, however, which is that output scrolled out of the visual region between renders is never actually emitted to the screen. i'm not convinced that's an actual bug, but it probably is (this is along the lines of XTerm faithfully displaying all output when presented with a great amount of it, whereas other terminals will skip some (see the fastScroll Xresource and this link https://lwn.net/Articles/751763/)).

i don't think that would lead to the situation you're seeing, though.

in fact, i'm not sure what could. notcurses-info ought generate 29 lines of output. of those, 7 are the closing stats, emitted to stderr. the other 22 are emitted to stdout, with a notcurses_render() between the two. so with 24 rows, no matter where you start the program, you ought get all 22 rows emitted, ending on either the 22nd, 23rd, or 24th row (depending on where you started). i then dump the 7 lines of the banner to stderr, which will force a scroll on any 24-row terminal. they ought then be in the scrollback buffer, and so far as i know, i have no interactions with said buffer.

it looks like the lost text is from the first run. what happens if you run notcurses-info, and then run some other scroll-generating program, dmesg | tail -n 29 or something like that? do you lose the material in the same fashion? i'm guessing not, suggesting the problem to be in something the second run is doing, when there's already material present.

thanks for this report. i expect it will require some investigation.

[j4james]

In retrospect, you're probably just using the indn capability, and given that ind is defined as LF, it seems personally reasonable to expect indn to work in an equivalent way. So unless VTE and Konsole define their own terminfo differently (which doesn't seem to be the case), I think there's a good argument to be made that their current behavior is incorrect. Feel free to close this as an external issue.

[j4james]

i have all expected output in my scrollback at 80x24 on XTerm 368:

Yeah, XTerm doesn't have the problem - it's only on Konsole and VTE-based terminals like Gnome Terminal.

[j4james]

it looks like the lost text is from the first run. what happens if you run notcurses-info, and then run some other scroll-generating program, dmesg | tail -n 29 or something like that?

That doesn't trigger it. Assumedly because it's just scrolling with a linefeed rather than an indn or SU sequence.

An easy way to tell whether a terminal will be affected is with a command like this:

for i in {1..100}; do echo $i; done; tput indn 10

If you scroll up and see a break in the sequence, you know the terminal has this bug.

[dankamongmen]

In retrospect, you're probably just using the indn capability, and given that ind is defined as LF, it seems personally reasonable to expect indn to work in an equivalent way. So unless VTE and Konsole define their own terminfo differently (which doesn't seem to be the case), I think there's a good argument to be made that their current behavior is incorrect. Feel free to close this as an external issue.

i am absolutely using indn. with that said, perhaps i can work around this issue? if not, i might want to try putting together PRs for the terminals with problems. but i thought you were seeing this on XTerm? let me read your other comments...

[j4james]

but i thought you were seeing this on XTerm?

I think I misled you by putting the TERM value before the actual terminal name in the original report. But that's your own fault for ordering them that way in your problem report template. 😛

i might want to try putting together PRs for the terminals with problems

If you're doing that, I think xterm.js may be another library with this problem that's worth addressing. At least I was seeing this behavior in the Hyper terminal, which I believe is using xterm.js.

And Windows Terminal also has this issue, but I'll probably file a PR for that myself, or at least raise an issue for it. While I kind of prefer the VTE interpretation myself, if we're going to set our TERM to xterm-256color then we really should be matching the XTerm behavior.

[dankamongmen]

but i thought you were seeing this on XTerm?

I think I misled you by putting the TERM value before the actual terminal name in the original report. But that's your own fault for ordering them that way in your problem report template.

oooh that's a really good subtle point, let me go change that

[dankamongmen]

That doesn't trigger it. Assumedly because it's just scrolling with a linefeed rather than an indn or SU sequence.
An easy way to tell whether a terminal will be affected is with a command like this:

for i in {1..100}; do echo $i; done; tput indn 10

If you scroll up and see a break in the sequence, you know the terminal has this bug.

ahhh, that makes things clear, thank you.

ok, i will ponder this. thanks as always for the quality writeup and analysis!

[dankamongmen]

ok, i think i might understand what's going on here now (haven't reread, but had an insight while looking at some code).

• with CLI mode (which notcurses-info uses), unlike with Direct mode, we're only generating output at user-requested rendering times
• but we've promised a traditional scrolling interface to both
• we expect with traditional scrolling that it's fed off of stdio, which we don't use with CLI mode
• we instead print to planes, and then render frames, scrolling as necessary relative to the previous frames
• but at no time are we generating real-world output for output which was virtually scrolled off of a plane
• in the trivial case of a single (standard) plane the same size as the viewing area, output which scrolls off the plane will never be output

so if you for instance ran notcurses-info in a viewing area smaller than its total output, you will only get partial output. the top won't be scrollbackable.

i think that's the problem this issue describes; if not, we need go file this somewhere.

[j4james]

Fitting on screen isn't the problem - at least that's not the problem I'm complaing about. Let's say your screen is 37 lines high, which should be plenty of space for notcurses-info, but before running it, you output a 36 line file with the letters A-Z and the digits 0-9 each on a separate line. At that point you've got an A on line 1, a 9 on line 36, and the shell prompt on line 37.

If you run notcurses-info now, I'd expect most of the previous file output to scroll off the top of the screen into the scrollback, which it does to some extent, but not entirely. If you scroll back to find what's left of your file, you'll see A-F, then V-Z and 0-9. In other words, lines G-U have been nuked. At least that's the case on some terminals - on others it'll work fine.

Terminals that preserve the entire file include XTerm, MLTerm, and Alacritty, while terminals that lose part of the file include Gnome Terminal, Konsole, and Windows Terminal. I'm of the opinion that the latter group is in error, and this is really their problem to fix, but you may still want to try and avoid the situation if you can.