Listr2 breaks if number of tasks overflow the terminal height

[MrJmpl3]

This is a issue from Listr original but affect this version too: SamVerschueren/listr#150

[cenk1cenk2]

I could not come up with a good solution to this problem, even though there are some improvements.

• It writes the last render as process.stdout.write.
• It updates the render via the render hook if there are any changes to the task itself like a state change from pending to completed.

But depending on the terminal itself it is still prone to breaking while rendering due to log-update.

Main problem is if you increase the size of the screen, then reduce it, there may be some parts that are logged twice due to log-update's own behavior. I could not find a better lightweight library that handles this, so I guess until a suggestion is posted, it will stay with this trade-off.

You may also experience breaking or double logging of stuff when process.stdout is used by something else that writes to the console.

If you have any suggestions please feel to state them.

[cenk1cenk2]

sindresorhus/log-update#36

This is a related issue and makes sense to me why this is not likely to be solvable.

[metasean]

@cenk1cenk2 - I'm just starting to use Listr2 in a project and am getting lots of duplicates - even when there are no commands that exceed the terminal window and I'm not resizing or anything like that. This obviously isn't directly related to this issue, but to me it indicates there are likely multiple causes.

I also have some listrs that definitely exceed the terminal window, so the following suggestion is related to similar situations. (I also know close to nothing about terminal displays, output, etc. so take this question/suggestion with a massive grain of salt.

Instead of displaying the list at the top of the output, could the task entries display at the end of the console output, with "bookmarks" for the start and end of tasks that have their own output and plain markers for tasks that don't have their own output?

For example, say there are 4 tasks, then the initial display would include all of the tasks

┌─── terminal window (left side)
│ ⠴ Task 1
│ ◽️Task 2
│ ◽️Task 3
│ ◽️Task 4
│
│
│
│
└───────────────────────────────

Then there's some output from the first task, so Listr2 could (a) clear the terminal, (b) add a starting bookmark for the first task (that will remain), (c) output whatever has been sent from the first task, (d) add the remaining portion of the listr, so the end result would be something like this.

┌─── terminal window (left side)
│ > Started: Task 1
│ some output from the first task
│ more output from the first task
│ and still more output from the first task
│ ⠴ Task 1 
│ ◽️Task 2
│ ◽️Task 3
│ ◽️Task 4
└───────────────────────────────

From that point one, while there's output from first task, Listr2 could (a) clear it's running "final" display, (b) output whatever has been sent from the first task, (c) and add the remaining portion of the listr

┌─── terminal window (left side)
│ more output from the first task
│ and still more output from the first task
│ subsequent first task output
│ and still more subsequent first task output
│ ⠴ Task 1
│ ◽️Task 2
│ ◽️Task 3
│ ◽️Task 4
└───────────────────────────────

Once there's a final status from the first task, Listr2 could (a) clear it's running "final" display, (b) output whatever has been sent from the first task, (c) output the ending bookmark for the task, and (d) add the remaining portion of the listr

┌─── terminal window (left side)
│ and still more output from the first task
│ subsequent first task output
│ and still more subsequent first task output
│ final output from the first task
│ ✔︎ Completed: Task 1
│ ⠴ Task 2
│ ◽️Task 3
│ ◽️Task 4
└───────────────────────────────

With all the remaining tasks that have their own output, the process would be similar (a) clear out the running "final" display, (b) if it's the first output from the task, output a started bookmark, (c) output whatever has been sent from the task, (d) if it's the final output from the task, output a completed bookmark, and (e) add the remaining portion of the listr

┌─── terminal window (left side)
│ ✔︎ Completed: Task 1
│ > Started: Task 2
│ task 2 output
│ more task 2 output
│ final task 2 output
│ ✔︎ Task 2
│ ⠴ Task 3
│ ◽️Task 4
└───────────────────────────────

For any task that doesn't have output, then treat that task like normal

┌─── terminal window (left side)
│ ✔︎ Completed: Task 1
│ > Started: Task 2
│ task 2 output
│ more task 2 output
│ final task 2 output
│ ✔︎ Completed: Task 2
│ ✔︎ Task 3
│ ⠴ Task 4
└───────────────────────────────

Once all the tasks have been completed (successful or not), display the complete list with statuses

┌─── terminal window (left side)
│ task 4 output
│ final task 4 output
│ ✔︎ Completed: Task 4
│  
│ ✔︎ Task 1
│ ✔︎ Task 2
│ ✔︎ Task 3
│ ✔︎ Task 4
└───────────────────────────────

Unless the listr itself exceeded the window (and hopefully people aren't making such ginormous listrs) than this approach wouldn't require clearing text from output that has moved out of the frame (my current understanding of the problem), it would only¹ ever mean removing the "running" list at the bottom of output, possibly adding starting or ending bookmarks, outputting the current task's output, and adding the "running" or final list back in.

Regardless of this issue, thank you for picking up the work of Listr and being such an active maintainer!

---

1: Again, I have no real understanding of what is actually going on under the hood, I'm just deducing based on the previous comment's link. So the "only" is totally an assumption on my part.

[cenk1cenk2]

@metasean sincerely thank you for a detailed post with illustrations. This is definitely an alternative solution but the problem is it will break mostly in the same cases where it is resized because it will not have access to clearing the parts that are outside of the terminal window at a given time, then will rerender it with the space available. The next time it will render in the same place so the ones in the top will seem to be duplicated since they are not cleared in the prior run.

Although I am looking into the issue the best solution is going to be in my mind at least as follows:

• Swap log-update for another library namely stdout-update. This has a inbuilt hook for getting full control of stdout and stderr, so if anything tries to write to the streams it will delay it until the hook is called to finish the render. Then it will dump everything else. One of the problems of duplicating is something writing to process.stdout or process.stderr and breaking the log-update.
• The second part of the solution will be to render the finished tasks at the start writing to process.stdout directly, then use updating the log method for the rest of the tasks. So it will just render the parts that are not static yet.
• An alternative solution is to provide another renderer that only just renders the running tasks, skipped, and tasks with errors. Or a renderer as you described.

So this will take some time with refactoring but I guess it is the closest to a solution that can be implemented with the given constraint of not being able to update the lines outside the view. I am initially waiting for an update on this stdout-update library since it has no clear method exposed. It seems to re-render less, which is also desirable in my initial tests.

On your occasion please be sure that nothing is trying to output to process.stdout because with your description it is only the way it can break. Please set the renderer to silent and see if anything writes to output while the tasks are running.

[metasean]

Yeah, my proposed solution definitely doesn't address the window resizing.

With only a cursory glance, stdout-update does look like a better approach than log-update. It's awesome that @keindev is working on exposing the clear method!

Yes, several of my tasks do output to stdout. In the case of my really short listr, the first task has an updating statusbar (based on which listr messages that repeat, this is probably the primary culprit for this listr's duplicates). All the other tasks (for successes at least) have a simple one line message with some important detail that needs to be shared. Pretty much every task in my other problematic listr has a lot of stdout output each of which exceeds the terminal window by a lot.

For the moment, I'm just going to set listr renderer to silent and add simple, static console messages at the beginning and end of each of the tasks, but I'm definitely going to keep up with this issue so that I can switch to the significantly more informative and stylish listr output as soon as it doesn't look like it has a stubborn digital stutter.

[keindev]

The problem of duplicate lines in the output is related to the restriction, within which it is not possible to change the value of a line outside the work area.

I will try to explain on the diagram:

Changing the state of the output is possible only within the active area, from row 0/col 0 to row N/col N. Where the number of rows and columns depends on the size of your console.

For more see:
TTY
readline
readline source
TTY keybindings

As a solution to this limitation, I would suggest to output a large amount of information to the console at the end of all operations or when an error occurs.

[metasean]

@keindev - The goal of my suggestion was to work within the constraint that you're describing.

Using my previous examples, it would function something like the following (note that console output is only ever removed from the active console area).

For example, say there are 4 tasks, then the initial display would include all of the tasks

┌─── terminal window (left side)
│ ⠴ Task 1                                           // added
│ ◽️ Task 2                                          // added
│ ◽️ Task 3                                          // added
│ ◽️ Task 4                                          // added
│
│
│
└────────────────────────────────── (end of active console area)

Then there's some output from the first task, so Listr2 could (a) clear the terminal, (b) add a starting bookmark for the first task (that will remain), (c) output whatever has been sent from the first task, (d) add the remaining portion of the listr, so the end result would be something like this.

(a) clear the terminal

┌─── terminal window (left side)
│                                                     // cleared
│                                                     // cleared
│                                                     // cleared
│                                                     // cleared
│
│
│
└────────────────────────────────── (end of active console area)

(b) add a starting bookmark for the first task (that will remain)

┌─── terminal window (left side)
│ > Started: Task 1                                     // added
│
│
│
│
│
│
└────────────────────────────────── (end of active console area)

(c) output whatever has been sent from the first task

┌─── terminal window (left side)
│ > Started: Task 1
│ some output from the first task                       // added
│ more output from the first task                       // added
│ and still more output from the first task             // added
│
│
│
└────────────────────────────────── (end of active console area)

(d) add the remaining portion of the listr, so the end result would be something like this.

┌─── terminal window (left side)
│ > Started: Task 1
│ some output from the first task
│ more output from the first task
│ and still more output from the first task
│ ⠴ Task 1                                              // added
│ ◽️Task 2                                               // added
│ ◽️Task 3                                               // added
│ ◽️Task 4                                               // added
└────────────────────────────────── (end of active console area)

---

From that point one, while there's output from first task, Listr2 could (a) clear its running "final" display, (b) output whatever has been sent from the first task, (c) and add the remaining portion of the listr

(a) clear its running "final" display

┌─── terminal window (left side)
│ > Started: Task 1
│ some output from the first task
│ more output from the first task
│ and still more output from the first task
│                                                     // cleared
│                                                     // cleared
│                                                     // cleared
│                                                     // cleared
└────────────────────────────────── (end of active console area)

(b) output whatever has been sent from the first task

┌─── terminal window (left side)
│ > Started: Task 1
│ some output from the first task
│ more output from the first task
│ and still more output from the first task
│ subsequent first task output                          // added
│ and still more subsequent first task output           // added
│
│
└────────────────────────────────── (end of active console area)

(c) and add the remaining portion of the listr

┌─── terminal window (left side)
│ > Started: Task 1
│ some output from the first task
├────────────────────────────────── (beginning of active console area)
│ more output from the first task
│ and still more output from the first task
│ subsequent first task output
│ and still more subsequent first task output
│ ⠴ Task 1                                              // added
│ ◽️ Task 2                                              // added
│ ◽️ Task 3                                              // added
│ ◽️ Task 4                                              // added
└────────────────────────────────── (end of active console area)

---

Once there's a final status from the first task, Listr2 could (a) clear it's running "final" display, (b) output whatever has been sent from the first task, (c) output the ending bookmark for the task, and (d) add the remaining portion of the listr

(a) clear its running "final" display

┌─── terminal window (left side)
│ > Started: Task 1
│ some output from the first task
├────────────────────────────────── (beginning of active console area)
│ more output from the first task
│ and still more output from the first task
│ subsequent first task output
│ and still more subsequent first task output
│                                                     // cleared
│                                                     // cleared
│                                                     // cleared
│                                                     // cleared
└────────────────────────────────── (end of active console area)

(b) output whatever has been sent from the first task

┌─── terminal window (left side)
│ > Started: Task 1
│ some output from the first task
├────────────────────────────────── (beginning of active console area)
│ more output from the first task
│ and still more output from the first task
│ subsequent first task output
│ and still more subsequent first task output
│ final output from the first task                      // added
│ 
│ 
│ 
└────────────────────────────────── (end of active console area)

(c) output the ending bookmark for the task

┌─── terminal window (left side)
│ > Started: Task 1
│ some output from the first task
├────────────────────────────────── (beginning of active console area)
│ more output from the first task
│ and still more output from the first task
│ subsequent first task output
│ and still more subsequent first task output
│ final output from the first task
│ ✔︎ Completed: Task 1                                   // added
│ 
│ 
└────────────────────────────────── (end of active console area)

(d) add the remaining portion of the listr

┌─── terminal window (left side)
│ > Started: Task 1
│ some output from the first task
│ more output from the first task
├────────────────────────────────── (beginning of active console area)
│ and still more output from the first task
│ subsequent first task output
│ and still more subsequent first task output
│ final output from the first task
│ ✔︎ Completed: Task 1                                   // added
│ ⠴ Task 2                                              // added
│ ◽️ Task 3                                              // added
│ ◽️ Task 4                                              // added
└────────────────────────────────── (end of active console area)

---

With all the remaining tasks that have their own output, the process would be similar (a) clear out the running "final" display, (b) if it's the first output from the task, output a started bookmark, (c) output whatever has been sent from the task, (d) if it's the final output from the task, output a completed bookmark, and (e) add the remaining portion of the listr

┌─── terminal window (left side)
│ > Started: Task 1
│ some output from the first task
│ more output from the first task
│ and still more output from the first task
│ subsequent first task output
│ and still more subsequent first task output
│ final output from the first task
├────────────────────────────────── (beginning of active console area)
│ ✔︎ Completed: Task 1
│ > Started: Task 2
│ task 2 output
│ more task 2 output
│ final task 2 output
│ ✔︎ Completed: Task 2
│ ⠴ Task 3
│ ◽️ Task 4
└────────────────────────────────── (end of active console area)

For any task that doesn't have output, then treat that task like normal

┌─── terminal window (left side)
│ > Started: Task 1
│ some output from the first task
│ more output from the first task
│ and still more output from the first task
│ subsequent first task output
│ and still more subsequent first task output
│ final output from the first task
├────────────────────────────────── (beginning of active console area)
│ ✔︎ Completed: Task 1
│ > Started: Task 2
│ task 2 output
│ more task 2 output
│ final task 2 output
│ ✔︎ Completed: Task 2
│ ✔︎ Task 3                                           // replaced
│ ⠴ Task 4
└────────────────────────────────── (end of active console area)

Once all the tasks have been completed (successful or not), display the complete list with statuses

┌─── terminal window (left side)
│ > Started: Task 1
│ some output from the first task
│ more output from the first task
│ and still more output from the first task
│ subsequent first task output
│ and still more subsequent first task output
│ final output from the first task
│ ✔︎ Completed: Task 1
│ > Started: Task 2
│ task 2 output
│ more task 2 output
│ final task 2 output
│ ✔︎ Completed: Task 2
│ ✔︎ Task 3
│ > Started: Task 4
├────────────────────────────────── (beginning of active console area)
│ task 4 output
│ final task 4 output
│ ✔︎ Completed: Task 4
│  
│ ✔︎ Task 1
│ ✔︎ Task 2
│ ✔︎ Task 3
│ ✔︎ Task 4
└────────────────────────────────── (end of active console area)

It doesn't address a window size change, but I think it could solve aspects of the duplication problem.
Or am I missing something?

[keindev]

@metasean Yes, at step (c), there is a shift 2 lines down beyond the active area. At this point (as far as I believe, since I was solving a similar problem in tasktree-cli):

• there is a new iteration of rendering (every 100ms for example), so that the spinner spins around tasks (⠴ => ⠦ => ⠧)
• during this rendering, the same list from step (c) is updated and it is not taken into account that the next step shifted the scroll up by 2 lines and made them inaccessible for cleaning.
• the drawing of the step (c) starts from the 0 position, excluding the 2 lines shifted upwards, and at this moment the output is duplicated

I have a test for this situation, I do not know how much of what is happening in it will be clear.

Overall I think stdout-update will solve this problem.

[cenk1cenk2]

There is a new version published with 4.0.0-beta.x, which has stdout-update implemented in.

Even though it is not finalized yet and breaks some tests because of the difference of clearing lines between applications, it seems to work a charm. Thank you again @keindev for this beautiful plugin.

It will be merged with a later set of changes that I have in mind as a major version which can be tracked in the milestones.

If you have time and still have to go on projects with Listr2 you can give it a try by installing it with yarn add listr2@beta.

I would love to have some feedback, if possible.

[Josh-Walker-GM]

@cenk1cenk2 Do you mind if I ask what is the current progress of getting this moved out of beta and in to the main version? Is there something else holding back this switch to stdout-update? I would be brilliant to get this fix out in the wild 😄!

[cenk1cenk2]

Hello @Josh-Walker-GM,

Unfortunately, that solution has its own problems with making it compatible with the current form of the library.

But the case is since node.js can not access more than the terminal height, I suppose it might be no solution for the initial issue mentioned here but a limitation of how terminals handle themselves if I am not missing something.

[cenk1cenk2]

This has been moved to looking for contrubitions since I have failed to implement in stdout-update a couple of times, in some cases, it just generates no output in tests which is very weird.

I will close this issue now since it is mentioned in the documentation.