Multiple bars display fix

[kolayne]

About

Fixes several issues with bars positioning when using multiple bars. Resolves #1000 and probably some other similar issues.

Other solutions

I know there already exists a PR fixing the problem (#1001), but I don't like it for a few reasons, including:

• it simply doesn't work for me (however, I didn't do my best to make it. I'll probably try it later and tell about my results on its page)
• it introduces negative positions, which I don't find intuitive, and (if I'm not mistaken) requires the library users to change their code to make it work correctly

Changed and unchanged behaviour

Here there are samples I was running to compare the modified tqdm with the original one. All the samples were run on Python with sys.platform == 'linux', sys.version == '3.8.5 (default, Jul 28 2020, 12:59:40) \n[GCC 9.3.0]'. The "before" screenshots are run on tqdm installed from master, the "after" screenshots are run on tqdm installed from the current branch (bars_position_fix).

Trivial

(Just to verify I haven't broken the very basics)

from time import sleep

from tqdm import trange


for _ in trange(10):
    sleep(0.1)

for _ in trange(10, leave=False):
    sleep(0.1)

Before

After

Nested loops

from time import sleep

from tqdm import trange


for _ in trange(10):
    for _ in trange(10):
        sleep(0.1)

Before

After

Multithreaded with `position`

Please, pay attention to the order of the bars (numbers in the beginnings of the lines): in the version before the PR bars change their order. Sometimes (when using enough many bars or running them long enough) a similar sample can also produce empty lines or duplicate bars (if you want, I can try to dig into it and provide such a sample)

from time import sleep
from threading import Thread

from tqdm import trange


def progresser(n):
    interval = 0.001 / (n + 2)
    total = 1000
    text = "#{}, est. {:<04.2}s".format(n, interval * total)
    for _ in trange(total, desc=text, position=n):
        sleep(interval)


if __name__ == '__main__':
    list(map(lambda x: Thread(target=progresser, args=(x,)).start(), range(9)))

Before

After

One-threaded with `position` and `leave=False`

from time import sleep

from tqdm import trange


def progresser(n):
    interval = 0.001 / (n + 2)
    total = 1000
    text = "#{}, est. {:<04.2}s".format(n, interval * total)
    for _ in trange(total, desc=text, position=n, leave=False):
        sleep(interval)


if __name__ == '__main__':
    list(map(progresser, range(9)))

Before

After

One-threaded with `position`

This is the only case (I can think of) where you have to change code to make it produce a perfect output (if not changing it, the produced output is still better, than the output before the PR, as for me). However, I don't think this is a serious issue, because I think, this case is extremely special: there is no need to use bars with position if the application is one-threaded. In theory, there is one more situation where this case may come (when you use a multi-threaded application, but for some reason, a subset of the bars you want to use both starts and closes before other bars are created).

from time import sleep

from tqdm import trange


def progresser(n):
    interval = 0.001 / (n + 2)
    total = 1000
    text = "#{}, est. {:<04.2}s".format(n, interval * total)
    for _ in trange(total, desc=text, position=n):
        sleep(interval)


if __name__ == '__main__':
    list(map(progresser, range(9)))

Before

After

Updated sample:

from time import sleep

from tqdm import trange


def progresser(n):
    interval = 0.001 / (n + 2)
    total = 1000
    text = "#{}, est. {:<04.2}s".format(n, interval * total)
    for _ in trange(total, desc=text, position=n, terminate_lines=False):
        sleep(interval)


if __name__ == '__main__':
    list(map(progresser, range(9)))
    print('\n' * 9, end='')

After edit

Tests

I have very little experience with tests (both in python and generally). My PR has obviously changed tqdm outputs in some cases, so I think some tests will fail and I'll need help from maintainers (@casperdcl?) to understand, where I should change something back and where tests should be updated instead

[kolayne]

I am not 100% sure about this change. I don't understand what is and not pos here for, but it affects the fourth example from the PR description (with leave=False): it leaves an unexpected newline symbol

[kolayne]

Have to combine the locking inside of _decr_instances and the locking located some lines below to avoid a race condition, which may result in 2 (or more) bars terminating lines, while only the very last one must

[haji-ali]

This does fix the issue that I raised (when I use terminate_lines) and wrote the negative-position patch for.
However, I don't understand why is it impossible for the case "One-threaded with position" to work correctly without modifications to user code? That seems like a standard case that tqdm should be able to handle.

Also, the nested loop example is still strange IMO. I would expect based on the documentation that all 10 bars should remain after execution (since leave is true) with another top bar for the outer loop.

[kolayne]

@haji-ali, thank you for the constructive feedback!

However, I don't understand why is it impossible for the case "One-threaded with position" to work correctly without modifications to user code? That seems like a standard case that tqdm should be able to handle.

I expect tqdm to move the cursor below all the bars after they close, to let me print something to the file. So, when the only bar is running, tqdm moves the cursor immediately after its completion. So, pos of the next bar is applied when the cursor is moved already, that's why empty lines appear.
But even if I'd change this behavior (i. e. make terminate_lines=True by default), the end-user will anyway have to either tell tqdm that all the bars are completed to make tqdm move the cursor or move it manually, to be able to print something later, so I don't think it's possible to avoid code changes in such case.

Also, the nested loop example is still strange IMO. I would expect based on the documentation that all 10 bars should remain after execution (since leave is true) with another top bar for the outer loop.

I don't think, this is a feature because the following sample produces a similar output for the current version of tqdm in master (which surely is a bug):

for _ in trange(10, position=0):
    for _ in trange(10, position=1):
        pass

I think this following part from the docs is supposed to tell that the sample from the PR's description should be interpreted just like the above sample (however, that's not obvious at all, I'd like to get a confirmation or a denial from @casperdcl):

tqdm/tqdm/std.py

Lines 918 to 920 in 8f9e03d

	position : int, optional
	Specify the line offset to print this bar (starting from 0)
	Automatic if unspecified.

But even if it's not a bug, it should be fixed in a separate PR, because this is an issue with the _get_free_pos method, used to figure out the position of a bar automatically:

tqdm/tqdm/std.py

Line 1079 in 8f9e03d

self.pos = self._get_free_pos(self)

[Knaifhogg]

I tried this PR with my nested multi-thread long function that I've been having issues with, and it seems to resolve most of my problems.
Setup:

• Windows 10 64-bit
• Python 3.8.236
• tqdm 4.51.0
• running in cmd

Before:

After:

Now I only have to figure out a way to count up the inner loop time spent before it is completed.

[kolayne]

Hello, @casperdcl. I'm sorry for bothering you, but I wonder if there is any news on this?

[casperdcl]

Ok finally reviewed this (!) Referring to the screenshots:

Trivial

ok

Nested loops

this is identical to setting leave=False for nested loops. That was the original nested implementation but it was changed to the current behaviour because more people wanted that.

Multithreaded with position

Same as above - bars are "left" (i.e. printed like normal lines) as they are completed.

One-threaded with position and leave=False

The whitespace causing a newline in the current version is odd - looks like a bug.

One-threaded with position

I'm not too concerned by this as both versions look bad to me. If someone manually specifies position in a naïve way (both cases) then the output will look bad.

[kolayne]

@casperdcl, thanks! :)
Here are my clarifications / questions

this is identical to setting leave=False for nested loops. That was the original nested implementation but it was changed to the current behaviour because more people wanted that

Oops, looks like I misunderstood the tqdm.__init__'s docstring. I guess I thought the default value of leave is None. I agree that the new behaviour is a bug. But the behaviour in master at the moment is a bug too, isn't it? If what you're saying is you don't like my PR is replacing one bug with another one, then I'm going to work on it and fix it to look in the way you want it to (how do you want it to, btw? Should the bar of the outer loop be first and the other bars be printed below one after another? I strongly believe this should be documented somewhere in tqdm.__init__ (in the section explaining the automatic position determination) or in tqdm._get_free_pos), but if you are saying the master's behaviour is not a bug (i.e. tqdm is expected to work this way), I think most users would disagree with you.

Same as above - bars are "left" (i.e. printed like normal lines) as they are completed

???
Are you saying it is the expected behaviour for bars to change their positions during the execution? Hope, no, but then I don't understand what you meant. The set of bars printed before and after the PR/patch is the same, but my PR fixes the order. Please, add more details

I'm not too concerned by this as both versions look bad to me. If someone manually specifies position in a naïve way (both cases) then the output will look bad.

Not sure if I understood you correctly. Which both of the three (Before, After, After edit) are you talking about?


Update: I think it is a good idea to "Request changes" on this PR (under the "Review changes" button in the "Files changed" tab) so that I can re-request the review after we finish the discussion and I push new commits

[casperdcl]

from time import sleep
print(sleep(1), "task 3 completed")
print(sleep(1), "task 2 completed")
print(sleep(1), "task 1 completed")

will print statements in order of completion (3, 2, 1). The same is true of any bar which has leave=True.

from time import sleep
from tqdm import tqdm
for i in tqdm([1], desc="task 1"):
  for j in tqdm([1], desc="task 2"):
    for k in tqdm([1], desc="task 3"):
      sleep(1)

The final order is (3, 2, 1).

The fact that while the bars are "running" the transient order is (1, 2, 3) is besides the point. If you would like to maintain this transient order even after the bars have closed (i.e. print final order (1, 2, 3)), this is identical to expecting print(3); print(2); print(1) to print in reverse order. While this is do-able, it's up to the end user to implement their own context manager/hack to get this highly non-standard behaviour.

The easiest way may be to set leave=False, and collect all tqdm objects in a list:

bars = [tqdm(..., leave=False), ...]
# do some work, update/close bars, etc...

# print them all
for bar in bars:
    print(bar)

[kolayne]

this is identical to expecting print(3); print(2); print(1) to print in reverse order. While this is do-able, it's up to the end user to implement their own context manager/hack to get this highly non-standard behaviour.

This is questionable. In my opinion, the standard behaviour for bars is to not be overwritten by other bars if leave=True. It sounds more standard if the bars' positions aren't messed up during the process, doesn't it?
And whether you agree with me or not, because there is a misinterpretation(?), this should be documented too!

[haji-ali]

I personally find the behavior with the patch more intuitive. The nested loop behavior is especially confusing. The bars being overwritten and unfinished bars being left over makes it difficult to tell the progress if I am not following how the new bars are being created.

[kolayne]

Looks like the discussion is stuck. Is there some well-established mechanism to find out the opinion of users on such questions? How do we figure out which of these do people actually find more intuitive?

[kolayne]

@casperdcl?..

Looks like the discussion is stuck. Is there some well-established mechanism to find out the opinion of users on such questions? How do we figure out which of these do people actually find more intuitive?

[espdev]

I have checked this PR. If we do not set position argument and use a semaphore, the code works incorrectly.

The code example:

from concurrent.futures import ThreadPoolExecutor, wait
from threading import Semaphore
import time
import random

from tqdm import tqdm

def worker(pos, sem):
    t = random.random() * 0.05
    with sem:
        # for _ in tqdm(range(100), desc=f'pos {pos}', position=pos):  # it works correctly
        for _ in tqdm(range(100), desc=f'pos {pos}'):  # it works incorrectly
            time.sleep(t)

def main():
    with ThreadPoolExecutor() as executor:
        sem = Semaphore(3)
        futures = []
        for pos in range(10):
            future = executor.submit(worker, pos, sem)
            futures.append(future)

        wait(futures)

if __name__ == '__main__':
    main()

The new bars overwrite the old bars if we do not set position argument:

[kolayne]

Hello! I'm back here. Sincerely sorry for such a long delay, I start getting closer to removing the Busy mark from my profile.

I have checked this PR. If we do not set position argument and use a semaphore, the code works incorrectly.

@espdev, thank you so much for pointing it out. I'm going to fix this (hopefully, soon).

In the model which I propose in this PR, this bug seems to be caused by the logic of _get_free_pos, which I didn't alter with the other changes. Before I can fix it, I'd like to find out (from @casperdcl, most likely): is that a bug or a feature that the tqdm._get_free_pos function does not do anything with the file argument of tqdm.__init__? Should it find the highest position in the file where the new bar is created instead of what it does now (end-to-end position numbering across files)?

Method's source:

tqdm/tqdm/std.py

Lines 579 to 584 in 140c948

	@classmethod
	def _get_free_pos(cls, instance=None):
	"""Skips specified instance."""
	positions = {abs(inst.pos) for inst in cls._instances
	if inst is not instance and hasattr(inst, "pos")}
	return min(set(range(len(positions) + 1)).difference(positions))

Example code:

from time import sleep
from threading import Thread

from tqdm import tqdm


def run_me(f):
    for _ in tqdm(list(range(20)), file=f):
        sleep(0.1)


if __name__ == "__main__":
    f = open("/tmp/f1", "w")
    t1 = Thread(target=run_me, args=(f,))
    t2 = Thread(target=run_me, args=(None,))
    t1.start()
    t2.start()
    t1.join()
    t2.join()

Result of the code execution: https://asciinema.org/a/431831

[kolayne]

@casperdcl, I have one more question regarding the expected bars' behavior. By the reaction you have left I can tell that you agree that the behavior found by espdev is incorrect. I assume that in that case it was expected that new bars appear under the completed ones. But I wonder about one more complicated case. Imagine there were three bars created: first with leave=True, second with leave=False, third with leave=True. At some point, the second bar closes, and after that, a new bar with leave=True is created. Should the third bar move to the place of the second one when the second gets closed (i.e. it collapses and other bars move up)? Should the third bar just leave an empty line so that the fourth will take its place when it's created? Or should the fourth bar take the fourth line (so that the second line will remain empty forever)?

[BenjaminChoou]

@kolayne

So far, I use Numba to help me achieve multi-progress bars display with multi-threads. But this cannot be implemented via the python multi-process library since communication between different processes is not straightforward.

Therefore, I wonder if it is possible for tqdm to maintain a unique internal pool for multi-progress bars in the master process while other processes can send a signal to update those bars. Then it would be easier for users to manage the multiple progress bars not only for multi-thread usage but also for complicated scenarios.

Please check my naive implementation.

from tqdm import tqdm
import time


class ProgressBar(object):
    def __init__(self, name, descs, totals):
        self.name = name
        self.bars = [tqdm(total=total, desc=desc, position=i)
                     for i, (desc, total) in enumerate(zip(descs, totals))]
        self.end_time = [0.0 for _ in range(len(totals))]

    def reset_start_time(self, bar_id):
        self.bars[bar_id].last_print_t = time.time()
        self.bars[bar_id].start_t = self.bars[bar_id].last_print_t

    def update(self, bar_id: int):
        assert 0 <= bar_id < len(self.bars)

        this_bar = self.bars[bar_id]
        cur_progress = this_bar.n
        total = this_bar.total

        if cur_progress < total:
            this_bar.update(1)
            if cur_progress == total - 1:
                this_bar.refresh()
            self.end_time[bar_id] = this_bar.last_print_t

    def close(self):
        for bar_id, bar in enumerate(self.bars):
            # reset the position of bar to 0 before close
            # so it won't be in wrong position
            bar.pos = 0
            # correct the time elapsed
            bar.start_t += (time.time() - self.end_time[bar_id])
            bar.close()


class ProgressBarManager(object):
    """
    A manager class for multiple progress bars.
    Potential usage is to display progress bars for multi-thread applications.
    """
    pool = None

    def __init__(self):
        raise RuntimeError('`ProgressBarPools` is not allowed to be instantiated')

    @staticmethod
    def create_pool(name, descs, totals):
        if ProgressBarManager.pool is None:
            ProgressBarManager.pool = ProgressBar(name, descs, totals)
        else:
            raise RuntimeError(
                'Progressbar pool should be empty before getting new pool:`%s` exists' % ProgressBarManager.pool.name)

    @staticmethod
    def reset_start_time(bar_id):
        assert isinstance(ProgressBarManager.pool, ProgressBar)
        ProgressBarManager.pool.reset_start_time(bar_id)

    @staticmethod
    def update(bar_id):
        assert isinstance(ProgressBarManager.pool, ProgressBar)
        ProgressBarManager.pool.update(bar_id)

    @staticmethod
    def get_total(bar_id):
        assert isinstance(ProgressBarManager.pool, ProgressBar)
        return ProgressBarManager.pool.bars[bar_id].total

    @staticmethod
    def close():
        assert isinstance(ProgressBarManager.pool, ProgressBar)
        ProgressBarManager.pool.close()
        ProgressBarManager.pool = None


create_pool = ProgressBarManager.create_pool
reset_start_time = ProgressBarManager.reset_start_time
update = ProgressBarManager.update
get_total = ProgressBarManager.get_total
close = ProgressBarManager.close

import numba
from numba import objmode
import time


@numba.jit(nopython=True, parallel=True)
def multi_thread_test():
    n_jobs = min(numba.get_num_threads(), 8)

    with objmode():
        # run code block by python interpreter
        descs = ['Batch # %d' % i for i in range(n_jobs)]
        totals = [30 * (i + 1) * (n_jobs - i) for i in range(n_jobs)]

        create_pool('Test', descs, totals)

    for batch_id in numba.prange(n_jobs):
        # run loops in parallel
        total = 30 * (batch_id + 1) * (n_jobs - batch_id)

        with objmode():
            reset_start_time(batch_id)

        for i in range(total):
            with objmode():
                time.sleep(0.01)
                update(batch_id)

    with objmode():
        close()


multi_thread_test()

[kolayne]

Therefore, I wonder if it is possible for tqdm to maintain a unique internal pool for multi-progress bars in the master process while other processes can send a signal to update those bars.

I believe the solutions to the problem exist, but I consider them going out of the frame of this PR: here I only focus on problems with bars positioning caused by logical bugs and don't focus on processes' synchronization. You might want to discuss this in another (a new one?) issue/PR.

Then it would be easier for users to manage the multiple progress bars not only for multi-thread usage but also for complicated scenarios.

To be honest, I wouldn't say the implementation you've provided is easy to use: I see that you need to manually call the update method. And, in my opinion, it's one of tqdm's key features that you don't need to modify code anyhow but just surrounding for loops with tqdm() to make it work.

Please check my naive implementation.

I am not familiar with numba, so I can't tell exactly what the code does, although I don't see any place where any synchronization is performed, so the only difference from using usual tqdms in multiple processes, which I see in your code, is explicitly specifying positions. Besides, if I was doing a code review, I would have a couple of comments here (including the one I've already mentioned about user-friendliness - the update calls - and responsibility-sharing problems (you have some checks which are performed by tqdm internally anyway)).

If you want to work on this problem, if I were you, I'd start by either finding or filing an issue with a detailed explanation of the problem you want to solve and code samples (preferably - using only standard libraries (like threading/multiprocessing), unless the problem only occurs with something specific like numba), then reading the current tqdm's source (at the tqdm/std.py file) and then making a PR.

[codecov]

Codecov Report

Merging #1054 (4f208e7) into master (4f208e7) will not change coverage.
The diff coverage is n/a.

❗ Current head 4f208e7 differs from pull request most recent head e10d99b. Consider uploading reports for the commit e10d99b to get more accurate results

@@           Coverage Diff           @@
##           master    #1054   +/-   ##
=======================================
  Coverage   89.05%   89.05%           
=======================================
  Files          26       26           
  Lines        1763     1763           
  Branches      344      344           
=======================================
  Hits         1570     1570           
  Misses        144      144           
  Partials       49       49           

[BenjaminChoou]

Therefore, I wonder if it is possible for tqdm to maintain a unique internal pool for multi-progress bars in the master process while other processes can send a signal to update those bars.

I believe the solutions to the problem exist, but I consider them going out of the frame of this PR: here I only focus on problems with bars positioning caused by logical bugs and don't focus on processes' synchronization. You might want to discuss this in another (a new one?) issue/PR.

Then it would be easier for users to manage the multiple progress bars not only for multi-thread usage but also for complicated scenarios.

To be honest, I wouldn't say the implementation you've provided is easy to use: I see that you need to manually call the update method. And, in my opinion, it's one of tqdm's key features that you don't need to modify code anyhow but just surrounding for loops with tqdm() to make it work.

Please check my naive implementation.

I am not familiar with numba, so I can't tell exactly what the code does, although I don't see any place where any synchronization is performed, so the only difference from using usual tqdms in multiple processes, which I see in your code, is explicitly specifying positions. Besides, if I was doing a code review, I would have a couple of comments here (including the one I've already mentioned about user-friendliness - the update calls - and responsibility-sharing problems (you have some checks which are performed by tqdm internally anyway)).

If you want to work on this problem, if I were you, I'd start by either finding or filing an issue with a detailed explanation of the problem you want to solve and code samples (preferably - using only standard libraries (like threading/multiprocessing), unless the problem only occurs with something specific like numba), then reading the current tqdm's source (at the tqdm/std.py file) and then making a PR.

Thanks for your reply.

[rckoepke]

5116501 worked perfectly for me in PyCharm on Ubuntu using "Emulate terminal in output console". Thank you!

[kolayne]

@rckoepke, glad to hear that, thank you for your feedback!

Why did you use this commit, not the top of the branch? Does the behavior somehow worsen in the following commits or you just haven't tested?

[rckoepke]

To be honest, I independently found the same two-line solution and it was sufficient for my use case. I went to submit a issue / PR for it and found this existing PR that already had the same solution, plus apparently more comprehensive coverage. It looked a bit stale and I so I wanted to provide additional positive feedback.

The other commits look very helpful and certainly better than the original two-line solution that I came up with, but I'm not sure how I would rigorously test them. I do have Windows+Mac+Linux and a variety of IDE's and terminal environments, so I'd be happy to do cross-platform testing if there was some kind of a test matrix that I could follow.

[casperdcl]

905453860: By the reaction you have left I can tell that you agree that the behavior found by espdev is incorrect

Oops, no that's not what I meant. Retracted the 👍. What I mean is the output of the code is as expected. The # it works incorrectly comment should instead be # auto-position it works as intended, while the # it works correctly comment should be # it doesn't work at all.

Here's my output of auto-position (unaltered copy-paste of this code):

If this is counter-intuitive it probably bears documenting.

communication between different processes is not straightforward

yes, the Numba implementation is interesting but I haven't had a look at it yet.

[carlosgmartin]

@kolayne @casperdcl Came across this from #1000. Any update on this?

[sigvaldm]

I also stumbled across this issue when using multiprocessing. The example in the README still does not work. Any update on when a solution might be merged into a stable release?