RuntimeError: cannot join current thread

[david30907d]

• I have visited the source website, and in particular
  read the known issues
• I have searched through the issue tracker for duplicates
• I have mentioned version numbers, operating system and
  environment, where applicable:

  import tqdm, sys
  print(tqdm.__version__, sys.version, sys.platform)

1. version info:

   4.25.0 3.5.2 (default, Nov 23 2017, 16:37:01)
   [GCC 5.4.0 20160609] linux
   

2. My code:

   for index, file_path in enumerate(tqdm.tqdm(Path(dirpath).iterdir())):
       do something
   

3. Error messages:

   Exception ignored in: <bound method tqdm.__del__ of 50it [00:31,  1.56it/s]>
   Traceback (most recent call last):
     File "/usr/local/lib/python3.5/dist-packages/tqdm/_tqdm.py", line 885, in __del__
       self.close()
     File "/usr/local/lib/python3.5/dist-packages/tqdm/_tqdm.py", line 1090, in close
       self._decr_instances(self)
     File "/usr/local/lib/python3.5/dist-packages/tqdm/_tqdm.py", line 454, in _decr_instances
       cls.monitor.exit()
     File "/usr/local/lib/python3.5/dist-packages/tqdm/_monitor.py", line 52, in exit
       self.join()
     File "/usr/lib/python3.5/threading.py", line 1051, in join
       raise RuntimeError("cannot join current thread")
   RuntimeError: cannot join current thread
   

[chengs]

@david30907d 你好
I am trying to reproduce this error, so far I only have py3.6. Before I install py3.5, l would like to confirm with you about the code. If I understand correctly, the following code will cause the error you mentioned in py3.5. Am I correct?

import tqdm
from pathlib import Path

for index, file_path in enumerate(tqdm.tqdm(Path('./').iterdir())):
      # let's do something 
      print(index,file_path)

And, if so, does it always raise an error? or occasionally?

[david30907d]

@chengs my fault

I didn't finish iterating the loop but break it in the middle

when i removed the break clause, nothing error messages pop up.

so tqdm doesn't support break in for loop right ?

my code:

import tqdm
from pathlib import Path

for index, file_path in enumerate(tqdm.tqdm(Path('./').iterdir())):
        if index ==  num:
                break
        do_something()

[chengs]

@david30907d
It should support, but currently there is a unresolved bug.
I am still collecting information to solve it.
It seems break raises errors occasionally.
Maybe u can try tqdm=v4.19.

[david30907d]

@chengs OK, thanks for your fast response

[casperdcl]

vis #548

[daquang]

I can confirm that break causes this error. It is not present in tqdm versions 4.24.0 and earlier.

[anntzer]

Another example which does not use break and also results in a "cannot join current thread" error: modify the https://github.com/tqdm/tqdm#redirecting-writing example to use a nested progress bar, i.e. replace

    for i in tqdm(range(3), file=orig_stdout, dynamic_ncols=True):
        sleep(.5)
        some_fun(i)

by

    for i in tqdm(range(10), file=orig_stdout, dynamic_ncols=True):
        for j in tqdm(range(10), file=orig_stdout, dynamic_ncols=True):
            sleep(.5)
            some_fun(i)

As of tqdm 4.28.1/Py3.7, this reliably raises the exception after the third outer iteration.

[chengs]

@anntzer thank you for the feedback. Let me have a look.

[chengs]

@anntzer can you let me know your environment? (Linux/windows)

[anntzer]

Linux (fedora), miniconda-installed python but packages installed by pip.

[chengs]

@anntzer I cannot reproduce the error. It will be quite helpful if you can instruct me.
I save the following code into a bugfix.py document and run it. But I cannot see any error on py37
Can you check if my code is correct to trigger the bug?

from time import sleep
import contextlib
import sys
import tqdm
print(tqdm.__version__)
from tqdm import tqdm


class DummyTqdmFile(object):
    """Dummy file-like that will write to tqdm"""
    file = None
    def __init__(self, file):
        self.file = file

    def write(self, x):
        # Avoid print() second call (useless \n)
        if len(x.rstrip()) > 0:
            tqdm.write(x, file=self.file)

    def flush(self):
        return getattr(self.file, "flush", lambda: None)()

@contextlib.contextmanager
def std_out_err_redirect_tqdm():
    orig_out_err = sys.stdout, sys.stderr
    try:
        sys.stdout, sys.stderr = map(DummyTqdmFile, orig_out_err)
        yield orig_out_err[0]
    # Relay exceptions
    except Exception as exc:
        raise exc
    # Always restore sys.stdout/err if necessary
    finally:
        sys.stdout, sys.stderr = orig_out_err

def some_fun(i):
    print("Fee, fi, fo,".split()[i])

# Redirect stdout to tqdm.write() (don't forget the `as save_stdout`)
with std_out_err_redirect_tqdm() as orig_stdout:
    # tqdm needs the original stdout
    # and dynamic_ncols=True to autodetect console width
    for i in tqdm(range(3), file=orig_stdout, dynamic_ncols=True):
        for j in tqdm(range(1), file=orig_stdout, dynamic_ncols=True):
            sleep(.1)
            some_fun(i)
# After the `with`, printing is restored
print("Done!")

[anntzer]

Ow, wait. I had changed the outer loop to go all the way to range(10) (as reported in my original message above) but that of course fails because some_fun raises an IndexError for such values. And indeed, the full traceback (the upper half of which I didn't pay attention to) is

Traceback (most recent call last):███████████████████                                                                                                                | 3/10 [00:15<00:35,  5.02s/it]
  File "/tmp/test.py", line 46, in <module>                                                                                                                                  | 0/10 [00:00<?, ?it/s]
    some_fun(i)
  File "/tmp/test.py", line 37, in some_fun
    print("Fee, fi, fo,".split()[i])
IndexError: list index out of range
Exception ignored in: <function tqdm.__del__ at 0x7ffa352597b8>
Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/tqdm/_tqdm.py", line 931, in __del__
    self.close()
  File "/usr/lib/python3.7/site-packages/tqdm/_tqdm.py", line 1133, in close
    self._decr_instances(self)
  File "/usr/lib/python3.7/site-packages/tqdm/_tqdm.py", line 496, in _decr_instances
    cls.monitor.exit()
  File "/usr/lib/python3.7/site-packages/tqdm/_monitor.py", line 52, in exit
    self.join()
  File "/usr/lib/python3.7/threading.py", line 1029, in join
    raise RuntimeError("cannot join current thread")
RuntimeError: cannot join current thread

so the RuntimeError occurs while trying to cleanup after an exception thrown from within the loop.

Some important points:

• The error only occurs with a sleep of 0.5, not a sleep of 0.1.
• The 'Exception ignored' message occurs even if the IndexError is caught with a try... except that surrounds the loops (if it did not I guess one may perhaps just consider this an edge case, but I think tqdm should not cause error messages to be displayed when the exception is ultimately caught and handled).

[chengs]

@anntzer Thank you, now I reproduce the error. It's interesting...
I made a quick fix in PR #641.
If you have time, can you help me test it?

[anntzer]

Looks like #641 fixes the issue for me, thanks for the quick fix.

[stefan-falk]

I have a similar issue. I don't know exactly what the reason is but - there is no break in my loop. The only thing that's there is a Pool

for stm_filepath, sph_filepath in tqdm.tqdm(filepath_pairs, desc='Converting %s data' % mode):
    _convert_sample(stm_filepath, sph_filepath, out_dir)

where _convert_sample() is:

def _convert_sample(stm_file, sph_file, output_dir):

    # ...

    with open(stm_file) as transcript_file, open(txt_file, 'w') as txt:
        jobs = list()
        for line_idx, transcript_line in enumerate(transcript_file):
            # ...
            jobs.append((sph_file, wav_file, start, str(float(end) - float(start))))

        pool = Pool(processes=cpu_count())
        pool.starmap(_slice_sph_and_convert_to_wav, jobs)
        pool.close()
        pool.join()

The error I see is

Exception ignored in: <bound method tqdm.__del__ of Generating samples for train:  19%|###################8                                                                            
           | 49746/268252 [13:58<5:06:42, 11.87it/s]>
Traceback (most recent call last):
  File "/home/sfalk/miniconda3/envs/t2t/lib/python3.5/site-packages/tqdm/_tqdm.py", line 931, in __del__
    self.close()
  File "/home/sfalk/miniconda3/envs/t2t/lib/python3.5/site-packages/tqdm/_tqdm.py", line 1133, in close
    self._decr_instances(self)
  File "/home/sfalk/miniconda3/envs/t2t/lib/python3.5/site-packages/tqdm/_tqdm.py", line 496, in _decr_instances
    cls.monitor.exit()
  File "/home/sfalk/miniconda3/envs/t2t/lib/python3.5/site-packages/tqdm/_monitor.py", line 52, in exit
    self.join()
  File "/home/sfalk/miniconda3/envs/t2t/lib/python3.5/threading.py", line 1051, in join
    raise RuntimeError("cannot join current thread")
RuntimeError: cannot join current thread

Environment

$ python --version
Python 3.5.6 :: Anaconda, Inc.

$ pip show tqdm
Name: tqdm
Version: 4.28.1

[chengs]

@stefan-falk let us wait for the merge of PR #641. The bug will be fixed.

[pseeth]

Hi all, I just ran into this bug as well with two nested tqdm loops keeping track of the progress of a multithreaded python machine learning library. Is there a timetable for this fix? It's a bit of a show-stopper. My program works fine when it's just one tqdm loop. It's only when I have two that everything breaks.

Thanks!

[verazuo]

I meet this bug too.
I am trying to crawl some data, and tqdm always raise this error after it crawled around 6k data.
I tested it on many different pages. And after I removed tqdm, there is no such error raised.

[cpcdoy]

I'm getting the same error when using "set_description" and placing a "break" right after..

[casperdcl]

might really need to fix using #610 or similar

[Fly-Pluche]

I can confirm that break causes this error. It is not present in tqdm versions 4.24.0 and earlier.

This is the cmd to install : pip install tqdm==4.24.0

[chengs]

this should be already fixed in the current version long time ago.