How to create a dynamic progress bar, which increases its maximum size while looping?

[evandrocoan]

I got this recursive function which given a directory, recursively finds all files on it:

def find_files_recursively(sourcepath)
    files = []

    if os.path.isdir( sourcepath ):

        def recursively_search(sourcepath):
            filenames = [ '.' ]
            filenames.extend( os.listdir( sourcepath ) )

            for somefile in filenames:
                fullpath = os.path.join( sourcepath, somefile )
                fullpath = os.path.normpath( fullpath )

                if os.path.isdir( fullpath ):
                    recursively_search( fullpath )

                else:
                    files.append( sourcepath )

        recursively_search( sourcepath )

    elif os.path.exists( sourcepath ):
        files.append( sourcepath )

How could I write a progress bar which updates while new files are being found?

I should update each time I list files in a directory, for example:

def find_files_recursively(sourcepath)
    files = []

    if os.path.isdir( sourcepath ):
        myprogressbar = tqdm.?

        def recursively_search(sourcepath):
            filenames = [ '.' ]
            filenames.extend( os.listdir( sourcepath ) )
            myprogressbar.update_maximum( len( filenames ) - 1 )

            for somefile in filenames:
                myprogressbar.update(1)

                fullpath = os.path.join( sourcepath, somefile )
                fullpath = os.path.normpath( fullpath )

                if os.path.isdir( fullpath ):
                    recursively_search( fullpath )

                else:
                    files.append( sourcepath )

        recursively_search( sourcepath )

    elif os.path.exists( sourcepath ):
        files.append( sourcepath )

[casperdcl]

not sure exactly what you want here - could you provide example desired output?

[evandrocoan]

For example:

  0%|            | 0/15 
  7%|▋           | 1/15 
 13%|█▎          | 2/60 
 20%|██          | 3/60 
 (7)%|██▋        | 4/60 
 (8)%|██▋        | 5/60 
 (10)%|███▋      | 6/60 
 (12)%|███▋       | 7/60 
 (13)%|███▋       | 8/60 
 (15)%|███▋       | 9/60 
 (7)%|███▋       | 10/150 
 (7)%|██▋       | 11/150 
 (8)%|██▋       | 12/150 
 (9)%|██▋       | 13/150 
 (9)%|██▋       | 14/150 
 (10)%|██▋       | 15/150 
 (11)%|██▋       | 16/150 
 (11)%|██▋       | 17/150 
 ...
 (93)%|███████▋ | 139/150 
 (93)%|████████▋| 140/150 
 (94)%|████████▋| 141/150 
 (95)%|████████▋| 142/150 
 (95)%|████████▋| 143/150 
 (96)%|████████▋| 144/150 
 (97)%|████████▋| 145/150 
 (97)%|████████▋| 146/150 
 (98)%|████████▋| 147/150 
 (99)%|████████▋| 148/150 
 (99)%|████████▋| 149/150 
 (100)%|████████| 150/150 

On this example, when new files are found, the maximum and initial limit was increased from 15 to 60 (because more 45 files where found). And later on, when more new files were found, the maximum limit was increased from 60 to 150 (because more 90 files were found). on the end, no more new files were found, and everything was parsed.

[casperdcl]

because more [...] files were found

I presume this increase in total is because of

filenames.extend( os.listdir( sourcepath ) )

but I was more concerned about how you increase n. I presume that's from

files.append( sourcepath )

Your original code also had some issues, e.g one of your files.append( sourcepath ) should have been files.append( fullpath ), and filenames = [ '.' ] causes infinite recursion. You also never returned your files.

You should really check out https://github.com/tqdm/tqdm/wiki/How-to-make-a-great-Progress-Bar

But for your specific example...

from tqdm import tqdm
import os.path

def find_files_recursively(sourcepath, show_progress=True):
    files = []
    # total=1 assumes sourcepath is a file
    with tqdm(total=1, desc="Searching", unit="file", disable=not show_progress) as myprogressbar:
        def append_found_file(fullpath):
            files.append(fullpath)
            #from time import sleep; sleep(0.1)  # for testing in small dirs
            myprogressbar.update()

        def list_found_dir(path):
            """returns os.listdir(path) assuming os.path.isdir(path)"""
            listing = os.listdir(path)
            # we subtract 1 since a "file" we found was actually this "path" (a directory)
            myprogressbar.total += len(listing) - 1
            myprogressbar.set_postfix(dir=path[-7:])  # fancy way to trigger refresh & give info
            # myprogressbar.update(0)  # alternative way to trigger a refresh without info
            return listing

        if os.path.isdir(sourcepath):
            def recursively_search(sourcepath):
                filenames = list_found_dir(sourcepath)

                for somefile in filenames:
                    fullpath = os.path.join(sourcepath, somefile)

                    if os.path.isdir(fullpath):
                        recursively_search(fullpath)
                    else:
                        append_found_file(fullpath)

            recursively_search(sourcepath)

        elif os.path.exists(sourcepath):
            append_found_file(sourcepath)
    return files

I'm going to add this to the documentation as a nice example I think.