tqdm deprecation warning

[GergelyKalmar]

Environment details

• OS type and version: Ubuntu 20.04 LTS
• Python version: 3.8.10
• pip version: 22.0.3
• google-cloud-bigquery version: 2.34.0

Steps to reproduce

from google.cloud import bigquery

query = bigquery.Client().query('SELECT 1')
data = query.to_dataframe(progress_bar_type='tqdm_notebook')

The snippet above causes a deprecation warning that breaks strict pytest runs (using 'filterwarnings': ['error']):

.../site-packages/google/cloud/bigquery/_tqdm_helpers.py:51: in get_progress_bar
    return tqdm.tqdm_notebook(desc=description, total=total, unit=unit)
...
        from .notebook import tqdm as _tqdm_notebook
>       warn("This function will be removed in tqdm==5.0.0\n"
             "Please use `tqdm.notebook.tqdm` instead of `tqdm.tqdm_notebook`",
             TqdmDeprecationWarning, stacklevel=2)
E       tqdm.std.TqdmDeprecationWarning: This function will be removed in tqdm==5.0.0
E       Please use `tqdm.notebook.tqdm` instead of `tqdm.tqdm_notebook`

.../site-packages/tqdm/__init__.py:25: TqdmDeprecationWarning

[tswast]

Thanks for bringing this to our attention! Seeing as we set the max tqdm version to <5.0.0, I'll mark this as "process" for now, but I do think we need to do this before tqdm 5.0.0 comes out since "extras" aren't always enforced by PyPI and in principle we should support the latest version of our dependencies.

[GergelyKalmar]

Thank you! It seems that this has been deprecated for years now though, the commit adding the deprecation warning is from 2019: tqdm/tqdm@49471d3#diff-c2228f13f6b0bdaf4c1012a108f937d0e0eceece0d87e60b0b3c3172f9038bedR5. It seems to be a relatively simple change, would it be possible to increase the priority a bit?

[GergelyKalmar]

🎉 Thank you!

[GergelyKalmar]

@chalmerlowe This feature seems to be completely broken now. With google-cloud-bigquery[pandas,tqdm]==3.3.5 and the reproducer above I'm getting:

Traceback (most recent call last):
  File "test.py", line 4, in <module>
    data = query.to_dataframe(progress_bar_type='tqdm_notebook')
  File ".../bigquery-test/lib/python3.8/site-packages/google/cloud/bigquery/job/query.py", line 1683, in to_dataframe
    query_result = wait_for_query(self, progress_bar_type, max_results=max_results)
  File ".../bigquery-test/lib/python3.8/site-packages/google/cloud/bigquery/_tqdm_helpers.py", line 85, in wait_for_query
    progress_bar = get_progress_bar(
  File ".../bigquery-test/lib/python3.8/site-packages/google/cloud/bigquery/_tqdm_helpers.py", line 52, in get_progress_bar
    return tqdm.notebook.tqdm(desc=description, total=total, unit=unit)
AttributeError: module 'tqdm' has no attribute 'notebook'

Note that import tqdm; tqdm.notebook.tqdm errors out, however from tqdm.notebook import tqdm works fine.

This is a pretty serious regression, would you mind issuing a patch whenever you can? Thank you!

[chalmerlowe]

@aribray can you take a look at this?

[aribray]

Hi @GergelyKalmar, can you pull down the latest version and send me the results of the to_dataframe() call?

[GergelyKalmar]

hi @aribray, looks like the issue is now solved! Thank you very much for that.

I only have one nitpick regarding the new progress bar setup: it's a bit weird to have two progress bars, and one of them is pushed to the right when the job ID appears:

I suppose it would be better to have just a single progress bar on its own line (job ID should be its own line too).

If you want to make the output more complete, I've been also showing the data processed at the very end in my own bigquery magic like this:

from humanize.filesize import naturalsize
...
if args.info:
    processed = naturalsize(query.total_bytes_processed)
    display(ipywidgets.HTML(value='Data processed: ' + (
        f'<b>{processed}</b>' if not query.cache_hit
        else 'none <b>(returned from cache)</b>'
    )))

With this addition the output looks like this:

It might look better if you add a colon after the "Job ID" too, but that's just my subjective preference.

[meredithslota]

Looks like the original bug has been fixed but there's a remaining feature request for improving the progress bar setup; I'm going to copy the last comment to a new issue so we can examine the request but also close out this bug. :)