Skip to content

Add progress for to_gbq function #166

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Apr 27, 2018
Merged

Add progress for to_gbq function #166

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
8b00e9c
Add progress for to_gbq function using tqdm
Apr 19, 2018
3a5dc2c
Add tqdm addition to changelog
aktech Apr 27, 2018
44e7cac
Add issue number to changelog for progress bar
tswast Apr 27, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions docs/source/changelog.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,9 @@ Changelog
- Project ID parameter is optional in ``read_gbq`` and ``to_gbq`` when it can
inferred from the environment. Note: you must still pass in a project ID when
using user-based authentication. (:issue:`103`)
- Progress bar added for ``to_gbq``, through an optional library `tqdm` as
dependency. (:issue:`162`)


Internal changes
~~~~~~~~~~~~~~~~
Expand Down
22 changes: 16 additions & 6 deletions pandas_gbq/gbq.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,11 @@
BIGQUERY_INSTALLED_VERSION = None
SHOW_VERBOSE_DEPRECATION = False

try:
import tqdm # noqa
except ImportError:
tqdm = None


def _check_google_client_version():
global BIGQUERY_INSTALLED_VERSION, SHOW_VERBOSE_DEPRECATION
Expand Down Expand Up @@ -563,16 +568,19 @@ def run_query(self, query, **kwargs):

def load_data(
self, dataframe, dataset_id, table_id, chunksize=None,
schema=None):
schema=None, progress_bar=True):
from pandas_gbq import load

total_rows = len(dataframe)
logger.info("\n\n")

try:
for remaining_rows in load.load_chunks(
self.client, dataframe, dataset_id, table_id,
chunksize=chunksize, schema=schema):
chunks = load.load_chunks(self.client, dataframe, dataset_id,
table_id, chunksize=chunksize,
schema=schema)
if progress_bar and tqdm:
chunks = tqdm.tqdm(chunks)
for remaining_rows in chunks:
logger.info("\rLoad is {0}% Complete".format(
Copy link
Contributor

@max-sixty max-sixty Apr 27, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought we were going to either tqdm or log, rather than both?
But future PR is OK if you want to get this in

Copy link
Contributor Author

@aktech aktech Apr 27, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, we can do it in another PR.

((total_rows - remaining_rows) * 100) / total_rows))
except self.http_error as ex:
Expand Down Expand Up @@ -870,7 +878,7 @@ def read_gbq(query, project_id=None, index_col=None, col_order=None,

def to_gbq(dataframe, destination_table, project_id=None, chunksize=None,
verbose=None, reauth=False, if_exists='fail', private_key=None,
auth_local_webserver=False, table_schema=None):
auth_local_webserver=False, table_schema=None, progress_bar=True):
"""Write a DataFrame to a Google BigQuery table.

The main method a user calls to export pandas DataFrame contents to
Expand Down Expand Up @@ -935,6 +943,8 @@ def to_gbq(dataframe, destination_table, project_id=None, chunksize=None,
names of a field.
.. versionadded:: 0.3.1
verbose : None, deprecated
progress_bar : boolean, True by default. It uses the library `tqdm` to show
the progress bar for the upload, chunk by chunk.
"""

_test_google_api_imports()
Expand Down Expand Up @@ -987,7 +997,7 @@ def to_gbq(dataframe, destination_table, project_id=None, chunksize=None,

connector.load_data(
dataframe, dataset_id, table_id, chunksize=chunksize,
schema=table_schema)
schema=table_schema, progress_bar=progress_bar)


def generate_bq_schema(df, default_type='STRING'):
Expand Down
1 change: 1 addition & 0 deletions requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,3 +2,4 @@ pandas
google-auth
google-auth-oauthlib
google-cloud-bigquery
tqdm
Copy link
Collaborator

@tswast tswast Apr 19, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

requirements.txt is only used by developers / CI testing. You need to add this to setup.py for it to actually get listed as a dependency.

As @maxim-lian says, we could make this an optional dependency. If you go that route, use the "extras" section of setup.py (example.

4 changes: 4 additions & 0 deletions setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,9 @@ def readme():
'google-cloud-bigquery>=0.29.0',
]

extras = {
'tqdm': 'tqdm>=4.23.0',
}

setup(
name=NAME,
Expand All @@ -50,6 +53,7 @@ def readme():
],
keywords='data',
install_requires=INSTALL_REQUIRES,
extras_require=extras,
packages=find_packages(exclude=['contrib', 'docs', 'tests*']),
test_suite='tests',
)