Enable data import from Google Cloud Storage files #17
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The import_data_from_uris method imports data from Google Cloud Storage. The wait_for_job method takes a job resource or jobId and waits until the job completes, polling with a user-specified interval and timing out after a different user-specified interval.
Ensure that wait_for_job accepts a jobId.
Got rid of the default values in import_data_from_uri. If values are not provided, they are not inserted into the load configuration. Also fixed the structure of the load configuration JSON. Still need to write tests.
Previous commits fixed the client after testing directly with BigQuery. This commit fixes the tests to reflect the actual return values from BigQuery, and adds tests for parameters to import_data_from_uris, some of which are not permitted when source_format is not "CSV".
bigquery/client.py
Outdated
There was a problem hiding this comment.
I realize the None values are being used in a falsey way, but the defaults should probably be False to be more explicit to users that these are bool values. Same goes for the other bool kwargs.
Optional parameters are not injected into the job configuration, default values are given by BigQuery. The docstring now correctly explains this. Also shortened some constants.
|
Looks great! I will cut a new release shortly. |
tylertreat
added a commit
that referenced
this pull request
Aug 26, 2014
Enable data import from Google Cloud Storage files
|
Awesome. TY. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Added two client methods which enable importing data from Google Cloud Storage files, and waiting until the job is complete:
These methods have been tested against live Google Cloud Storage and BigQuery instances, and are currently being used in production.