Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BigQuery Beta 2 Changes #4245

Merged
merged 162 commits into from
Oct 24, 2017
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
162 commits
Select commit Hold shift + click to select a range
81ffeb2
Rename job classes (#3797)
tseaver Aug 18, 2017
63e0ebe
Rename class: 'dataset.AccessGrant' -> 'dataset.AccessEntry'. (#3798)
tseaver Aug 18, 2017
040a39e
Add 'QueryJob.query_plan' property. (#3799)
tseaver Sep 7, 2017
2583611
Add new scalar statistics properties to 'QueryJob' (#3800)
tseaver Sep 7, 2017
d4e2feb
Add 'QueryJob.referenced_tables' property. (#3801)
tseaver Sep 8, 2017
daff546
Add 'QueryJob.undeclared_query_parameters' property. (#3802)
tseaver Sep 8, 2017
0379e77
Fix system test broken by PR #3798. (#3936)
tseaver Sep 8, 2017
0aac7a0
Add 'Client.get_job' API wrapper. (#3804)
tseaver Sep 11, 2017
3a1c0fb
Add 'ExtractTableStorageJob.destination_uri_file_counts' property. (#…
tseaver Sep 11, 2017
f17bb9c
bigquery add DatasetReference class and tests (#3938)
alixhami Sep 11, 2017
10f5a2d
BigQuery: Add TestReference class. Add table function to DatasetRefer…
alixhami Sep 13, 2017
ec90007
BigQuery: Modify client.dataset() to return DatasetReference instead …
alixhami Sep 13, 2017
a71a8c3
bigquery: rename TableReference.dataset_ref (#3953)
jba Sep 13, 2017
4aba841
bigquery: rename name field of Dataset to dataset_id (#3955)
jba Sep 14, 2017
6f1c8f3
bigquery: rename name field of Table to table_id (#3959)
jba Sep 14, 2017
6923e8e
BQ: rename XJob.name to XJob.job_id. (#3962)
tswast Sep 15, 2017
2ac3cd6
Parse timestamps in query parameters using canonical format (#3945)
tswast Sep 15, 2017
4e94c48
BigQuery: Adds client.get_dataset() and removes dataset.reload() (#3973)
alixhami Sep 15, 2017
e79384f
BigQuery: Changes DatasetReference project_id property to project to …
alixhami Sep 18, 2017
bef85b7
bigquery: add client.create_dataset; remove dataset.create (#3982)
jba Sep 18, 2017
ecf88e4
bigquery: remove dataset.exists (#3996)
jba Sep 18, 2017
01b812b
BigQuery: Updates Table constructor to use TableReference as paramete…
alixhami Sep 19, 2017
3b81a14
BQ: client.extract_table starts extract job (#3991)
tswast Sep 19, 2017
1f4381f
BigQuery: Adds client.get_table() and removes table.reload() (#4004)
alixhami Sep 20, 2017
04da9c0
bigquery: add Client.update_dataset (#4003)
jba Sep 20, 2017
970f6f7
BigQuery: Remove dataset property from TableReference and add project…
alixhami Sep 20, 2017
e5b91ea
bigquery: add client.delete_dataset (#4012)
jba Sep 20, 2017
31a6219
updates dataset.table() to return a TableReference instead of a Table…
alixhami Sep 21, 2017
97f4e1c
bigquery: add client.list_dataset_tables (#4013)
jba Sep 21, 2017
a8169a4
bigquery: remove client from Dataset (#4018)
jba Sep 21, 2017
5779792
bigquery: dataset labels (#4026)
jba Sep 22, 2017
d643b6e
bigquery: Dataset constructor takes a DatasetReference (#4036)
jba Sep 22, 2017
f6d7582
bigquery: Job and Query default_dataset is DatasetReference (#4037)
jba Sep 22, 2017
ce9a06c
BigQuery: add to/from API representation for Table & Dataset referenc…
tswast Sep 22, 2017
1036b82
BigQuery: replaces table.create() with client.create_table() (#4038)
alixhami Sep 22, 2017
b0b77c3
BigQuery: Remove unnecessary line from client.create_table() test (#4…
alixhami Sep 23, 2017
6890ebe
bigquery: add system test for dataset update with etag (#4052)
jba Sep 25, 2017
ae155bb
BQ: remove redundant __eq__ from Query Parameters. (#4055)
tswast Sep 25, 2017
f38f447
bigquery: modify CopyJob (#4051)
jba Sep 25, 2017
722a831
bigquery: fix copy job system test (#4059)
jba Sep 26, 2017
9ab2715
BigQuery: replaces table.delete() with client.delete_table() (#4066)
alixhami Sep 27, 2017
0832c6f
BigQuery: add client.query_rows(), remove client.run_sync_query(). (#…
tswast Sep 27, 2017
f90bbb7
bigquery: modify LoadJob (#4103)
jba Oct 4, 2017
33244cf
Make QueryResults read-only. (#4094)
tswast Oct 6, 2017
ccf0ec2
bigquery: add Client.list_rows, remove Table.fetch_data (#4119)
jba Oct 6, 2017
0ddee3a
BigQuery: replaces table.update() and table.patch() with client.updat…
alixhami Oct 9, 2017
3a324ad
bigquery: add config getters to LoadJob (#4137)
jba Oct 9, 2017
f352eea
BQ: remove client.run_async_query, add client.query (#4130)
tswast Oct 9, 2017
e329a19
bigquery: add Client.load_table_from_file (#4136)
jba Oct 9, 2017
f72a691
BQ: Pass selected_fields as a string to tabledata.list. (#4143)
tswast Oct 9, 2017
d1b564d
bigquery: remove Table.exists (#4145)
jba Oct 9, 2017
112b441
bigquery: remove unused function (#4147)
jba Oct 10, 2017
5df465f
bigquery: add Client.list_partitions (#4146)
jba Oct 10, 2017
3f65b7c
BQ: remove fetch_data and client references from QueryResults (#4144)
tswast Oct 10, 2017
c2d84d9
bigquery: add Row class (#4149)
jba Oct 10, 2017
0313a00
BQ: move *QueryParameter and UDFResource classes to query. (#4156)
tswast Oct 11, 2017
b7f916c
BigQuery: Replace table.insert_data() with client.create_rows() (#4151)
alixhami Oct 11, 2017
53e9a44
BQ: remove unused ConfigurationProperty classes. (#4157)
tswast Oct 11, 2017
02b0069
BigQuery: removes Client from Table class (#4159)
alixhami Oct 11, 2017
3ef7386
bigquery: add streaming buffer info (#4161)
jba Oct 12, 2017
4e71918
bigquery: factor out common values in test_client.py (#4162)
jba Oct 12, 2017
a06b84b
BQ: use random job ID for system tests. (#4163)
tswast Oct 12, 2017
d7ad25e
bigquery: generate row IDs in create_rows (#4173)
jba Oct 12, 2017
fd691a2
bigquery: retry idempotent RPCs (#4148)
jba Oct 12, 2017
14b3760
BQ: use a string or list of string for client.extract_table. (#4177)
tswast Oct 12, 2017
e6cc4b4
bigquery: retry Client.create_rows (#4178)
jba Oct 13, 2017
0372544
bigquery: classes to support external table config (#4182)
jba Oct 14, 2017
2c71305
BigQuery: adds client.create_rows_json (#4189)
alixhami Oct 16, 2017
2830480
bigquery: make views default to standard SQL (#4192)
jba Oct 16, 2017
a97bd6b
BQ: update samples in README for beta 2 changes. (#4179)
tswast Oct 16, 2017
8360b6a
bigquery: support external table definitions for query jobs (#4191)
jba Oct 16, 2017
89c479f
bigquery: support external data definition for tables (#4193)
jba Oct 16, 2017
297d397
bigquery: support update of dataset access entries (#4197)
jba Oct 16, 2017
fc5e051
biqquery: factor out common values in system.py (#4194)
jba Oct 16, 2017
8e9aa23
BigQuery: option for job ID generation with user-supplied prefix (#4198)
alixhami Oct 16, 2017
bc520a8
Rename job classes (#3797)
tseaver Aug 18, 2017
d6504ad
Rename class: 'dataset.AccessGrant' -> 'dataset.AccessEntry'. (#3798)
tseaver Aug 18, 2017
828ad3a
Add 'QueryJob.query_plan' property. (#3799)
tseaver Sep 7, 2017
9103ba0
Add new scalar statistics properties to 'QueryJob' (#3800)
tseaver Sep 7, 2017
d20afc4
Add 'QueryJob.referenced_tables' property. (#3801)
tseaver Sep 8, 2017
8f2941f
Add 'QueryJob.undeclared_query_parameters' property. (#3802)
tseaver Sep 8, 2017
bf2dcb2
Fix system test broken by PR #3798. (#3936)
tseaver Sep 8, 2017
f2478ba
Add 'Client.get_job' API wrapper. (#3804)
tseaver Sep 11, 2017
3b777dd
Add 'ExtractTableStorageJob.destination_uri_file_counts' property. (#…
tseaver Sep 11, 2017
4aed26c
bigquery add DatasetReference class and tests (#3938)
alixhami Sep 11, 2017
ada8a31
BigQuery: Add TestReference class. Add table function to DatasetRefer…
alixhami Sep 13, 2017
f17942d
BigQuery: Modify client.dataset() to return DatasetReference instead …
alixhami Sep 13, 2017
4dfdaa7
bigquery: rename TableReference.dataset_ref (#3953)
jba Sep 13, 2017
24c9313
bigquery: rename name field of Dataset to dataset_id (#3955)
jba Sep 14, 2017
8abc913
bigquery: rename name field of Table to table_id (#3959)
jba Sep 14, 2017
26247da
BQ: rename XJob.name to XJob.job_id. (#3962)
tswast Sep 15, 2017
2df2990
Parse timestamps in query parameters using canonical format (#3945)
tswast Sep 15, 2017
e9bb79a
BigQuery: Adds client.get_dataset() and removes dataset.reload() (#3973)
alixhami Sep 15, 2017
4032d0b
BigQuery: Changes DatasetReference project_id property to project to …
alixhami Sep 18, 2017
4f79432
bigquery: add client.create_dataset; remove dataset.create (#3982)
jba Sep 18, 2017
6e8f3b9
bigquery: remove dataset.exists (#3996)
jba Sep 18, 2017
37a6b23
BigQuery: Updates Table constructor to use TableReference as paramete…
alixhami Sep 19, 2017
3488f1c
BQ: client.extract_table starts extract job (#3991)
tswast Sep 19, 2017
465e523
BigQuery: Adds client.get_table() and removes table.reload() (#4004)
alixhami Sep 20, 2017
a200750
bigquery: add Client.update_dataset (#4003)
jba Sep 20, 2017
ca10fee
BigQuery: Remove dataset property from TableReference and add project…
alixhami Sep 20, 2017
6aab47a
bigquery: add client.delete_dataset (#4012)
jba Sep 20, 2017
15b3d73
updates dataset.table() to return a TableReference instead of a Table…
alixhami Sep 21, 2017
82c401b
bigquery: add client.list_dataset_tables (#4013)
jba Sep 21, 2017
30205ed
bigquery: remove client from Dataset (#4018)
jba Sep 21, 2017
2b3d68b
bigquery: dataset labels (#4026)
jba Sep 22, 2017
fefe2ea
bigquery: Dataset constructor takes a DatasetReference (#4036)
jba Sep 22, 2017
3dc30d0
bigquery: Job and Query default_dataset is DatasetReference (#4037)
jba Sep 22, 2017
1df6975
BigQuery: add to/from API representation for Table & Dataset referenc…
tswast Sep 22, 2017
26f4fa7
BigQuery: replaces table.create() with client.create_table() (#4038)
alixhami Sep 22, 2017
ce8a1dc
BigQuery: Remove unnecessary line from client.create_table() test (#4…
alixhami Sep 23, 2017
83319e1
bigquery: add system test for dataset update with etag (#4052)
jba Sep 25, 2017
ed8b2fe
BQ: remove redundant __eq__ from Query Parameters. (#4055)
tswast Sep 25, 2017
ed06f4b
bigquery: modify CopyJob (#4051)
jba Sep 25, 2017
bfa18c3
bigquery: fix copy job system test (#4059)
jba Sep 26, 2017
de03ec9
BigQuery: replaces table.delete() with client.delete_table() (#4066)
alixhami Sep 27, 2017
d3712c2
BigQuery: add client.query_rows(), remove client.run_sync_query(). (#…
tswast Sep 27, 2017
b649544
bigquery: modify LoadJob (#4103)
jba Oct 4, 2017
05781ea
Make QueryResults read-only. (#4094)
tswast Oct 6, 2017
3a99312
bigquery: add Client.list_rows, remove Table.fetch_data (#4119)
jba Oct 6, 2017
a3d7465
BigQuery: replaces table.update() and table.patch() with client.updat…
alixhami Oct 9, 2017
010c254
bigquery: add config getters to LoadJob (#4137)
jba Oct 9, 2017
de2a3b6
BQ: remove client.run_async_query, add client.query (#4130)
tswast Oct 9, 2017
9876ad6
bigquery: add Client.load_table_from_file (#4136)
jba Oct 9, 2017
f7a52cb
BQ: Pass selected_fields as a string to tabledata.list. (#4143)
tswast Oct 9, 2017
76d303f
bigquery: remove Table.exists (#4145)
jba Oct 9, 2017
74af691
bigquery: remove unused function (#4147)
jba Oct 10, 2017
7faa1b5
bigquery: add Client.list_partitions (#4146)
jba Oct 10, 2017
868f7c7
BQ: remove fetch_data and client references from QueryResults (#4144)
tswast Oct 10, 2017
79e69c8
bigquery: add Row class (#4149)
jba Oct 10, 2017
23afc0c
BQ: move *QueryParameter and UDFResource classes to query. (#4156)
tswast Oct 11, 2017
27bc0b2
BigQuery: Replace table.insert_data() with client.create_rows() (#4151)
alixhami Oct 11, 2017
4afa7ba
BQ: remove unused ConfigurationProperty classes. (#4157)
tswast Oct 11, 2017
8e7d88e
BigQuery: removes Client from Table class (#4159)
alixhami Oct 11, 2017
4922316
bigquery: add streaming buffer info (#4161)
jba Oct 12, 2017
61acc32
bigquery: factor out common values in test_client.py (#4162)
jba Oct 12, 2017
f5bf7a1
BQ: use random job ID for system tests. (#4163)
tswast Oct 12, 2017
a9a0b30
bigquery: generate row IDs in create_rows (#4173)
jba Oct 12, 2017
0209991
bigquery: retry idempotent RPCs (#4148)
jba Oct 12, 2017
e031a29
BQ: use a string or list of string for client.extract_table. (#4177)
tswast Oct 12, 2017
053a6ba
bigquery: retry Client.create_rows (#4178)
jba Oct 13, 2017
c70d6e4
bigquery: classes to support external table config (#4182)
jba Oct 14, 2017
e29e93b
BigQuery: adds client.create_rows_json (#4189)
alixhami Oct 16, 2017
8c8a0bc
bigquery: make views default to standard SQL (#4192)
jba Oct 16, 2017
ab392d5
BQ: update samples in README for beta 2 changes. (#4179)
tswast Oct 16, 2017
7ea861f
bigquery: support external table definitions for query jobs (#4191)
jba Oct 16, 2017
3af22d7
bigquery: support external data definition for tables (#4193)
jba Oct 16, 2017
a7e627c
bigquery: support update of dataset access entries (#4197)
jba Oct 16, 2017
9f095bf
biqquery: factor out common values in system.py (#4194)
jba Oct 16, 2017
112f6e3
BigQuery: option for job ID generation with user-supplied prefix (#4198)
alixhami Oct 16, 2017
56216a7
BigQuery: skip snippets tests until they can be updated.
tswast Oct 16, 2017
9f4bd6c
Merge remote-tracking branch 'upstream/bigquery-b2' into bigquery-b2
tswast Oct 16, 2017
78df6e6
bigquery: add filter to list_datasets (#4205)
jba Oct 16, 2017
6dae47f
bigquery: support table labels (#4207)
jba Oct 16, 2017
f192cb6
BigQuery: removes LoadJob error for autodetect + schema (#4213)
alixhami Oct 18, 2017
af3da30
BigQuery: populate timeout parameter for getQueryResults (#4209)
tswast Oct 18, 2017
c82d512
BigQuery: Adds helper function for snake to camel case conversion (#4…
alixhami Oct 18, 2017
d19b6c3
Renames client.load_table_from_storage() to client.load_table_from_ur…
alixhami Oct 20, 2017
4ce4a90
BigQuery: Updates snippets for BigQuery Beta 2 changes (#4237)
alixhami Oct 23, 2017
c35eae6
Merge remote-tracking branch 'upstream/master' into bigquery-b2
tswast Oct 23, 2017
61bb7a0
BigQuery: make docstrings use bigquery module, like the samples do. (…
tswast Oct 23, 2017
df016c2
BigQuery: Make job.begin() method private. (#4242)
tswast Oct 24, 2017
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
2 changes: 2 additions & 0 deletions bigquery/.coveragerc
Original file line number Diff line number Diff line change
Expand Up @@ -9,3 +9,5 @@ exclude_lines =
pragma: NO COVER
# Ignore debug-only repr
def __repr__
# Ignore abstract methods
raise NotImplementedError
61 changes: 40 additions & 21 deletions bigquery/README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -38,12 +38,27 @@ Using the API

Querying massive datasets can be time consuming and expensive without the
right hardware and infrastructure. Google `BigQuery`_ (`BigQuery API docs`_)
solves this problem by enabling super-fast, SQL-like queries against
append-only tables, using the processing power of Google's infrastructure.
solves this problem by enabling super-fast, SQL queries against
append-mostly tables, using the processing power of Google's infrastructure.

.. _BigQuery: https://cloud.google.com/bigquery/what-is-bigquery
.. _BigQuery API docs: https://cloud.google.com/bigquery/docs/reference/v2/

Create a dataset
~~~~~~~~~~~~~~~~

.. code:: python

from google.cloud import bigquery
from google.cloud.bigquery import Dataset

client = bigquery.Client()

dataset_ref = client.dataset('dataset_name')
dataset = Dataset(dataset_ref)
dataset.description = 'my dataset'
dataset = client.create_dataset(dataset) # API request

Load data from CSV
~~~~~~~~~~~~~~~~~~

Expand All @@ -52,39 +67,43 @@ Load data from CSV
import csv

from google.cloud import bigquery
from google.cloud.bigquery import LoadJobConfig
from google.cloud.bigquery import SchemaField

client = bigquery.Client()

dataset = client.dataset('dataset_name')
dataset.create() # API request

SCHEMA = [
SchemaField('full_name', 'STRING', mode='required'),
SchemaField('age', 'INTEGER', mode='required'),
]
table = dataset.table('table_name', SCHEMA)
table.create()
table_ref = client.dataset('dataset_name').table('table_name')

load_config = LoadJobConfig()
load_config.skip_leading_rows = 1
load_config.schema = SCHEMA

with open('csv_file', 'rb') as readable:
table.upload_from_file(
readable, source_format='CSV', skip_leading_rows=1)
# Contents of csv_file.csv:
# Name,Age
# Tim,99
with open('csv_file.csv', 'rb') as readable:
client.load_table_from_file(
readable, table_ref, job_config=load_config) # API request

Perform a synchronous query
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Perform a query
~~~~~~~~~~~~~~~

.. code:: python

# Perform a synchronous query.
# Perform a query.
QUERY = (
'SELECT name FROM [bigquery-public-data:usa_names.usa_1910_2013] '
'WHERE state = "TX"')
query = client.run_sync_query('%s LIMIT 100' % QUERY)
query.timeout_ms = TIMEOUT_MS
query.run()

for row in query.rows:
print(row)
'SELECT name FROM `bigquery-public-data.usa_names.usa_1910_2013` '
'WHERE state = "TX" '
'LIMIT 100')
query_job = client.query(QUERY) # API request
rows = query_job.result() # Waits for query to finish

for row in rows:
print(row.name)


See the ``google-cloud-python`` API `BigQuery documentation`_ to learn how
Expand Down
58 changes: 52 additions & 6 deletions bigquery/google/cloud/bigquery/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,17 +26,63 @@
from pkg_resources import get_distribution
__version__ = get_distribution('google-cloud-bigquery').version

from google.cloud.bigquery._helpers import ArrayQueryParameter
from google.cloud.bigquery._helpers import ScalarQueryParameter
from google.cloud.bigquery._helpers import StructQueryParameter
from google.cloud.bigquery._helpers import DEFAULT_RETRY
from google.cloud.bigquery.client import Client
from google.cloud.bigquery.dataset import AccessGrant
from google.cloud.bigquery.dataset import AccessEntry
from google.cloud.bigquery.dataset import Dataset
from google.cloud.bigquery.dataset import DatasetReference
from google.cloud.bigquery.job import CopyJob
from google.cloud.bigquery.job import CopyJobConfig
from google.cloud.bigquery.job import ExtractJob
from google.cloud.bigquery.job import ExtractJobConfig
from google.cloud.bigquery.job import QueryJob
from google.cloud.bigquery.job import QueryJobConfig
from google.cloud.bigquery.job import LoadJob
from google.cloud.bigquery.job import LoadJobConfig
from google.cloud.bigquery.query import ArrayQueryParameter
from google.cloud.bigquery.query import ScalarQueryParameter
from google.cloud.bigquery.query import StructQueryParameter
from google.cloud.bigquery.query import UDFResource
from google.cloud.bigquery.schema import SchemaField
from google.cloud.bigquery.table import Table
from google.cloud.bigquery.table import TableReference
from google.cloud.bigquery.external_config import ExternalConfig
from google.cloud.bigquery.external_config import BigtableOptions
from google.cloud.bigquery.external_config import BigtableColumnFamily
from google.cloud.bigquery.external_config import BigtableColumn
from google.cloud.bigquery.external_config import CSVOptions
from google.cloud.bigquery.external_config import GoogleSheetsOptions

__all__ = [
'__version__', 'AccessGrant', 'ArrayQueryParameter', 'Client',
'Dataset', 'ScalarQueryParameter', 'SchemaField', 'StructQueryParameter',
'__version__',
'Client',
# Queries
'QueryJob',
'QueryJobConfig',
'ArrayQueryParameter',
'ScalarQueryParameter',
'StructQueryParameter',
# Datasets
'Dataset',
'DatasetReference',
'AccessEntry',
# Tables
'Table',
'TableReference',
'CopyJob',
'CopyJobConfig',
'ExtractJob',
'ExtractJobConfig',
'LoadJob',
'LoadJobConfig',
# Shared helpers
'SchemaField',
'UDFResource',
'ExternalConfig',
'BigtableOptions',
'BigtableColumnFamily',
'BigtableColumn',
'CSVOptions',
'GoogleSheetsOptions',
'DEFAULT_RETRY',
]