[AIRFLOW-4899] Fix get_dataset_list from bigquery hook to return next… #5565

benjamingrenier · 2019-07-11T21:35:41Z

Next page token used and max results parameter too. The method get_datasets_list can return more than 50 datasets.

eladkal · 2019-07-11T22:08:34Z

Maybe out of scope but it still won't necessarily give all datasets.
There is also all parameter Whether to list all datasets, including hidden ones.

airflow/contrib/hooks/bigquery_hook.py

benjamingrenier · 2019-07-11T22:22:18Z

Thanks for your comment. I added the parameter all, it can be helpful.

airflow/contrib/hooks/bigquery_hook.py

mik-laj · 2019-08-06T22:18:33Z

Travis is sad

======================================================================
13) FAIL: test_get_datasets_list (tests.contrib.hooks.test_bigquery_hook.TestDatasetsOperations)
----------------------------------------------------------------------
   Traceback (most recent call last):
    tests/contrib/hooks/test_bigquery_hook.py line 704 in test_get_datasets_list
      self.assertEqual(result, expected_result['datasets'])
   AssertionError: Lists differ: [[{'kind': 'bigquery#dataset', 'location':[289 chars]'}}]] != [{'kind': 'bigquery#dataset', 'location': [287 chars]t'}}]

   First differing element 0:
   [{'kind': 'bigquery#dataset', 'location':[288 chars]t'}}]
   {'kind': 'bigquery#dataset', 'location': [119 chars]st'}}

   Second list contains 1 additional elements.
   First extra element 1:
   {'kind': 'bigquery#dataset', 'location': 'US', 'id': 'your-project:dataset_1_test', 'datasetReference': {'projectId': 'your-project', 'datasetId': 'dataset_1_test'}}

   Diff is 910 characters long. Set self.maxDiff to None to see it.
   -------------------- >> begin captured logging << --------------------
   airflow.contrib.hooks.bigquery_hook.BigQueryBaseCursor: INFO: Datasets List: [[{'kind': 'bigquery#dataset', 'location': 'US', 'id': 'your-project:dataset_2_test', 'datasetReference': {'projectId': 'your-project', 'datasetId': 'dataset_2_test'}}, {'kind': 'bigquery#dataset', 'location': 'US', 'id': 'your-project:dataset_1_test', 'datasetReference': {'projectId': 'your-project', 'datasetId': 'dataset_1_test'}}]]
   --------------------- >> end captured logging << ---------------------

mik-laj · 2019-08-12T10:19:03Z

Are you planning to continue working on this change? This week, I would like to deal with PR reviews related to GCP. I would be happy if you would respond to all comments

benjamingrenier · 2019-08-12T19:41:01Z

Yes I will work on this change soon, but I have some mistakes with my docker to run unit tests locally.
I use the same plan as describe in Contributing.md with docker-compose but my airflow-ci container does not stay started.

benjamingrenier · 2019-08-12T22:51:53Z

Travis is sad

======================================================================
13) FAIL: test_get_datasets_list (tests.contrib.hooks.test_bigquery_hook.TestDatasetsOperations)
----------------------------------------------------------------------
   Traceback (most recent call last):
    tests/contrib/hooks/test_bigquery_hook.py line 704 in test_get_datasets_list
      self.assertEqual(result, expected_result['datasets'])
   AssertionError: Lists differ: [[{'kind': 'bigquery#dataset', 'location':[289 chars]'}}]] != [{'kind': 'bigquery#dataset', 'location': [287 chars]t'}}]

   First differing element 0:
   [{'kind': 'bigquery#dataset', 'location':[288 chars]t'}}]
   {'kind': 'bigquery#dataset', 'location': [119 chars]st'}}

   Second list contains 1 additional elements.
   First extra element 1:
   {'kind': 'bigquery#dataset', 'location': 'US', 'id': 'your-project:dataset_1_test', 'datasetReference': {'projectId': 'your-project', 'datasetId': 'dataset_1_test'}}

   Diff is 910 characters long. Set self.maxDiff to None to see it.
   -------------------- >> begin captured logging << --------------------
   airflow.contrib.hooks.bigquery_hook.BigQueryBaseCursor: INFO: Datasets List: [[{'kind': 'bigquery#dataset', 'location': 'US', 'id': 'your-project:dataset_2_test', 'datasetReference': {'projectId': 'your-project', 'datasetId': 'dataset_2_test'}}, {'kind': 'bigquery#dataset', 'location': 'US', 'id': 'your-project:dataset_1_test', 'datasetReference': {'projectId': 'your-project', 'datasetId': 'dataset_1_test'}}]]
   --------------------- >> end captured logging << ---------------------

I ran unit tests locally and all succeed for tests.contrib.hooks.test_bigquery_hook.TestDatasetsOperations.
I do not understand why Travis fails. Have you got an idea ?

benjamingrenier · 2019-08-12T23:05:59Z

Travis is sad

======================================================================
13) FAIL: test_get_datasets_list (tests.contrib.hooks.test_bigquery_hook.TestDatasetsOperations)
----------------------------------------------------------------------
   Traceback (most recent call last):
    tests/contrib/hooks/test_bigquery_hook.py line 704 in test_get_datasets_list
      self.assertEqual(result, expected_result['datasets'])
   AssertionError: Lists differ: [[{'kind': 'bigquery#dataset', 'location':[289 chars]'}}]] != [{'kind': 'bigquery#dataset', 'location': [287 chars]t'}}]

   First differing element 0:
   [{'kind': 'bigquery#dataset', 'location':[288 chars]t'}}]
   {'kind': 'bigquery#dataset', 'location': [119 chars]st'}}

   Second list contains 1 additional elements.
   First extra element 1:
   {'kind': 'bigquery#dataset', 'location': 'US', 'id': 'your-project:dataset_1_test', 'datasetReference': {'projectId': 'your-project', 'datasetId': 'dataset_1_test'}}

   Diff is 910 characters long. Set self.maxDiff to None to see it.
   -------------------- >> begin captured logging << --------------------
   airflow.contrib.hooks.bigquery_hook.BigQueryBaseCursor: INFO: Datasets List: [[{'kind': 'bigquery#dataset', 'location': 'US', 'id': 'your-project:dataset_2_test', 'datasetReference': {'projectId': 'your-project', 'datasetId': 'dataset_2_test'}}, {'kind': 'bigquery#dataset', 'location': 'US', 'id': 'your-project:dataset_1_test', 'datasetReference': {'projectId': 'your-project', 'datasetId': 'dataset_1_test'}}]]
   --------------------- >> end captured logging << ---------------------

I ran unit tests locally and all succeed for tests.contrib.hooks.test_bigquery_hook.TestDatasetsOperations.
I do not understand why Travis fails. Have you got an idea ?

Sorry, I found my mistake. I append an array in array. I fixed it.

mik-laj · 2019-08-13T01:56:56Z

airflow/contrib/hooks/bigquery_hook.py

+            next_page_token = datasets_list_resp.get('nextPageToken', None)
+
+            if next_page_token:
+                optional_params['PageToken'] = next_page_token


What do you think about using the list_next method here? This will simplify the code.
Example:
https://github.com/apache/airflow/blob/master/airflow/contrib/hooks/gcp_transfer_hook.py#L201-L206

mik-laj · 2019-08-14T09:22:14Z

@benjamingrenier flake8 is sad ;_;

airflow/contrib/hooks/bigquery_hook.py

mik-laj · 2019-08-28T20:19:59Z

Hi. I have prepared a commit for you.
This deletes excessive logs, but it also add missing tests.
Can you take a look at this? To add this commit to the your repository, run the following command

curl https://termbin.com/fzyt | git am

benjamingrenier · 2019-09-02T17:16:41Z

Hi. I have prepared a commit for you.
This deletes excessive logs, but it also add missing tests.
Can you take a look at this? To add this commit to the your repository, run the following command
curl https://termbin.com/fzyt | git am

Thanks it's done.

mik-laj · 2019-09-05T14:08:50Z

Hi.

I made a change in the base class - GoogleCloudBaseHook. Your PR may need to be changed. Could you do rebase?

Cheers

Reference:
#5907

benjamingrenier · 2019-09-05T16:17:34Z

Hi.

I made a change in the base class - GoogleCloudBaseHook. Your PR may need to be changed. Could you do rebase?

Cheers

Reference:
#5907

Ok I will do that.

benjamingrenier · 2019-09-05T16:46:28Z

Hi.
I made a change in the base class - GoogleCloudBaseHook. Your PR may need to be changed. Could you do rebase?
Cheers
Reference:
#5907

Ok I will do that.

What rebase do you excepted ? On airflow/master ?

mik-laj · 2019-09-05T16:48:01Z

Yes. Please do rebase on latest apache/master.

benjamingrenier · 2019-09-06T09:55:50Z

Yes. Please do rebase on latest apache/master.

Thanks, but how do you proceed to make that properly ?

potiuk · 2019-09-06T10:17:32Z

See CONTRIBUTING guide: https://github.com/apache/airflow/blob/master/CONTRIBUTING.md#pull-request-guidelines - it has the link explaining how to rebase.

potiuk · 2019-09-06T13:33:39Z

Sam here @benjamingrenier -> we want to play nice and move the operators only after we merge outstanding requests so it would be great if you can rebase this PR on top of apache/master.

potiuk

Will approve after rebase.

benjamingrenier · 2019-09-07T17:02:28Z

I done the rebase on airflow/master such as you expect no? Le sam. 7 sept. 2019 à 17:31, Jarek Potiuk <notifications@github.com> a écrit :

…

***@***.**** requested changes on this pull request. Will approve after rebase. — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#5565?email_source=notifications&email_token=AMS27S4J2NDMKZPQOAMSUPDQIPCOLA5CNFSM4IBV3MLKYY3PNVWWK3TUL52HS4DFWFIHK3DMKJSXC5LFON2FEZLWNFSXPKTDN5WW2ZLOORPWSZGOCD7XFOI#pullrequestreview-285176505>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AMS27SYSEDSP3XN5TVDQKOLQIPCOLANCNFSM4IBV3MLA> .

potiuk · 2019-09-21T19:02:42Z

Same here @benjamingrenier :(. Rebase is needed.

…e#5565)

benjamingrenier · 2019-09-22T11:41:48Z

Will approve after rebase.

Rebase done, but I don't understand why there are errors ?

stale · 2019-11-20T16:52:45Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

OmerJog · 2019-11-27T18:54:10Z

@potiuk wasn't this mergable? the failed test was just a timeout.
What was missing here?

OmerJog · 2019-12-10T07:32:44Z

@benjamingrenier did you stop working on this PR ? seems like it was marked stale by mistake

benjamingrenier · 2019-12-10T19:16:34Z

Yes it was a mistake I will rebases it soon. Le mar. 10 déc. 2019 à 08:32, OmerJog <notifications@github.com> a écrit :

…

@benjamingrenier <https://github.com/benjamingrenier> did you stop working on this PR ? seems like it was marked stale by mistake — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5565?email_source=notifications&email_token=AMS27SZWNXIOXDNX5GW3Z6DQX5AZ5A5CNFSM4IBV3MLKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGOIBVA#issuecomment-563904724>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AMS27S75U32WX4CP6A4VDD3QX5AZ5ANCNFSM4IBV3MLA> .

benjamingrenier · 2019-12-10T19:27:53Z

Done : #6780

eladkal requested changes Jul 11, 2019

View reviewed changes

airflow/contrib/hooks/bigquery_hook.py Outdated Show resolved Hide resolved

eladkal requested changes Jul 11, 2019

View reviewed changes

airflow/contrib/hooks/bigquery_hook.py Outdated Show resolved Hide resolved

airflow/contrib/hooks/bigquery_hook.py Outdated Show resolved Hide resolved

mik-laj added the provider:google Google (including GCP) related issues label Jul 22, 2019

mik-laj reviewed Jul 27, 2019

View reviewed changes

airflow/contrib/hooks/bigquery_hook.py Outdated Show resolved Hide resolved

mik-laj reviewed Aug 13, 2019

View reviewed changes

benjamingrenier force-pushed the AIRFLOW-4899 branch 3 times, most recently from ea7560e to 06dae40 Compare August 20, 2019 12:30

mik-laj reviewed Aug 22, 2019

View reviewed changes

airflow/contrib/hooks/bigquery_hook.py Outdated Show resolved Hide resolved

benjamingrenier force-pushed the AIRFLOW-4899 branch from 06dae40 to e634388 Compare September 2, 2019 17:14

benjamingrenier force-pushed the AIRFLOW-4899 branch from e634388 to e670a3e Compare September 2, 2019 18:09

benjamingrenier force-pushed the AIRFLOW-4899 branch from e670a3e to 14d8e21 Compare September 5, 2019 16:24

benjamingrenier closed this Sep 6, 2019

benjamingrenier force-pushed the AIRFLOW-4899 branch from 4efff72 to 91d0857 Compare September 6, 2019 20:52

benjamingrenier reopened this Sep 6, 2019

benjamingrenier force-pushed the AIRFLOW-4899 branch from 4192043 to a8471aa Compare September 6, 2019 21:22

potiuk requested changes Sep 7, 2019

View reviewed changes

[AIRFLOW-4899] Fix BQ Hook - get_datasets_list uses pagination (apach…

2146e77

…e#5565)

benjamingrenier force-pushed the AIRFLOW-4899 branch from a8471aa to 2146e77 Compare September 22, 2019 09:27

benjamingrenier requested a review from potiuk September 22, 2019 09:28

kaxil force-pushed the master branch from 558f9f1 to d719e1f Compare October 2, 2019 15:30

stale bot added the stale Stale PRs per the .github/workflows/stale.yml policy file label Nov 20, 2019

stale bot closed this Nov 27, 2019

Mofef mentioned this pull request Dec 8, 2020

[AIRFLOW-4899] Fix BQ Hook - get_datasets_list uses pagination (#6780) #6780

Closed

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[AIRFLOW-4899] Fix get_dataset_list from bigquery hook to return next… #5565

[AIRFLOW-4899] Fix get_dataset_list from bigquery hook to return next… #5565

benjamingrenier commented Jul 11, 2019

eladkal commented Jul 11, 2019

benjamingrenier commented Jul 11, 2019 •

edited

mik-laj commented Aug 6, 2019

mik-laj commented Aug 12, 2019

benjamingrenier commented Aug 12, 2019

benjamingrenier commented Aug 12, 2019

benjamingrenier commented Aug 12, 2019

mik-laj Aug 13, 2019 •

edited

mik-laj commented Aug 14, 2019

mik-laj commented Aug 28, 2019

benjamingrenier commented Sep 2, 2019

mik-laj commented Sep 5, 2019

benjamingrenier commented Sep 5, 2019

benjamingrenier commented Sep 5, 2019

mik-laj commented Sep 5, 2019

benjamingrenier commented Sep 6, 2019

potiuk commented Sep 6, 2019

potiuk commented Sep 6, 2019

potiuk left a comment

benjamingrenier commented Sep 7, 2019 via email

potiuk commented Sep 21, 2019

benjamingrenier commented Sep 22, 2019

stale bot commented Nov 20, 2019

OmerJog commented Nov 27, 2019

OmerJog commented Dec 10, 2019

benjamingrenier commented Dec 10, 2019 via email

benjamingrenier commented Dec 10, 2019

[AIRFLOW-4899] Fix get_dataset_list from bigquery hook to return next… #5565

[AIRFLOW-4899] Fix get_dataset_list from bigquery hook to return next… #5565

Conversation

benjamingrenier commented Jul 11, 2019

eladkal commented Jul 11, 2019

benjamingrenier commented Jul 11, 2019 • edited

mik-laj commented Aug 6, 2019

mik-laj commented Aug 12, 2019

benjamingrenier commented Aug 12, 2019

benjamingrenier commented Aug 12, 2019

benjamingrenier commented Aug 12, 2019

mik-laj Aug 13, 2019 • edited

Choose a reason for hiding this comment

mik-laj commented Aug 14, 2019

mik-laj commented Aug 28, 2019

benjamingrenier commented Sep 2, 2019

mik-laj commented Sep 5, 2019

benjamingrenier commented Sep 5, 2019

benjamingrenier commented Sep 5, 2019

mik-laj commented Sep 5, 2019

benjamingrenier commented Sep 6, 2019

potiuk commented Sep 6, 2019

potiuk commented Sep 6, 2019

potiuk left a comment

Choose a reason for hiding this comment

benjamingrenier commented Sep 7, 2019 via email

potiuk commented Sep 21, 2019

benjamingrenier commented Sep 22, 2019

stale bot commented Nov 20, 2019

OmerJog commented Nov 27, 2019

OmerJog commented Dec 10, 2019

benjamingrenier commented Dec 10, 2019 via email

benjamingrenier commented Dec 10, 2019

benjamingrenier commented Jul 11, 2019 •

edited

mik-laj Aug 13, 2019 •

edited