BigQuery: Fix Add maximum_bytes_billed option to magics #8179

HemangChothani · 2019-05-27T13:40:27Z

sduskis · 2019-05-28T14:49:38Z

bigquery/google/cloud/bigquery/magics.py

+        Raises:
+            ValueError: If the parameters are invalid.
+        """
+        return self._maximum_bytes_billed


Shouldn't this be return job_config.maximum_bytes_billed?

I think this part is OK. A potential alternative implementation is to have a self._default_job_config object, but I think that can wait until we have more than one default property to set.

I think we should just have a default job config that users can set properties on. That way, users can set properties on the default config that we don't have magics arguments for. No need to make properties for each of the job config properties. That approach doesn't scale well, because it would be too much repetitive code to add each job config property separately.

@tswast I have created a property for the class Context because i found Set job_config.maximum_bytes_billed = google.cloud.bigquery.magics.context.maximum_bytes_billed line in the description of feature request,So can i remove that property from Context class which is in magic.py file, What and where to set a defalut value or a value of google.cloud.bigquery.magics.context.maximum_bytes_billed

Sorry for the "flip flopping" @HemangChothani. Alix convinced me offline that a default_query_job_config property is the better implementation.

I've added commit 30f8326 that implements this.

sduskis · 2019-05-28T14:50:04Z

bigquery/google/cloud/bigquery/magics.py

+    def maximum_bytes_billed(self, value):
+        try:
+            value = int(value)
+            self._maximum_bytes_billed = value


Why do we need self._maximum_bytes_billed? Can't we just use job_config.maximum_bytes_billed?

Why is job_config even in this function?

One way of implementing this feature would be to create a self._default_job_config to hold the default job config (used when constructing the BigQuery client), but that doesn't appear to be what's going on here.

sduskis · 2019-05-28T14:51:36Z

bigquery/tests/unit/test_magics.py

+    query_job_mock.to_dataframe.return_value = result
+    with run_query_patch as run_query_mock, default_patch:
+        run_query_mock.return_value = query_job_mock
+        return_value = ip.run_cell_magic("bigquery", "--maximum_bytes_billed=123456789", sql)


Can you please check to make sure that the maximum_bytes_billed is used?

sduskis · 2019-05-28T14:52:16Z

bigquery/tests/unit/test_magics.py

+        return_value = ip.run_cell_magic("bigquery", "--maximum_bytes_billed=None", sql)
+
+        bqstorage_mock.assert_not_called()
+        query_job_mock.to_dataframe.assert_called_once_with(bqstorage_client=None)


Can you please check to make sure that the maximum_bytes_billed is correct?

tswast

+1 To Solomon's requests for test changes that verify the value is actually set.

tswast · 2019-05-28T17:13:47Z

bigquery/google/cloud/bigquery/magics.py

+    def maximum_bytes_billed(self, value):
+        try:
+            value = int(value)
+            self._maximum_bytes_billed = value


Why is job_config even in this function?

One way of implementing this feature would be to create a self._default_job_config to hold the default job config (used when constructing the BigQuery client), but that doesn't appear to be what's going on here.

tswast · 2019-05-28T17:14:10Z

bigquery/google/cloud/bigquery/magics.py

+            job_config = bigquery.job.QueryJobConfig()
+            job_config.maximum_bytes_billed = self._maximum_bytes_billed
+
+        except Exception:


This catch block is not needed. Just let int() throw.

tswast · 2019-05-28T17:15:18Z

bigquery/google/cloud/bigquery/magics.py

+        try:
+            value = int(args.maximum_bytes_billed)
+            job_config.maximum_bytes_billed = value
+        except Exception:


Ditto. Don't catch Exception. We do need to think about having better error messages in our magics, but I don't think this is the way to do it.

tswast · 2019-05-28T17:18:10Z

bigquery/google/cloud/bigquery/magics.py

+        Raises:
+            ValueError: If the parameters are invalid.
+        """
+        return self._maximum_bytes_billed


I think this part is OK. A potential alternative implementation is to have a self._default_job_config object, but I think that can wait until we have more than one default property to set.

Test API params get set.

bigquery/tests/unit/test_magics.py

tswast · 2019-06-03T18:22:55Z

Coverage failure is a weird one. It's not happening locally.

tests/unit/test_magics.py                     410      0      4      1    99%   246->exit

That line is:

assert all(re.match("Query executing: .*s", line) for line in execution_updates)

Maybe it's getting confused by the generator expression? I can try converting that to a plain old for loop.

Maybe that'll fix the coverage issues?

Addressed requested changes in latest commits.

tswast · 2019-06-03T19:55:04Z

Thank you @HemangChothani for the contribution. This feature will appear in the next release of google-cloud-bigquery.

HemangChothani requested a review from a team May 27, 2019 13:40

googlebot added the cla: yes This human has signed the Contributor License Agreement. label May 27, 2019

sduskis previously requested changes May 28, 2019

View reviewed changes

sduskis added the api: bigquery Issues related to the BigQuery API. label May 28, 2019

tswast self-requested a review May 28, 2019 17:10

tswast requested changes May 28, 2019

View reviewed changes

tseaver added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label May 28, 2019

yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label May 28, 2019

HemangChothani and others added 3 commits May 31, 2019 10:32

Add_maximum_bytes_billed_to_magics

7b843d0

Add_maximum_bytes_billed_to_magics and test cases

629d34b

Add default_query_job_config property.

30f8326

Test API params get set.

tswast force-pushed the feature/Add_maximum_bytes_billed_to_magics branch from e140b64 to 30f8326 Compare May 31, 2019 23:56

Add test for default_query_job_config setter

a4de333

tswast added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jun 3, 2019

tswast requested review from sduskis, alixhami and tswast and removed request for alixhami June 3, 2019 16:29

yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jun 3, 2019

alixhami requested changes Jun 3, 2019

View reviewed changes

bigquery/tests/unit/test_magics.py Outdated Show resolved Hide resolved

bigquery/tests/unit/test_magics.py Show resolved Hide resolved

tswast added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jun 3, 2019

yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jun 3, 2019

Remove redundant maximum_bytes_billed test.

266de84

tswast added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jun 3, 2019

yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jun 3, 2019

tswast mentioned this pull request Jun 3, 2019

BigQuery: Add dryrun option to magic. #8143

Closed

tswast requested a review from alixhami June 3, 2019 17:44

Use for loop instead of all and generator expression.

7ebfcc5

Maybe that'll fix the coverage issues?

tswast added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jun 3, 2019

yoshi-kokoro removed kokoro:force-run Add this label to force Kokoro to re-run the tests. labels Jun 3, 2019

alixhami approved these changes Jun 3, 2019

View reviewed changes

tswast approved these changes Jun 3, 2019

View reviewed changes

tswast merged commit 0975724 into googleapis:master Jun 3, 2019

This was referenced Jun 3, 2019

Bigquery: Add page_size parameter to QueryJob.result. #8206

Merged

BigQuery: coverage failures on master branch. #8212

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BigQuery: Fix Add maximum_bytes_billed option to magics #8179

BigQuery: Fix Add maximum_bytes_billed option to magics #8179

HemangChothani commented May 27, 2019 •

edited

Loading

sduskis May 28, 2019

tswast May 28, 2019

alixhami May 28, 2019

HemangChothani May 29, 2019 •

edited

Loading

tswast May 31, 2019

sduskis May 28, 2019

tswast May 28, 2019 •

edited

Loading

sduskis May 28, 2019

sduskis May 28, 2019

tswast left a comment

tswast May 28, 2019 •

edited

Loading

tswast May 28, 2019

tswast May 28, 2019

tswast May 28, 2019

tswast commented Jun 3, 2019

tswast commented Jun 3, 2019

BigQuery: Fix Add maximum_bytes_billed option to magics #8179

BigQuery: Fix Add maximum_bytes_billed option to magics #8179

Conversation

HemangChothani commented May 27, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

HemangChothani May 29, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tswast May 28, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tswast left a comment

Choose a reason for hiding this comment

tswast May 28, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tswast commented Jun 3, 2019

tswast commented Jun 3, 2019

HemangChothani commented May 27, 2019 •

edited

Loading

HemangChothani May 29, 2019 •

edited

Loading

tswast May 28, 2019 •

edited

Loading

tswast May 28, 2019 •

edited

Loading