feat: Add missing storage related fields to Table, TableInfo and StandardTableDefinition #2673

sumeetgajjar · 2023-04-29T00:51:13Z

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
Ensure the tests and linter pass
Code coverage does not decrease (if any source code was changed)
Appropriate docs were updated (if necessary)

Fixes #2672 ☕️

sumeetgajjar · 2023-04-29T00:55:13Z

@Neenu1995 @farhan0102 can you please review this PR?

Thanks in advance

sumeetgajjar · 2023-04-29T00:57:57Z

google-cloud-bigquery/src/test/java/com/google/cloud/bigquery/it/ITBigQueryTest.java

@@ -1451,6 +1451,9 @@ public void testCreateAndGetTable() {
    assertNotNull(remoteTable.getLastModifiedTime());
    assertNotNull(remoteTable.<StandardTableDefinition>getDefinition().getNumBytes());
    assertNotNull(remoteTable.<StandardTableDefinition>getDefinition().getNumLongTermBytes());
+    assertNotNull(remoteTable.<StandardTableDefinition>getDefinition().getNumTotalLogicalBytes());
+    assertNotNull(remoteTable.<StandardTableDefinition>getDefinition().getNumActiveLogicalBytes());
+    assertNotNull(remoteTable.<StandardTableDefinition>getDefinition().getNumLongTermLogicalBytes());


Note: We cannot add the rest of the newly added fields like TimeTravelBytes etc since they are not available for a table created few seconds ago.

sumeetgajjar · 2023-05-03T02:33:51Z

The ITBigQueryTest.testCreateAndGetTable integration test is failing

[ERROR] com.google.cloud.bigquery.it.ITBigQueryTest.testCreateAndGetTable  Time elapsed: 0.524 s  <<< FAILURE!
java.lang.AssertionError
	at org.junit.Assert.fail(Assert.java:87)
	at org.junit.Assert.assertTrue(Assert.java:42)
	at org.junit.Assert.assertNotNull(Assert.java:713)
	at org.junit.Assert.assertNotNull(Assert.java:723)
	at com.google.cloud.bigquery.it.ITBigQueryTest.testCreateAndGetTable(ITBigQueryTest.java:1454)

This is due to the fact there's a bug in the generated code of google-api-services-bigquery

https://github.com/googleapis/google-api-java-client-services/blob/09b3ea51aac423a1529b459d82abd5db736d4ed9/clients/google-api-services-bigquery/v2/2.0.0/com/google/api/services/bigquery/model/Table.java#L260-L261

The JSON mapper is expecting the key to be num_total_logical_bytes (snake cased).
However, the actual JSON object has the key as numTotalLogicalBytes (camel cased).

I ran the integration test while attached to a debugger and inspected the com.google.api.services.bigquery.model.Table object which is feed into TableInfo.BuilderImpl constructor.

tablePb.getNumTotalLogicalBytes() // shall return null
tablePb.get("numTotalLogicalBytes") // shall return a valid long value

I have filed an issue googleapis/google-api-java-client-services#16915

@Neenu1995 @prash-mi I took a look at the google-api-services-bigquery but it seems there input JSON required for generating the bigquery client is not open-sourced or at least not available in that repository, please correct me if I am wrong and please let me know how to proceed next?

…ndardTableDefinition

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

sumeetgajjar · 2023-06-21T22:10:14Z

Hi @Neenu1995 - based on this comment googleapis/google-api-java-client-services#16915 (comment) from shollyman, I checked the discovery document and the revision number is 20230520 which is beyond 20230504.

I ran a test of my own to verify the newly added fields are populated.

Can you please re-run the necessary tests and remove the do not merge label from this PR?

sumeetgajjar · 2023-06-21T22:46:18Z

Hi @Neenu1995 thanks for adding run labels.

It seems the ci / clirr job failed since we added a bunch of new methods.
Can you please share steps/actions required from my-side to mitigate this error?

Error:  Unable to locate enclosing class com.google.cloud.bigquery. for nested class com.google.cloud.bigquery.$AutoValue_Labels
Error:  7013: com.google.cloud.bigquery.StandardTableDefinition: Abstract method 'public java.lang.Long getNumActiveLogicalBytes()' has been added
Error:  7013: com.google.cloud.bigquery.StandardTableDefinition: Abstract method 'public java.lang.Long getNumActivePhysicalBytes()' has been added
Error:  7013: com.google.cloud.bigquery.StandardTableDefinition: Abstract method 'public java.lang.Long getNumLongTermLogicalBytes()' has been added
Error:  7013: com.google.cloud.bigquery.StandardTableDefinition: Abstract method 'public java.lang.Long getNumLongTermPhysicalBytes()' has been added
Error:  7013: com.google.cloud.bigquery.StandardTableDefinition: Abstract method 'public java.lang.Long getNumTimeTravelPhysicalBytes()' has been added
Error:  7013: com.google.cloud.bigquery.StandardTableDefinition: Abstract method 'public java.lang.Long getNumTotalLogicalBytes()' has been added
Error:  7013: com.google.cloud.bigquery.StandardTableDefinition: Abstract method 'public java.lang.Long getNumTotalPhysicalBytes()' has been added
Error:  7013: com.google.cloud.bigquery.StandardTableDefinition$Builder: Abstract method 'public com.google.cloud.bigquery.StandardTableDefinition$Builder setNumActiveLogicalBytes(java.lang.Long)' has been added
Error:  7013: com.google.cloud.bigquery.StandardTableDefinition$Builder: Abstract method 'public com.google.cloud.bigquery.StandardTableDefinition$Builder setNumActivePhysicalBytes(java.lang.Long)' has been added
Error:  7013: com.google.cloud.bigquery.StandardTableDefinition$Builder: Abstract method 'public com.google.cloud.bigquery.StandardTableDefinition$Builder setNumLongTermLogicalBytes(java.lang.Long)' has been added
Error:  7013: com.google.cloud.bigquery.StandardTableDefinition$Builder: Abstract method 'public com.google.cloud.bigquery.StandardTableDefinition$Builder setNumLongTermPhysicalBytes(java.lang.Long)' has been added
Error:  7013: com.google.cloud.bigquery.StandardTableDefinition$Builder: Abstract method 'public com.google.cloud.bigquery.StandardTableDefinition$Builder setNumTimeTravelPhysicalBytes(java.lang.Long)' has been added
Error:  7013: com.google.cloud.bigquery.StandardTableDefinition$Builder: Abstract method 'public com.google.cloud.bigquery.StandardTableDefinition$Builder setNumTotalLogicalBytes(java.lang.Long)' has been added
Error:  7013: com.google.cloud.bigquery.StandardTableDefinition$Builder: Abstract method 'public com.google.cloud.bigquery.StandardTableDefinition$Builder setNumTotalPhysicalBytes(java.lang.Long)' has been added
[INFO] ------------------------------------------------------------------------

Neenu1995 · 2023-06-22T19:15:39Z

To fix, the clirr check, please add exceptions to the clirr-ignored-differences file according to this doc.

sumeetgajjar · 2023-06-22T22:05:39Z

To fix, the clirr check, please add exceptions to the clirr-ignored-differences file according to this doc.

Thanks @Neenu1995 for the doc, fixed it in the latest commit.
Please re-run the checks 😄

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

Neenu1995 · 2023-06-23T15:32:20Z

google-cloud-bigquery/src/test/java/com/google/cloud/bigquery/BigQueryImplTest.java

@@ -2863,4 +2865,27 @@ public void testTestIamPermissionsWhenNoPermissionsGranted() {
    assertEquals(perms, ImmutableList.of());
    verify(bigqueryRpcMock).testIamPermissions(resourceId, checkedPermissions, EMPTY_RPC_OPTIONS);
  }
+
+  @Test
+  public void testSumeet() throws Exception {


Please clean up personal tests.

Removed the personal test(s).
Thanks for pointing it out.

Please re-run the checks 😄

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

sumeetgajjar · 2023-06-23T19:24:05Z

Thank you @Neenu1995 for approving the PR and guiding us with TODOs to get in this change.

Looking forward to future contributions.

Neenu1995 · 2023-06-23T19:31:49Z

Thanks for contributing ti the Bigquery library @sumeetgajjar

sumeetgajjar requested a review from a team as a code owner April 29, 2023 00:51

sumeetgajjar requested a review from prash-mi April 29, 2023 00:51

product-auto-label bot added size: l Pull request size is large. api: bigquery Issues related to the googleapis/java-bigquery API. labels Apr 29, 2023

sumeetgajjar force-pushed the add_storage_fields_to_table branch from c3c117b to df85341 Compare April 29, 2023 00:53

sumeetgajjar commented Apr 29, 2023

View reviewed changes

Neenu1995 added kokoro:force-run Add this label to force Kokoro to re-run the tests. owlbot:run Add this label to trigger the Owlbot post processor. labels May 1, 2023

gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label May 1, 2023

yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label May 1, 2023

sumeetgajjar changed the title ~~Add missing storage related fields to Table, TableInfo and StandardTableDefinition~~ chore: Add missing storage related fields to Table, TableInfo and StandardTableDefinition May 1, 2023

Neenu1995 added the kokoro:run Add this label to force Kokoro to re-run the tests. label May 2, 2023

yoshi-kokoro removed the kokoro:run Add this label to force Kokoro to re-run the tests. label May 2, 2023

sumeetgajjar mentioned this pull request May 3, 2023

BigQuery: Invalid data key for numTotalLogicalBytes in com.google.api.services.bigquery.model.Table googleapis/google-api-java-client-services#16915

Closed

Neenu1995 added the do not merge Indicates a pull request not ready for merge, due to either quality or timing. label May 3, 2023

sumeetgajjar and others added 2 commits June 21, 2023 15:02

chore: Add missing storage related fields to Table, TableInfo and Sta…

de3e9d4

…ndardTableDefinition

🦉 Updates from OwlBot post-processor

a39b96a

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

sumeetgajjar force-pushed the add_storage_fields_to_table branch from 965557e to a39b96a Compare June 21, 2023 22:03

gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Jun 21, 2023

yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jun 21, 2023

Fix clir check failure

85a6b8a

Neenu1995 added kokoro:force-run Add this label to force Kokoro to re-run the tests. owlbot:run Add this label to trigger the Owlbot post processor. labels Jun 23, 2023

gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Jun 23, 2023

Neenu1995 changed the title ~~chore: Add missing storage related fields to Table, TableInfo and StandardTableDefinition~~ feat: Add missing storage related fields to Table, TableInfo and StandardTableDefinition Jun 23, 2023

yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jun 23, 2023

🦉 Updates from OwlBot post-processor

e2f20dd

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

Neenu1995 reviewed Jun 23, 2023

View reviewed changes

Remove unwanted test(s)

8f965da

Neenu1995 added kokoro:force-run Add this label to force Kokoro to re-run the tests. owlbot:run Add this label to trigger the Owlbot post processor. labels Jun 23, 2023

gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Jun 23, 2023

yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jun 23, 2023

🦉 Updates from OwlBot post-processor

2408895

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

Neenu1995 added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jun 23, 2023

yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jun 23, 2023

Neenu1995 self-requested a review June 23, 2023 17:22

Neenu1995 approved these changes Jun 23, 2023

View reviewed changes

Neenu1995 added automerge Merge the pull request once unit tests and other checks pass. and removed do not merge Indicates a pull request not ready for merge, due to either quality or timing. labels Jun 23, 2023

gcf-merge-on-green bot merged commit e3003f4 into googleapis:main Jun 23, 2023
16 checks passed

gcf-merge-on-green bot removed the automerge Merge the pull request once unit tests and other checks pass. label Jun 23, 2023

release-please bot mentioned this pull request Jun 23, 2023

chore(main): release 2.30.0 #2774

Merged

This was referenced Jul 25, 2023

July 24, 2023 kitta65/bq-extension-vscode#212

Closed

July 24, 2023 kitta65/prettier-plugin-bq#219

Closed

July 24, 2023 kitta65/bq2cst#230

Closed

sumeetgajjar deleted the add_storage_fields_to_table branch February 2, 2024 04:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Add missing storage related fields to Table, TableInfo and StandardTableDefinition #2673

feat: Add missing storage related fields to Table, TableInfo and StandardTableDefinition #2673

sumeetgajjar commented Apr 29, 2023

sumeetgajjar commented Apr 29, 2023

sumeetgajjar Apr 29, 2023 •

edited

sumeetgajjar commented May 3, 2023

sumeetgajjar commented Jun 21, 2023

sumeetgajjar commented Jun 21, 2023

Neenu1995 commented Jun 22, 2023

sumeetgajjar commented Jun 22, 2023

Neenu1995 Jun 23, 2023

sumeetgajjar Jun 23, 2023

sumeetgajjar commented Jun 23, 2023

Neenu1995 commented Jun 23, 2023

feat: Add missing storage related fields to Table, TableInfo and StandardTableDefinition #2673

feat: Add missing storage related fields to Table, TableInfo and StandardTableDefinition #2673

Conversation

sumeetgajjar commented Apr 29, 2023

sumeetgajjar commented Apr 29, 2023

sumeetgajjar Apr 29, 2023 • edited

Choose a reason for hiding this comment

sumeetgajjar commented May 3, 2023

sumeetgajjar commented Jun 21, 2023

sumeetgajjar commented Jun 21, 2023

Neenu1995 commented Jun 22, 2023

sumeetgajjar commented Jun 22, 2023

Neenu1995 Jun 23, 2023

Choose a reason for hiding this comment

sumeetgajjar Jun 23, 2023

Choose a reason for hiding this comment

sumeetgajjar commented Jun 23, 2023

Neenu1995 commented Jun 23, 2023

sumeetgajjar Apr 29, 2023 •

edited