Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: v1beta2 manual client library for BigQueryWrite #714

Merged
merged 15 commits into from Dec 9, 2020

Conversation

yirutang
Copy link
Contributor

@yirutang yirutang commented Dec 8, 2020

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

  • Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
  • Ensure the tests and linter pass
  • Code coverage does not decrease (if any source code was changed)
  • Appropriate docs were updated (if necessary)

Fixes #<issue_number_goes_here> ☕️

@yirutang yirutang requested a review from as a code owner Dec 8, 2020
@yirutang yirutang requested a review from shollyman Dec 8, 2020
@product-auto-label product-auto-label bot added the api: bigquerystorage label Dec 8, 2020
@google-cla google-cla bot added the cla: yes label Dec 8, 2020
yirutang and others added 3 commits Dec 8, 2020
…gquery/storage/v1beta2/JsonStreamWriter.java

Co-authored-by: yoshi-code-bot <70984784+yoshi-code-bot@users.noreply.github.com>
…gquery/storage/v1beta2/OnSchemaUpdateRunnable.java

Co-authored-by: yoshi-code-bot <70984784+yoshi-code-bot@users.noreply.github.com>
Co-authored-by: yoshi-code-bot <70984784+yoshi-code-bot@users.noreply.github.com>
@yirutang yirutang changed the title feat: v1beta2 manual client library for BigQueryWrite. feat: v1beta2 manual client library for BigQueryWrite Dec 8, 2020
Co-authored-by: yoshi-code-bot <70984784+yoshi-code-bot@users.noreply.github.com>
@stephaniewang526 stephaniewang526 self-requested a review Dec 8, 2020
@stephaniewang526
Copy link
Member

@stephaniewang526 stephaniewang526 commented Dec 8, 2020

@yirutang there are a number of checkstyle errors: https://github.com/googleapis/java-bigquerystorage/pull/714/checks?check_run_id=1514447927. Running mvn checkstyle:check can repro locally.

@yirutang
Copy link
Contributor Author

@yirutang yirutang commented Dec 8, 2020

@codecov
Copy link

@codecov codecov bot commented Dec 8, 2020

Codecov Report

Merging #714 (a9fc8dd) into master (20aef28) will increase coverage by 1.63%.
The diff coverage is 86.08%.

Impacted file tree graph

@@             Coverage Diff              @@
##             master     #714      +/-   ##
============================================
+ Coverage     78.79%   80.42%   +1.63%     
- Complexity      718      959     +241     
============================================
  Files            65       73       +8     
  Lines          4263     5334    +1071     
  Branches        253      405     +152     
============================================
+ Hits           3359     4290     +931     
- Misses          786      877      +91     
- Partials        118      167      +49     
Impacted Files Coverage Δ Complexity Δ
.../google/cloud/bigquery/storage/v1beta2/Waiter.java 61.17% <61.17%> (ø) 16.00 <16.00> (?)
...oud/bigquery/storage/v1beta2/JsonStreamWriter.java 77.19% <77.19%> (ø) 12.00 <12.00> (?)
...e/cloud/bigquery/storage/v1beta2/StreamWriter.java 84.98% <84.98%> (ø) 37.00 <37.00> (?)
...gquery/storage/v1beta2/OnSchemaUpdateRunnable.java 85.71% <85.71%> (ø) 4.00 <4.00> (?)
.../bigquery/storage/v1beta2/SchemaCompatibility.java 91.30% <91.30%> (ø) 99.00 <99.00> (?)
...bigquery/storage/v1beta2/ProtoSchemaConverter.java 96.22% <96.22%> (ø) 10.00 <10.00> (?)
...d/bigquery/storage/v1beta2/JsonToProtoMessage.java 96.96% <96.96%> (ø) 47.00 <47.00> (?)
...torage/v1beta2/BQTableSchemaToProtoDescriptor.java 98.48% <98.48%> (ø) 8.00 <8.00> (?)
... and 9 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 20aef28...a9fc8dd. Read the comment docs.

@@ -1,7 +1,6 @@
/*
* Copyright 2020 Google LLC
*
Copy link
Member

@stephaniewang526 stephaniewang526 Dec 8, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The bot incorrectly proposed these changes. Raised this issue to track a fix.

@yirutang
Copy link
Contributor Author

@yirutang yirutang commented Dec 8, 2020

@stephaniewang526
Copy link
Member

@stephaniewang526 stephaniewang526 commented Dec 8, 2020

One more IT failure:

com.google.api.gax.rpc.FailedPreconditionException: io.grpc.StatusRuntimeException: FAILED_PRECONDITION: Precondition check failed.
	at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:59)
	at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:72)
	at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:60)
	at com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure(GrpcExceptionCallable.java:97)
	at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:68)
	at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1041)
	at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
	at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1215)
	at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983)
	at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:771)
	at io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:563)
	at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:533)
	at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:617)
	at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:803)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:782)
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
	Suppressed: com.google.api.gax.rpc.AsyncTaskException: Asynchronous task failed
		at com.google.api.gax.rpc.ApiExceptions.callAndTranslateApiException(ApiExceptions.java:57)
		at com.google.api.gax.rpc.UnaryCallable.call(UnaryCallable.java:112)
		at com.google.cloud.bigquery.storage.v1beta2.BigQueryWriteClient.flushRows(BigQueryWriteClient.java:652)
		at com.google.cloud.bigquery.storage.v1beta2.it.ITBigQueryWriteManualClientTest.testFlushRows(ITBigQueryWriteManualClientTest.java:606)
		at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
		at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
		at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
		at java.lang.reflect.Method.invoke(Method.java:498)
		at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
		at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
		at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
		at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
		at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
		at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
		at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
		at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
		at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
		at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
		at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
		at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
		at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
		at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
		at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
		at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
		at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
		at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
		at org.junit.runners.Suite.runChild(Suite.java:128)
		at org.junit.runners.Suite.runChild(Suite.java:27)
		at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
		at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
		at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
		at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
		at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
		at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
		at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
		at org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55)
		at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137)
		at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107)
		at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:83)
		at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:75)
		at org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:158)
		at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:377)
		at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:138)
		at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:465)
		at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:451)
Caused by: io.grpc.StatusRuntimeException: FAILED_PRECONDITION: Precondition check failed.
	at io.grpc.Status.asRuntimeException(Status.java:533)
	... 14 more

import static org.junit.Assert.assertTrue;
import static org.junit.Assert.fail;
Copy link
Contributor

@yoshi-code-bot yoshi-code-bot Dec 9, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
import static org.junit.Assert.assertTrue;
import static org.junit.Assert.fail;
import static org.junit.Assert.assertTrue;

@stephaniewang526 stephaniewang526 merged commit 53a9c3a into googleapis:master Dec 9, 2020
19 checks passed
@yirutang yirutang deleted the beta-client branch Dec 10, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: bigquerystorage cla: yes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants