Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: create manual client for Write API #112

Merged
merged 36 commits into from Apr 3, 2020

Conversation

yirutang
Copy link
Contributor

@yirutang yirutang commented Mar 19, 2020

Fixes #111 ☕️

A StreamWriter that provides built-in capabilities to help making bi-di streaming client easy to use:

  • handle batching of messages;
  • controlling memory utilization (through flow control);
  • automatic connection re-establishment;
  • request cleanup (only keeps write schema on first request in the stream).

Example:
WriteStream writeStream =
client.createWriteStream(
CreateWriteStreamRequest.newBuilder()
.setParent(tableId2)
.setWriteStream(WriteStream.newBuilder().setType(WriteStream.Type.PENDING).build())
.build());
StreamWriter streamWriter = StreamWriter.newBuilder(writeStream.getName()).build();
ApiFuture response =
streamWriter.append(
createAppendRequestComplicateType(writeStream.getName(), new String[] {"aaa"})
.setOffset(Int64Value.of(0L))
.build());
assertEquals(0, response.get().getOffset());
streamWriter.shutdown();
}

yirutang added 12 commits Feb 25, 2020
First version, test to be developed.
…brary helps to convert a ProtoDescriptorProto out of ProtoDescriptor into a self contained ProtoDescriptorProto, that can be passed into API and reconstructed on the server side.

	modified:   google-cloud-bigquerystorage/pom.xml
	new file:   google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/ProtoSchemaConverter.java
	new file:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/ProtoSchemaConverterTest.java
	new file:   google-cloud-bigquerystorage/src/test/proto/test.proto
	modified:   pom.xml
First version, test to be developed.
.
	modified:   google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriter.java
	modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/FakeBigQueryWriteImpl.java
	modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/FakeScheduledExecutorService.java
	modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriterTest.java
	modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/it/ITBigQueryWriteManualClientTest.java
@googlebot googlebot added the cla: yes label Mar 19, 2020
yirutang added 2 commits Mar 19, 2020
	modified:   google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriter.java
@yirutang yirutang changed the title Create manual client for Write API feat: create manual client for Write API Mar 19, 2020
@stephaniewang526 stephaniewang526 added the kokoro:force-run label Mar 20, 2020
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run label Mar 20, 2020
@codecov
Copy link

@codecov codecov bot commented Mar 20, 2020

Codecov Report

Merging #112 into master will increase coverage by 1.52%.
The diff coverage is 81.05%.

Impacted file tree graph

@@             Coverage Diff              @@
##             master     #112      +/-   ##
============================================
+ Coverage     72.36%   73.88%   +1.52%     
- Complexity      281      319      +38     
============================================
  Files            39       42       +3     
  Lines          1809     2229     +420     
  Branches          4       70      +66     
============================================
+ Hits           1309     1647     +338     
- Misses          496      552      +56     
- Partials          4       30      +26
Impacted Files Coverage Δ Complexity Δ
...google/cloud/bigquery/storage/v1alpha2/Waiter.java 72.54% <72.54%> (ø) 14 <14> (?)
.../cloud/bigquery/storage/v1alpha2/StreamWriter.java 81.17% <81.17%> (ø) 28 <28> (?)
...igquery/storage/v1alpha2/ProtoSchemaConverter.java 93.54% <93.54%> (ø) 6 <6> (?)
...igquery/storage/v1beta1/BigQueryStorageClient.java 68.75% <0%> (-1.84%) 16% <0%> (-3%)
...ery/storage/v1beta1/BaseBigQueryStorageClient.java 68.75% <0%> (-1.84%) 16% <0%> (-3%)
...d/bigquery/storage/v1beta2/BigQueryReadClient.java 51.51% <0%> (-1.43%) 8% <0%> (-1%)
.../cloud/bigquery/storage/v1/BigQueryReadClient.java 51.51% <0%> (-1.43%) 8% <0%> (-1%)
...ud/bigquery/storage/v1/BaseBigQueryReadClient.java 54.54% <0%> (-1.34%) 9% <0%> (-1%)
...gquery/storage/v1beta2/BaseBigQueryReadClient.java 54.54% <0%> (-1.34%) 9% <0%> (-1%)
...ery/storage/v1beta2/stub/GrpcBigQueryReadStub.java 89.74% <0%> (-0.62%) 9% <0%> (-1%)
... and 15 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c4814dc...f942f48. Read the comment docs.

@stephaniewang526
Copy link
Member

@stephaniewang526 stephaniewang526 commented Mar 20, 2020

please take a look at the various build failures.

@stephaniewang526 stephaniewang526 self-requested a review Mar 20, 2020
yirutang added 5 commits Mar 20, 2020
.
	modified:   google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriter.java
	modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/it/ITBigQueryWriteManualClientTest.java
…work

	modified:   google-cloud-bigquerystorage/pom.xml
	modified:   google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriter.java
	modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriterTest.java
	modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/it/ITBigQueryWriteManualClientTest.java
	modified:   google-cloud-bigquerystorage/src/test/proto/test.proto

	modified:   google-cloud-bigquerystorage/pom.xml
	modified:   google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriter.java
	modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriterTest.java
	modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/it/ITBigQueryWriteManualClientTest.java
	modified:   google-cloud-bigquerystorage/src/test/proto/test.proto
@stephaniewang526
Copy link
Member

@stephaniewang526 stephaniewang526 commented Mar 24, 2020

As a general note, all public methods need to be documented clearly in accompanying Java doc to help developers understand what the inputs, outputs, exceptions, etc are. I see a number of public methods without Java doc. Please make sure to put them in.

@stephaniewang526 stephaniewang526 requested a review from bigang-li Mar 26, 2020
bigang-li
Copy link
Collaborator

@bigang-li bigang-li commented on 76622fc Mar 27, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should be 0 to keep the same with the original code?

yirutang and others added 5 commits Apr 1, 2020
…gquery/storage/v1alpha2/Waiter.java

Co-Authored-By: Stephanie Wang <stephaniewang526@users.noreply.github.com>
…gquery/storage/v1alpha2/it/ITBigQueryWriteManualClientTest.java

Co-Authored-By: Stephanie Wang <stephaniewang526@users.noreply.github.com>
…gquery/storage/v1alpha2/it/ITBigQueryWriteManualClientTest.java

Co-Authored-By: Stephanie Wang <stephaniewang526@users.noreply.github.com>
Copy link
Contributor Author

@yirutang yirutang left a comment

PTAL

@stephaniewang526 stephaniewang526 added the kokoro:force-run label Apr 2, 2020
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run label Apr 2, 2020
@stephaniewang526 stephaniewang526 added the kokoro:force-run label Apr 2, 2020
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run label Apr 2, 2020
@stephaniewang526 stephaniewang526 merged commit 98851e9 into googleapis:master Apr 3, 2020
14 of 15 checks passed
yoshi-automation added a commit that referenced this issue Apr 3, 2020
98851e9
commit 98851e9
Author: Yiru Tang <yiru@google.com>
Date:   Thu Apr 2 17:45:37 2020 -0700

    feat: create manual client for Write API (#112)

    * First writeapi manual client

    First version, test to be developed.

    * Manual client with e2e

    * StreamWriter library

    * ProtoSchema convert library WriteApi client library addition. This library helps to convert a ProtoDescriptorProto out of ProtoDescriptor into a self contained ProtoDescriptorProto, that can be passed into API and reconstructed on the server side.

            modified:   google-cloud-bigquerystorage/pom.xml
            new file:   google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/ProtoSchemaConverter.java
            new file:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/ProtoSchemaConverterTest.java
            new file:   google-cloud-bigquerystorage/src/test/proto/test.proto
            modified:   pom.xml

    * First writeapi manual client

    First version, test to be developed.

    * .

    * incremental development

    * .
            modified:   google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriter.java
            modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/FakeBigQueryWriteImpl.java
            modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/FakeScheduledExecutorService.java
            modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriterTest.java
            modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/it/ITBigQueryWriteManualClientTest.java

    * .

    * .

    * .

    * feat: create manual java client for writeapi

            modified:   google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriter.java

    * feat: add integration test final verification.

    * .

    * .

            modified:   google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriter.java
            modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/it/ITBigQueryWriteManualClientTest.java

    * .

    * .

    * .

    * .

    * avoid duplicate count++ and make it easier to read

    * Update google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/Waiter.java

    Co-Authored-By: Stephanie Wang <stephaniewang526@users.noreply.github.com>

    * Update google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/it/ITBigQueryWriteManualClientTest.java

    Co-Authored-By: Stephanie Wang <stephaniewang526@users.noreply.github.com>

    * Update google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/it/ITBigQueryWriteManualClientTest.java

    Co-Authored-By: Stephanie Wang <stephaniewang526@users.noreply.github.com>

    * .

    * .

    * .

    * .

    * .

    * dep: add undeclared dep in parent pom

    * dep: remove duplicate and add undeclared

    remove dupe gax; add undeclared commons-lang3

    Co-authored-by: Stephanie Wang <stephaniewang526@users.noreply.github.com>
stephaniewang526 pushed a commit that referenced this issue Apr 3, 2020
…nerator. (#134)

* feat: create manual client for Write API (#112)

98851e9
commit 98851e9
Author: Yiru Tang <yiru@google.com>
Date:   Thu Apr 2 17:45:37 2020 -0700

    feat: create manual client for Write API (#112)

    * First writeapi manual client

    First version, test to be developed.

    * Manual client with e2e

    * StreamWriter library

    * ProtoSchema convert library WriteApi client library addition. This library helps to convert a ProtoDescriptorProto out of ProtoDescriptor into a self contained ProtoDescriptorProto, that can be passed into API and reconstructed on the server side.

            modified:   google-cloud-bigquerystorage/pom.xml
            new file:   google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/ProtoSchemaConverter.java
            new file:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/ProtoSchemaConverterTest.java
            new file:   google-cloud-bigquerystorage/src/test/proto/test.proto
            modified:   pom.xml

    * First writeapi manual client

    First version, test to be developed.

    * .

    * incremental development

    * .
            modified:   google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriter.java
            modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/FakeBigQueryWriteImpl.java
            modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/FakeScheduledExecutorService.java
            modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriterTest.java
            modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/it/ITBigQueryWriteManualClientTest.java

    * .

    * .

    * .

    * feat: create manual java client for writeapi

            modified:   google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriter.java

    * feat: add integration test final verification.

    * .

    * .

            modified:   google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriter.java
            modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/it/ITBigQueryWriteManualClientTest.java

    * .

    * .

    * .

    * .

    * avoid duplicate count++ and make it easier to read

    * Update google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/Waiter.java

    Co-Authored-By: Stephanie Wang <stephaniewang526@users.noreply.github.com>

    * Update google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/it/ITBigQueryWriteManualClientTest.java

    Co-Authored-By: Stephanie Wang <stephaniewang526@users.noreply.github.com>

    * Update google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/it/ITBigQueryWriteManualClientTest.java

    Co-Authored-By: Stephanie Wang <stephaniewang526@users.noreply.github.com>

    * .

    * .

    * .

    * .

    * .

    * dep: add undeclared dep in parent pom

    * dep: remove duplicate and add undeclared

    remove dupe gax; add undeclared commons-lang3

    Co-authored-by: Stephanie Wang <stephaniewang526@users.noreply.github.com>

* deps: update dependency io.grpc:grpc-bom to v1.28.1 (#132)

711c5c7
commit 711c5c7
Author: WhiteSource Renovate <bot@renovateapp.com>
Date:   Fri Apr 3 03:08:41 2020 +0200

    deps: update dependency io.grpc:grpc-bom to v1.28.1 (#132)

* chore: release 0.127.0 (#122)

45fc919
commit 45fc919
Author: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
Date:   Thu Apr 2 21:49:46 2020 -0400

    chore: release 0.127.0 (#122)

    * updated CHANGELOG.md [ci skip]

    * updated README.md [ci skip]

    * updated versions.txt [ci skip]

    * updated google-cloud-bigquerystorage-bom/pom.xml [ci skip]

    * updated grpc-google-cloud-bigquerystorage-v1/pom.xml [ci skip]

    * updated grpc-google-cloud-bigquerystorage-v1alpha2/pom.xml [ci skip]

    * updated grpc-google-cloud-bigquerystorage-v1beta1/pom.xml [ci skip]

    * updated grpc-google-cloud-bigquerystorage-v1beta2/pom.xml [ci skip]

    * updated proto-google-cloud-bigquerystorage-v1/pom.xml [ci skip]

    * updated proto-google-cloud-bigquerystorage-v1alpha2/pom.xml [ci skip]

    * updated proto-google-cloud-bigquerystorage-v1beta1/pom.xml [ci skip]

    * updated proto-google-cloud-bigquerystorage-v1beta2/pom.xml [ci skip]

    * updated samples/install-without-bom/pom.xml [ci skip]

    * updated samples/pom.xml [ci skip]

    * updated samples/snapshot/pom.xml [ci skip]

    * updated samples/snippets/pom.xml [ci skip]

    * updated google-cloud-bigquerystorage/pom.xml [ci skip]

    * updated pom.xml [ci skip]

    Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
@yirutang yirutang deleted the work branch Jun 5, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cla: yes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants