Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Direct writer #165

Merged
merged 6 commits into from Apr 16, 2020
Merged

feat: Direct writer #165

merged 6 commits into from Apr 16, 2020

Conversation

yirutang
Copy link
Contributor

@yirutang yirutang commented Apr 13, 2020

Fixes #164 ☕️

Implementation of DirectWriter which allows users to write to BigQuery in one call. Deduplication is not supported but it should be enough for most common write scenarios.

yirutang added 2 commits Apr 13, 2020
	new file:   google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/DirectWriter.java
	modified:   google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriter.java
	new file:   google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/WriterCache.java
	new file:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/DirectWriterTest.java
	modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriterTest.java
	new file:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/WriterCacheTest.java
	modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/it/ITBigQueryWriteManualClientTest.java

    r 39ea964 feat:Direct Writer
    r de2cb8c feat:Direct Writer 2
    pick 8e67681 feat:direct writer 3
	new file:   google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/DirectWriter.java
	modified:   google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriter.java
	new file:   google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/WriterCache.java
	new file:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/DirectWriterTest.java
	modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriterTest.java
	new file:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/WriterCacheTest.java
	modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/it/ITBigQueryWriteManualClientTest.java
@googlebot googlebot added the cla: yes label Apr 13, 2020
@stephaniewang526 stephaniewang526 self-requested a review Apr 13, 2020
	modified:   google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/DirectWriter.java
	modified:   google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriter.java
	modified:   google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/WriterCache.java
	modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/DirectWriterTest.java
	modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/FakeBigQueryWrite.java
	modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/FakeBigQueryWriteImpl.java
	modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/MockBigQueryWriteImpl.java
	modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriterTest.java
	modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/WriterCacheTest.java
	modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/it/ITBigQueryWriteManualClientTest.java
	modified:   google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriter.java
@stephaniewang526 stephaniewang526 added the kokoro:force-run label Apr 13, 2020
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run label Apr 13, 2020
@yirutang yirutang force-pushed the direct-writer branch 5 times, most recently from 12654a4 to 9b58fba Compare Apr 14, 2020
@yirutang yirutang force-pushed the direct-writer branch 2 times, most recently from e91622f to 5c973e9 Compare Apr 15, 2020
	modified:   google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/DirectWriter.java
	new file:   google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/SchemaCompact.java
	modified:   google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriter.java
	modified:   google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/WriterCache.java
	modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/DirectWriterTest.java
	new file:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/SchemaCompactTest.java
	modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriterTest.java
	modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/WriterCacheTest.java
	modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/it/ITBigQueryWriteManualClientTest.java
@stephaniewang526 stephaniewang526 added the kokoro:force-run label Apr 15, 2020
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run label Apr 15, 2020
@yirutang
Copy link
Contributor Author

@yirutang yirutang commented Apr 15, 2020

@yirutang
Copy link
Contributor Author

@yirutang yirutang commented Apr 15, 2020

@stephaniewang526 stephaniewang526 added the kokoro:force-run label Apr 15, 2020
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run label Apr 15, 2020
@stephaniewang526
Copy link
Member

@stephaniewang526 stephaniewang526 commented Apr 15, 2020

Yes, I fixed the second e2e test, for the first one, it didn't fail on my machine but maybe there is a timing issue. I added some sleep to the test.

On Wed, Apr 15, 2020 at 2:46 PM Stephanie Wang @.***> wrote: @yirutang https://github.com/yirutang Could you look at the two flaky tests that are breaking the build: https://source.cloud.google.com/results/invocations/eb9b4e28-07c8-4d19-b259-f9ff65354df0/targets/github%2Fjava-bigquerystorage%2Fgoogle-cloud-bigquerystorage%2Ftarget%2Ffailsafe-reports%2Fintegration%2Fcom.google.cloud.bigquery.storage.v1alpha2.it.ITBigQueryWriteManualClientTest/tests;group=com.google.cloud.bigquery.storage.v1alpha2.it.ITBigQueryWriteManualClientTest%28sponge_log%29;test=testComplicateSchemaWithPendingStream;row=1 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#165 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHGYVEWTETDOG2FA2TDEKWLRMYTMJANCNFSM4MHF6LJQ .
-- Thanks. Yiru

https://source.cloud.google.com/results/invocations/8cba25ad-8919-4fdd-8446-d00222bfb028/targets/github%2Fjava-bigquerystorage%2Fgoogle-cloud-bigquerystorage%2Ftarget%2Ffailsafe-reports%2Fintegration%2Fcom.google.cloud.bigquery.storage.v1beta2.it.ITBigQueryStorageLongRunningTest/tests

those two test seem to be running better now but this one above is flaky.

In addition, there's a lint failure also. Please run mvn com.coveo:fmt-maven-plugin:format before each commit to ensure that code formate is compliant.

@yirutang
Copy link
Contributor Author

@yirutang yirutang commented Apr 16, 2020

@stephaniewang526 stephaniewang526 added the kokoro:force-run label Apr 16, 2020
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run label Apr 16, 2020
	modified:   google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriter.java
	modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/it/ITBigQueryWriteManualClientTest.java
@yirutang
Copy link
Contributor Author

@yirutang yirutang commented Apr 16, 2020

@stephaniewang526 stephaniewang526 added the kokoro:force-run label Apr 16, 2020
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run label Apr 16, 2020
@codecov
Copy link

@codecov codecov bot commented Apr 16, 2020

Codecov Report

Merging #165 into master will increase coverage by 0.40%.
The diff coverage is 83.44%.

Impacted file tree graph

@@             Coverage Diff              @@
##             master     #165      +/-   ##
============================================
+ Coverage     74.18%   74.58%   +0.40%     
- Complexity      320      344      +24     
============================================
  Files            42       45       +3     
  Lines          2227     2361     +134     
  Branches         71       82      +11     
============================================
+ Hits           1652     1761     +109     
- Misses          545      562      +17     
- Partials         30       38       +8     
Impacted Files Coverage Δ Complexity Δ
...cloud/bigquery/storage/v1alpha2/SchemaCompact.java 76.19% <76.19%> (ø) 7.00 <7.00> (?)
.../cloud/bigquery/storage/v1alpha2/DirectWriter.java 77.77% <77.77%> (ø) 4.00 <4.00> (?)
.../cloud/bigquery/storage/v1alpha2/StreamWriter.java 82.16% <82.85%> (-0.70%) 32.00 <6.00> (+3.00) ⬇️
...e/cloud/bigquery/storage/v1alpha2/WriterCache.java 88.70% <88.70%> (ø) 10.00 <10.00> (?)
...gquery/storage/v1alpha2/BigQueryWriteSettings.java 27.02% <0.00%> (-2.71%) 5.00% <0.00%> (-1.00%)
...bigquery/storage/v1alpha2/BigQueryWriteClient.java 64.51% <0.00%> (+3.22%) 15.00% <0.00%> (+1.00%)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 74564ff...bf3784b. Read the comment docs.

@stephaniewang526 stephaniewang526 merged commit ed718c1 into googleapis:master Apr 16, 2020
13 checks passed
yoshi-automation added a commit that referenced this issue Apr 16, 2020
* feat:Direct Writer

	new file:   google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/DirectWriter.java
	modified:   google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriter.java
	new file:   google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/WriterCache.java
	new file:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/DirectWriterTest.java
	modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriterTest.java
	new file:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/WriterCacheTest.java
	modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/it/ITBigQueryWriteManualClientTest.java

    r 39ea964 feat:Direct Writer
    r de2cb8c feat:Direct Writer 2
    pick 8e67681 feat:direct writer 3
	new file:   google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/DirectWriter.java
	modified:   google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriter.java
	new file:   google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/WriterCache.java
	new file:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/DirectWriterTest.java
	modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriterTest.java
	new file:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/WriterCacheTest.java
	modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/it/ITBigQueryWriteManualClientTest.java

* feat:Direct Writer 2

* feat:direct writer 3

	modified:   google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/DirectWriter.java
	modified:   google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriter.java
	modified:   google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/WriterCache.java
	modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/DirectWriterTest.java
	modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/FakeBigQueryWrite.java
	modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/FakeBigQueryWriteImpl.java
	modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/MockBigQueryWriteImpl.java
	modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriterTest.java
	modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/WriterCacheTest.java
	modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/it/ITBigQueryWriteManualClientTest.java
	modified:   google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriter.java

* Fix a logging

* Add very basic schema compact check

	modified:   google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/DirectWriter.java
	new file:   google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/SchemaCompact.java
	modified:   google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriter.java
	modified:   google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/WriterCache.java
	modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/DirectWriterTest.java
	new file:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/SchemaCompactTest.java
	modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriterTest.java
	modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/WriterCacheTest.java
	modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/it/ITBigQueryWriteManualClientTest.java

* fix e2e

	modified:   google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriter.java
	modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/it/ITBigQueryWriteManualClientTest.java

Source-Author: Yiru Tang <yiru@google.com>
Source-Date: Thu Apr 16 12:49:37 2020 -0700
Source-Repo: googleapis/java-bigquerystorage
Source-Sha: ed718c1
Source-Link: ed718c1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cla: yes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants