[BEAM-12422][no need to review] Testing BEAM-12422 by installing the vendored gRPC to local Maven repository #15103

suztomo · 2021-06-29T23:55:46Z

[BEAM-12422] Removing unnecessary log4j-api dependency
[BEAM-12422] Locally-installing vendored gRPC 1.36.0-0.2

Please add a meaningful description for your change here

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Choose reviewer(s) and mention them in a comment (R: @username).
Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

`ValidatesRunner` compliance status (on master branch)

Lang	ULR	Samza	Twister2
Go	---	---	---
Java
Python	---	---	---
XLang		---	---

Examples testing status on various runners

Lang	ULR	Dataflow	Flink	Samza	Spark	Twister2
Go	---	---	---	---	---	---	---
Java	---		---	---	---	---	---
Python	---	---	---	---	---	---	---
XLang	---	---	---	---	---	---	---

Post-Commit SDK/Transform Integration Tests Status (on master branch)

Go	Java	Python

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website	Whitespace	Typescript
Non-portable
Portable	---			---	---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

suztomo · 2021-06-30T00:10:44Z

Run Java PostCommit

suztomo · 2021-06-30T00:10:54Z

Run PostCommit_Java_Dataflow

suztomo · 2021-06-30T00:11:00Z

Run PostCommit_Java_DataflowV2

suztomo · 2021-06-30T00:11:09Z

Run Spark ValidatesRunner

suztomo · 2021-06-30T00:11:14Z

Run SQL Postcommit

suztomo · 2021-06-30T00:11:49Z

buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy

@@ -647,7 +647,7 @@ class BeamModulePlugin implements Plugin<Project> {
        testcontainers_postgresql                   : "org.testcontainers:postgresql:$testcontainers_version",
        testcontainers_gcloud                       : "org.testcontainers:gcloud:$testcontainers_version",
        vendored_bytebuddy_1_11_0                   : "org.apache.beam:beam-vendor-bytebuddy-1_11_0:0.1",
-        vendored_grpc_1_36_0                        : "org.apache.beam:beam-vendor-grpc-1_36_0:0.1",
+        vendored_grpc_1_36_0                        : "org.apache.beam:beam-vendor-grpc-1_36_0:0.2",


This PR runs test with the new version of the vendored gRPC (this hasn't been published to Maven Central yet)

codecov · 2021-06-30T00:22:36Z

Codecov Report

Merging #15103 (cbccc59) into master (000ac07) will increase coverage by 1.23%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #15103      +/-   ##
==========================================
+ Coverage   82.55%   83.79%   +1.23%     
==========================================
  Files         455      441      -14     
  Lines       55143    59539    +4396     
==========================================
+ Hits        45526    49889    +4363     
- Misses       9617     9650      +33

Impacted Files	Coverage Δ
...python/apache_beam/examples/wordcount_dataframe.py	`0.00% <0.00%> (-92.60%)`	⬇️
...s/python/apache_beam/examples/snippets/snippets.py	`76.70% <0.00%> (-12.83%)`	⬇️
sdks/python/apache_beam/io/kafka.py	`80.00% <0.00%> (-5.72%)`	⬇️
...n/apache_beam/runners/direct/test_direct_runner.py	`37.50% <0.00%> (-4.81%)`	⬇️
...pache_beam/runners/interactive/interactive_beam.py	`74.72% <0.00%> (-4.81%)`	⬇️
test_config.py	`66.66% <0.00%> (-4.77%)`	⬇️
sdks/python/apache_beam/runners/test/__init__.py	`66.66% <0.00%> (-4.77%)`	⬇️
sdks/python/apache_beam/io/gcp/__init__.py	`80.00% <0.00%> (-4.62%)`	⬇️
...thon/apache_beam/runners/worker/channel_factory.py	`75.00% <0.00%> (-3.95%)`	⬇️
...thon/apache_beam/runners/worker/sdk_worker_main.py	`72.26% <0.00%> (-3.66%)`	⬇️
... and 459 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 37fd13e...cbccc59. Read the comment docs.

suztomo · 2021-06-30T00:50:31Z

Run Java_Examples_Dataflow PreCommit

suztomo · 2021-06-30T00:51:01Z

Run PostCommit_Java_Dataflow

suztomo · 2021-06-30T01:34:47Z

Run Java Precommit

suztomo · 2021-06-30T02:23:41Z

Run Java Precommit

suztomo · 2021-06-30T03:44:39Z

https://ci-beam.apache.org/job/beam_PreCommit_Java_Phrase/3798/testReport/junit/org.apache.beam.runners.flink/PortableStateExecutionTest/testExecution_streaming__false_/

Test Result (2 failures / +1)
org.apache.beam.runners.flink.PortableStateExecutionTest.testExecution[streaming: false]
org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTest.testSizes

[grpc-default-executor-5] ERROR org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer - Failed to handle for url: "InProcessServer_46"

org.apache.beam.vendor.grpc.v1p36p0.io.grpc.StatusRuntimeException: CANCELLED: Multiplexer hanging up
	at org.apache.beam.vendor.grpc.v1p36p0.io.grpc.Status.asRuntimeException(Status.java:535)
	at org.apache.beam.vendor.grpc.v1p36p0.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:478)
	at org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:553)
	at org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:68)
	at org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:739)
	at org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:718)
	at org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
...
[pool-69-thread-3] ERROR org.apache.beam.fn.harness.control.BeamFnControlClient - Exception while trying to handle InstructionRequest 1
java.lang.NullPointerException
	at org.apache.beam.fn.harness.control.BeamFnControlClient.delegateOnInstructionRequestType(BeamFnControlClient.java:139)
	at org.apache.beam.fn.harness.control.BeamFnControlClient$InboundObserver.lambda$onNext$0(BeamFnControlClient.java:110)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[grpc-default-executor-3] INFO org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService - getProcessBundleDescriptor request with id 9-3
Exception in thread "pool-69-thread-3" java.lang.NullPointerException
	at org.apache.beam.fn.harness.control.BeamFnControlClient.sendInstructionResponse(BeamFnControlClient.java:165)
	at org.apache.beam.fn.harness.control.BeamFnControlClient$InboundObserver.lambda$onNext$0(BeamFnControlClient.java:111)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

This seems unrelated to log4j.

suztomo · 2021-06-30T03:44:45Z

Run Java PreCommit

This commit is not intended to be merged into master. This commit installs the vendored gRPC 1.36.0-0.2 to local Maven repository and runs the tests against the version.

suztomo · 2021-06-30T04:00:42Z

Run Java PostCommit

suztomo · 2021-06-30T04:00:50Z

Run PostCommit_Java_Dataflow

suztomo · 2021-06-30T04:00:58Z

Run PostCommit_Java_DataflowV2

suztomo · 2021-06-30T04:01:04Z

Run Spark ValidatesRunner

suztomo · 2021-06-30T04:01:10Z

Run SQL Postcommit

suztomo · 2021-07-02T00:12:01Z

retest this please

suztomo · 2021-07-02T00:12:46Z

Run Java PostCommit

suztomo · 2021-07-02T00:12:52Z

Run PostCommit_Java_Dataflow

suztomo · 2021-07-02T00:13:21Z

Run PostCommit_Java_DataflowV2

github-actions · 2021-07-02T00:16:33Z

The Workflow run is cancelling this PR. It is an earlier duplicate of 2173354 run.

github-actions · 2021-07-02T00:16:36Z

The Workflow run is cancelling this PR. It is an earlier duplicate of 2173354 run.

suztomo · 2021-07-02T03:07:22Z

Run SQL Postcommit

suztomo · 2021-07-02T03:07:35Z

Run Java Precommit

suztomo · 2021-07-02T14:17:58Z

Run Python PreCommit

suztomo · 2021-07-02T14:21:13Z

Run Python PreCommit

suztomo marked this pull request as draft June 29, 2021 23:56

suztomo mentioned this pull request Jun 29, 2021

[BEAM-12422] Vendored gRPC: Removing unnecessary log4j-api dependency #15098

Merged

4 tasks

suztomo commented Jun 30, 2021

View reviewed changes

[BEAM-12422] Locally-installing vendored gRPC 1.36.0-0.2

6ed8c01

This commit is not intended to be merged into master. This commit installs the vendored gRPC 1.36.0-0.2 to local Maven repository and runs the tests against the version.

suztomo force-pushed the test_vendored_grpc branch from 293a129 to 6ed8c01 Compare June 30, 2021 03:59

suztomo closed this Jun 30, 2021

Merge remote-tracking branch 'origin/master' into test_vendored_grpc

cbccc59

suztomo reopened this Jul 2, 2021

suztomo closed this Sep 10, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BEAM-12422][no need to review] Testing BEAM-12422 by installing the vendored gRPC to local Maven repository #15103

[BEAM-12422][no need to review] Testing BEAM-12422 by installing the vendored gRPC to local Maven repository #15103

suztomo commented Jun 29, 2021

suztomo commented Jun 30, 2021

suztomo commented Jun 30, 2021

suztomo commented Jun 30, 2021

suztomo commented Jun 30, 2021

suztomo commented Jun 30, 2021

suztomo Jun 30, 2021

codecov bot commented Jun 30, 2021 •

edited

Loading

suztomo commented Jun 30, 2021

suztomo commented Jun 30, 2021

suztomo commented Jun 30, 2021

suztomo commented Jun 30, 2021

suztomo commented Jun 30, 2021 •

edited

Loading

suztomo commented Jun 30, 2021

suztomo commented Jun 30, 2021

suztomo commented Jun 30, 2021

suztomo commented Jun 30, 2021

suztomo commented Jun 30, 2021

suztomo commented Jun 30, 2021

suztomo commented Jul 2, 2021

suztomo commented Jul 2, 2021

suztomo commented Jul 2, 2021

suztomo commented Jul 2, 2021

github-actions bot commented Jul 2, 2021

github-actions bot commented Jul 2, 2021

suztomo commented Jul 2, 2021

suztomo commented Jul 2, 2021

suztomo commented Jul 2, 2021

suztomo commented Jul 2, 2021

[BEAM-12422][no need to review] Testing BEAM-12422 by installing the vendored gRPC to local Maven repository #15103

[BEAM-12422][no need to review] Testing BEAM-12422 by installing the vendored gRPC to local Maven repository #15103

Conversation

suztomo commented Jun 29, 2021

ValidatesRunner compliance status (on master branch)

Examples testing status on various runners

Post-Commit SDK/Transform Integration Tests Status (on master branch)

Pre-Commit Tests Status (on master branch)

GitHub Actions Tests Status (on master branch)

suztomo commented Jun 30, 2021

suztomo commented Jun 30, 2021

suztomo commented Jun 30, 2021

suztomo commented Jun 30, 2021

suztomo commented Jun 30, 2021

suztomo Jun 30, 2021

Choose a reason for hiding this comment

codecov bot commented Jun 30, 2021 • edited Loading

Codecov Report

suztomo commented Jun 30, 2021

suztomo commented Jun 30, 2021

suztomo commented Jun 30, 2021

suztomo commented Jun 30, 2021

suztomo commented Jun 30, 2021 • edited Loading

suztomo commented Jun 30, 2021

suztomo commented Jun 30, 2021

suztomo commented Jun 30, 2021

suztomo commented Jun 30, 2021

suztomo commented Jun 30, 2021

suztomo commented Jun 30, 2021

suztomo commented Jul 2, 2021

suztomo commented Jul 2, 2021

suztomo commented Jul 2, 2021

suztomo commented Jul 2, 2021

github-actions bot commented Jul 2, 2021

github-actions bot commented Jul 2, 2021

suztomo commented Jul 2, 2021

suztomo commented Jul 2, 2021

suztomo commented Jul 2, 2021

suztomo commented Jul 2, 2021

`ValidatesRunner` compliance status (on master branch)

codecov bot commented Jun 30, 2021 •

edited

Loading

suztomo commented Jun 30, 2021 •

edited

Loading