Upgrade to Gradle 7.4

[kennknowles]

Routine upgrade. Most prior minor versions have not required other changes. Will run various tests.

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Choose reviewer(s) and mention them in a comment (R: @username).
• Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

[codecov]

Codecov Report

Merging #16931 (e7ae895) into master (5b0e92f) will decrease coverage by 0.75%.
The diff coverage is n/a.

❗ Current head e7ae895 differs from pull request most recent head 72cbbe1. Consider uploading reports for the commit 72cbbe1 to get more accurate results

@@            Coverage Diff             @@
##           master   #16931      +/-   ##
==========================================
- Coverage   74.11%   73.36%   -0.76%     
==========================================
  Files         679      660      -19     
  Lines       89112    86796    -2316     
==========================================
- Hits        66049    63681    -2368     
- Misses      21910    22077     +167     
+ Partials     1153     1038     -115     

Flag	Coverage Δ
python	83.62% <ø> (-0.05%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
sdks/go/pkg/beam/runners/direct/impulse.go	0.00% <0.00%> (-100.00%)	⬇️
sdks/go/pkg/beam/core/util/reflectx/types.go	0.00% <0.00%> (-94.74%)	⬇️
sdks/go/pkg/beam/core/util/reflectx/call.go	0.00% <0.00%> (-89.75%)	⬇️
sdks/go/pkg/beam/runners/direct/buffer.go	0.00% <0.00%> (-77.47%)	⬇️
sdks/go/pkg/beam/core/util/reflectx/structs.go	0.00% <0.00%> (-76.93%)	⬇️
sdks/go/pkg/beam/core/runtime/graphx/cogbk.go	0.00% <0.00%> (-72.73%)	⬇️
sdks/go/pkg/beam/runners/direct/direct.go	1.36% <0.00%> (-65.00%)	⬇️
sdks/go/pkg/beam/io/datastoreio/datastore.go	16.03% <0.00%> (-51.78%)	⬇️
sdks/go/pkg/beam/runners/direct/gbk.go	22.76% <0.00%> (-49.60%)	⬇️
sdks/go/pkg/beam/io/textio/textio.go	23.77% <0.00%> (-41.81%)	⬇️
... and 107 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5b0e92f...72cbbe1. Read the comment docs.

[kennknowles]

Failure is cyclic dependency (at the module level) at evaluation time:

:runners:google-cloud-dataflow-java
→ :runners:google-cloud-dataflow-java:buildAndPushDockerPythonContainer
→ :sdks:python
→ :runners:google-cloud-dataflow-java:worker

at

beam/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy

Line 2455 in 0e6e77b

project.evaluationDependsOn(":runners:google-cloud-dataflow-java:worker")

It seems strange for applyPythonNature to have a hard dep on the gradle config for the Dataflow Java worker. I think this goes back a long way, and something new in how evaluation proceeds has made it a problem. @tvalentyn do you have any context on this? Maybe we can just remove it?

[tvalentyn]

It's probably to allow legacy JRH + custom Dataflow worker jar for Python FnAPI pipelines. Do we still need that or UW is now a strict superset?

[tvalentyn]

It's probably to allow legacy JRH + custom Dataflow worker jar for Python FnAPI pipelines. Do we still need that or UW is now a strict superset?

That may have been the overall reason why it came into the picture, but I am not sure what's the impact of that particular line in applyPythonNature. We could try to remove it and see if anything breaks. Likely, we should see it immediately in a precommit failure

[kennknowles]

Looks like the fix was straightforward - move the dependency to the gradle file that actually has the dependency.

[tvalentyn]

LGTM if tests pass, also, we should consider Gradle 7.5 once available. It may help with flakes: gradle/gradle#14693 (comment)

[kennknowles]

Failure in ElasticSearchIOTest:

org.testcontainers.containers.ContainerFetchException: Can't get Docker image: RemoteDockerImage(imageName=docker.elastic.co/elasticsearch/elasticsearch:5.6.3, imagePullPolicy=DefaultPullPolicy())

[kennknowles]

run java precommit

[kennknowles]

Failure in org.apache.beam.sdk.io.gcp.spanner.changestreams.SpannerChangeStreamErrorTest.testUnavailableExceptionRetries:

java.lang.AssertionError: 
Expected: (an instance of org.apache.beam.sdk.Pipeline$PipelineExecutionException and exception with message a string containing "RESOURCE_EXHAUSTED")
     but: exception with message a string containing "RESOURCE_EXHAUSTED" message was "com.google.cloud.spanner.SpannerException: INTERNAL: io.grpc.StatusRuntimeException: INTERNAL: There is no result registered for the statement: SELECT t.table_name FROM information_schema.tables AS t WHERE t.table_catalog = '' AND t.table_schema = '' AND t.table_name = 'CDC_Partitions_Metadata_my_database_d17215d2_25b5_4096_8928_bd581b7f8d4b'
Call TestSpannerImpl#addStatementResult(StatementResult) before executing the statement. - Statement: 'SELECT t.table_name FROM information_schema.tables AS t WHERE t.table_catalog = '' AND t.table_schema = '' AND t.table_name = 'CDC_Partitions_Metadata_my_database_d17215d2_25b5_4096_8928_bd581b7f8d4b''"
Stacktrace was: org.apache.beam.sdk.Pipeline$PipelineExecutionException: com.google.cloud.spanner.SpannerException: INTERNAL: io.grpc.StatusRuntimeException: INTERNAL: There is no result registered for the statement: SELECT t.table_name FROM information_schema.tables AS t WHERE t.table_catalog = '' AND t.table_schema = '' AND t.table_name = 'CDC_Partitions_Metadata_my_database_d17215d2_25b5_4096_8928_bd581b7f8d4b'
Call TestSpannerImpl#addStatementResult(StatementResult) before executing the statement. - Statement: 'SELECT t.table_name FROM information_schema.tables AS t WHERE t.table_catalog = '' AND t.table_schema = '' AND t.table_name = 'CDC_Partitions_Metadata_my_database_d17215d2_25b5_4096_8928_bd581b7f8d4b''

[kennknowles]

run java precommit

[kennknowles]

This time org.apache.beam.sdk.io.gcp.spanner.changestreams.SpannerChangeStreamErrorTest.testInvalidRecordReceived failed with

java.lang.AssertionError: 
Expected: (an instance of org.apache.beam.sdk.Pipeline$PipelineExecutionException and exception with message a string containing "Field not found")
     but: exception with message a string containing "Field not found" message was "java.lang.IllegalArgumentException: Trying to claim offset 1645992603152000 before start of the range [1645992603152001, 253402300799999999)"
Stacktrace was: org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.lang.IllegalArgumentException: Trying to claim offset 1645992603152000 before start of the range [1645992603152001, 253402300799999999)
	at org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:373)
	at org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:341)
	at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:218)
	at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
	at org.apache.beam.sdk.Pipeline.run(Pipeline.java:323)
	at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:399)
	at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:335)
	at org.apache.beam.sdk.io.gcp.spanner.changestreams.SpannerChangeStreamErrorTest.testInvalidRecordReceived(SpannerChangeStreamErrorTest.java:234)

(https://ci-beam.apache.org/job/beam_PreCommit_Java_Phrase/4668/testReport/junit/org.apache.beam.sdk.io.gcp.spanner.changestreams/SpannerChangeStreamErrorTest/testInvalidRecordReceived/)

[kennknowles]

Run Java PreCommit

[kennknowles]

A couple of interesting failures that look real but probably not related, unless it is dependency resolution that causes different tasks to run (which previous issue suggests might be possible): https://scans.gradle.com/s/3bkkuratojayi

[kennknowles]

./gradlew :sdks:java:fn-execution:analyzeClassesDependencies --scan passed after a rebase: https://gradle.com/s/uh5uf2rhavjpa

./gradlew :sdks:java:fn-execution:checkstyleMain --scan passes as well: https://gradle.com/s/zxvaa3ruuynrs

[kennknowles]

Gradle build daemon disappeared unexpectedly

[kennknowles]

Run Java_Examples_Dataflow_Java17 PreCommit

[kennknowles]

Run CommunityMetrics PreCommit

[kennknowles]

Failures:

org.apache.beam.sdk.io.gcp.bigquery.BigQueryIOWriteTest.testWriteDynamicDestinationsBatchWithSchemas[3]

Can't add an element past the end of time (294247-01-10T04:00:54.775Z), got timestamp 294247-01-10T04:00:54.775Z

org.apache.beam.sdk.io.gcp.spanner.changestreams.SpannerChangeStreamErrorTest.testInvalidRecordReceived

java.lang.IllegalArgumentException: Trying to claim offset 1646078806589000 before start of the range [1646078806589001, 253402300799999999)

Both look like actual test failures. Not likely gradle related, nor infra flakes.

[kennknowles]

The SQL failures are both the Gradle bug:

12:07:08 Execution failed for task ':runners:flink:1.14:job-server:shadowJar'.
12:07:08 > Failed to load cache entry for task ':runners:flink:1.14:job-server:shadowJar'

[kennknowles]

At this point, I propose we merge and continue with further upgrades.

[kennknowles]

As you note, Gradle 7.5 may help with the failure that occurred on the SQL builds.

[kennknowles]

Run SQL_Java11 PreCommit

[kennknowles]

Run SQL_Java17 PreCommit

[kennknowles]

Run CommunityMetrics PreCommit

[kennknowles]

Run Java PreCommit

[aaltay]

What is the next step for this PR? Merge/Close?

[kennknowles]

I believe the PR is fine, and just never got past flakes. it is old enough now that syncing to see where we are at might make sense

[kileys]

Run GoPortable PreCommit

[kileys]

Run Java PreCommit

[aaltay]

Tests are still failing. Still flakes? Real issues?

[kennknowles]

It was a blend. Rebased and pushed to see where we are at now.

[kennknowles]

Run CommunityMetrics PreCommit

[kennknowles]

Community metrics is known unhealthy: https://issues.apache.org/jira/browse/BEAM-14017

[kennknowles]

PulsarIOTest.testReadFromSimpleTopic is very flaky: https://issues.apache.org/jira/browse/BEAM-14269

[kennknowles]

Portable python precommit failure is ":sdks:python:test-suites:portable:py36:portableWordCountFlinkRunnerBatch" failing because Python 3.6 is not installed in the image. I am unsure of the status here. Does not seem Gradle related, unless the container build was not trigger properly due to the version upgrade. I don't see a Jira for it.

[kennknowles]

Run Portable_Python PreCommit

[kennknowles]

OK we have narrowed it down to known flakes and red tests. I am going to merge so we can move on since there are follow-up upgrades too.

[kennknowles]

Oh, no I am not :-)

I am going to wait for someone to LGTM and then we can merge. Unless you disagree and think we can deflake these tests.

[robertwb]

The test failure in org.apache.beam.sdk.io.pulsar.PulsarIOTest.testReadFromSimpleTopic looks bad but irrelevant.

[Abacn]

Hi, Github Action of Java Wordcount Dataflow failure since this update:

FAILURE: Build failed with an exception.

* Where:
Build file '/home/runner/work/beam/beam/runners/google-cloud-dataflow-java/build.gradle' line: 47

* What went wrong:
A problem occurred evaluating project ':runners:google-cloud-dataflow-java'.
> Circular referencing during evaluation for project ':examples:java'.

Does anyone have an idea about resolving this?

[kennknowles]

Interesting. We need to find a way to break the cycle :-)

At a high level, it is clear to me that :runners:google-cloud-dataflow-java should not depend on :examples:java. The problem is likely the integration testing is set up in the wrong module. Previously this did not create a cycle, but now it does. @y1chi does deferred evaluation help this?

[y1chi]

deferred configuration would still run into circular referencing issue when the task is being executed.

[Abacn]

By setting manual print statements, found the error happens when evaluating :runnders:google-cloud-dataflow-java at

runners/google-cloud-dataflow-java/build.gradle L47:
evaluationDependsOn(":examples:java")
examples/java/build.gradle L32:
enableJavaPerformanceTesting()
beam/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy::BeamModulePlugin.apply() L1790:
project.evaluationDependsOn(":runners:google-cloud-dataflow-java:worker:legacy-worker")

circular happens here.
It seems that gradle somehow detects this as a circular dependency, which should not been so. Probably a gradle bug?

[Abacn]

Found an existing issue of gradle (gradle/gradle#847) seems related. Here the full name of legacy-worker project contains the name :runnders:google-cloud-dataflow-java .