[BEAM-11934] Remove Dataflow override of streaming WriteFiles with runner determined sharding

[nehsyc]

• Remove Dataflow override of streaming WriteFiles with runner determined sharding in favor of auto sharding.
• Update FileIO documentation.

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Choose reviewer(s) and mention them in a comment (R: @username).
• Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

ValidatesRunner compliance status (on master branch)

Lang	ULR	Dataflow	Flink	Samza	Spark	Twister2
Go	---					---
Java
Python	---			---		---
XLang				---		---

Examples testing status on various runners

Lang	ULR	Dataflow	Flink	Samza	Spark	Twister2
Go	---	---	---	---	---	---	---
Java	---		---	---	---	---	---
Python	---	---	---	---	---	---	---
XLang	---	---	---	---	---	---	---

Post-Commit SDK/Transform Integration Tests Status (on master branch)

Go	Java	Python

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website	Whitespace	Typescript
Non-portable
Portable	---			---	---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

[nehsyc]

R: @chamikaramj

[chamikaramj]

Do you think we should keep the old behavior as an option in case some customers run into issues ?

[lukecwik]

Do you think we should keep the old behavior as an option in case some customers run into issues ?

We should make it the default for runner v2.

@reuvenlax Is it worth making this the default and requiring users to pass in a flag to maintain support for pipeline update for older versions?

[nehsyc]

Do you think we should keep the old behavior as an option in case some customers run into issues ?

The override was not used because before the introduction of auto sharding, streaming write was required to specify a fixed number of shards otherwise fails:

beam/sdks/java/core/src/main/java/org/apache/beam/sdk/io/WriteFiles.java

Line 319 in d7eb872

getComputeNumShards() != null || getNumShardsProvider() != null,

I forgot to remove the override when I loosed the check to allow runner determined sharding for unbounded data :\

[nehsyc]

Do you think we should keep the old behavior as an option in case some customers run into issues ?

We should make it the default for runner v2.

@reuvenlax Is it worth making this the default and requiring users to pass in a flag to maintain support for pipeline update for older versions?

I assume that you meant making runner determined sharding default for runner v2.

Older versions don't support runner determined sharding for unbounded data (strictly speaking, versions before 2.29.0) so fixed sharding option will be carried over on pipeline update. We could make new pipelines opt in by default.

[chamikaramj]

I guess the only concern will be users who are upgrading from Beam 2.29.0 or later then, right ?

Since it's just a couple versions (and approximately three months) I don't think there's a need to be concerned about preserving backwards compatibility.

[chamikaramj]

Run Java PreCommit

[chamikaramj]

Run Java_Examples_Dataflow PreCommit

[lukecwik]

Do you think we should keep the old behavior as an option in case some customers run into issues ?

We should make it the default for runner v2.
@reuvenlax Is it worth making this the default and requiring users to pass in a flag to maintain support for pipeline update for older versions?

I assume that you meant making runner determined sharding default for runner v2.

Older versions don't support runner determined sharding for unbounded data (strictly speaking, versions before 2.29.0) so fixed sharding option will be carried over on pipeline update. We could make new pipelines opt in by default.

SGTM

[nehsyc]

PreCommit failures look irrelevant:

org.apache.beam.sdk.transforms.ParDoLifecycleTest.testTeardownCalledAfterExceptionInProcessElementStateful
org.apache.beam.runners.flink.FlinkSavepointTest.testSavepointRestoreLegacy
org.apache.beam.runners.flink.FlinkSavepointTest.testSavepointRestoreLegacy

[chamikaramj]

Run Java PreCommit

[nehsyc]

cc:
@angoenka

[nehsyc]

All tests are passing now :D

[chamikaramj]

Thanks. LGTM.

[reuvenlax]

i suspect this PR is causing the WordCount test to become extremely flaky. I think this code actually was being actuated before, and now without it every bundle is becoming a separate file to writet.