[BEAM-5978] Changing parallelism for wordcount to 1

[angoenka]

Local execution environment sets the parallelism of the pipeline same as the number of cores on the machine. In my case, its 12. As we have 12 threads in SDKHarness, we get a deadlock on my machine.
Running on a cluster does not have this issue as the default parallelism there is 1.

Follow this checklist to help us incorporate your contribution quickly and easily:

• Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

It will help us expedite review of your Pull Request if you tag someone (e.g. @username) to look at it.

Post-Commit Tests Status (on master branch)

Lang	SDK	Apex	Dataflow	Flink	Gearpump	Samza	Spark
Go		---	---	---	---	---	---
Java
Python		---			---	---	---

[tweise]

Please see #6954 (comment)

This is a problem specific to batch mode. Please set the workaround accordingly (there is already another one below) and add a comment pointing to the JIRA.

[angoenka]

Thanks. I will apply this parameter to batch only then.

[angoenka]

Updated!

[tweise]

Let's wait for @mxm to take a look as well. I experimented a bit with the parallelism settings and documented it on https://issues.apache.org/jira/browse/BEAM-5167

Even though not needed for streaming currently, it is generally advisable to be explicit about the parallelism (predictability WRT network buffers etc.)

[tweise]

We could also fold this change into #7180

Unrelated to this change, note that currently wordcount doesn't work on macOS with the docker jobserver:

./gradlew :beam-sdks-python:portableWordCount
...
INFO:root:Using latest locally built Python SDK docker image.
docker: Error response from daemon: driver failed programming external connectivity on endpoint cranky_pasteur (f9b3843dada38679d655f258ca40af36b030b469f2abbd2277db199683018b53):  (iptables failed: iptables --wait -t nat -A DOCKER -p tcp -d 0/0 --dport 32817 -j DNAT --to-destination 172.17.0.2:0 ! -i docker0: iptables v1.6.2: Port `0' not valid

Try `iptables -h' or 'iptables --help' for more information.
 (exit status 2)).

[mxm]

As we have 12 threads in SDKHarness, we get a deadlock on my machine

Default should be 1. You would get auto-set 12 threads with it set to 0.

IMHO ok to merge this as a fix for as long as the underlying issue is still pending.

[angoenka]

@mxm The underlying issue of flink and sdkHarness scheduling is still pending and requires more thought and discussion.

@tweise Mac wordcount should be working with #7178 Let me know if its otherwise.

[angoenka]

Shall we merge this PR?

[mxm]

Go ahead :)

[tweise]

if the intention is to use it only for batch, you would need to add {..}

[mxm]

Good catch. That's why it's better to always use brackets for if blocks.

[angoenka]

Ahh, Totally missed it. BTW python works with indentation so it might have worked with Python :)

Should also apply this change to streaming as we don't want parallelism in streaming as well.

[mxm]

Wouldn't make this more restrictive than necessary. So only batch for now.

[tweise]

@angoenka the error posted above applies to Docker execution (the default), as shown in the command.

[angoenka]

Updated the PR