Add helper task to print pipeline options for Dataflow portability

[swegner]

Currently it is difficult to manually run a portabiltity pipeline as a
Beam user. This change adds a new Gradle task to conveniently generate
necessary artifacts and print the necessary commandline options.

Usage:

./gradlew -p runners/google-cloud-dataflow-java printFnApiPipelineOptions

Sample output:

> Task :beam-runners-google-cloud-dataflow-java:printFnApiPipelineOptions
To run a Dataflow job with portability, add the following pipeline options to your command-line:
--dataflowWorkerJar=/usr/local/google/home/swegner/beam/runners/google-cloud-dataflow-java/worker/build/libs/beam-runners-google-cloud-dataflow-java-fn-api-worker-2.9.0-SNAPSHOT.jar --workerHarnessContainerImage=gcr.io/dataflow-build/swegner/beam/java:20181107141351 --experiments=beam_fn_api

---

Follow this checklist to help us incorporate your contribution quickly and easily:

• Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

It will help us expedite review of your Pull Request if you tag someone (e.g. @username) to look at it.

Post-Commit Tests Status (on master branch)

Lang	SDK	Apex	Dataflow	Flink	Gearpump	Samza	Spark
Go		---	---	---	---	---	---
Java
Python		---			---	---	---

[swegner]

R: @boyuanzz
CC: @lukecwik @rohdesamuel @ajamato @Ardagan @HuangLED @huygaa11

[swegner]

Run Dataflow PortabilityApi ValidatesRunner

[boyuanzz]

We also launch portable pipelines in javaPrecommit: https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/examples/build.gradle#L79. Do you think it's necessary adding this help into the precommit? @swegner

[swegner]

The reason I found a need for this: I'm trying to run a simple prototype pipeline from outside of the Gradle build (as a user, following the Java Quickstart). So the commandline I need to run is something like:

mvn compile exec:java -Dexec.mainClass=example.MyPrototype -Dexec.args=" \
  --runner=DataflowRunner \
  --project=<..> --gcpTempLocation=<..> \
  [extra portability args]" \
  -Pdataflow-runner

I don't believe we publish maven artifacts to fulfill [extra portability args], so we need to run the Gradle tasks and use their results.

It seems there's a lot of redundant logic in the examples project you link above (i.e. buildAndPushDockerContainer. @boyuanzz do you know if the redundancy is intentional or should I unify them?

[boyuanzz]

For maven build, the worker jar is not published: https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/worker/build.gradle#L57 in purpose. For end users, they are supposed to use the worker jar cached by Dataflow service.

[boyuanzz]

For redundant logic, I'm not sure whether it's worthy unifying them since there are just 2 usages across the whole project, but feel free to refactor them if needed.

[swegner]

Run Dataflow PortabilityApi ValidatesRunner

[swegner]

For end users, they are supposed to use the worker jar cached by Dataflow service.

@boyuanzz do you know if the service caches worker jars for portable pipelines (in addition to legacy?) Also, are we still publishing a 'latest' image for running from a -SNAPSHOT build? /cc @lukecwik

[boyuanzz]

For end users, they are supposed to use the worker jar cached by Dataflow service.

@boyuanzz do you know if the service caches worker jars for portable pipelines (in addition to legacy?)

I believe the service also caches portable worker jar. If the dataflowWorkerJar option is missing, then it will pick the caches worker jar. Here is the design doc, maybe more illustrative: https://docs.google.com/document/d/1-m-GzkYWIODKOEl1ZSUNXYbcGRvRr3QkasfHsJxbuoA/edit#heading=h.gh88g5y0rekp

[swegner]

Thanks, good to know. I think this helper is still useful in order to use the worker image from HEAD since the cached version in the service may be updated less frequently.

I believe this is ready for review. @boyuanzz PTAL

[boyuanzz]

Yes, it would be very helpful if dev is trying to test their dataflow worker changes. Btw, the ValidatesRunner test takes ~3 hr to complete.

[boyuanzz]

Run Java PortabilityApi PostCommit

[swegner]

FYI, this appears to be breaking post-commits: https://issues.apache.org/jira/browse/BEAM-6028