[BEAM-6049] Add option to load job to GCS in Dataflow Runner

[foegler]

Support an option "uploadGraph" which will load the Job object to the user's staging directory, remove the embedded graph before submission, and retrieve the graph from within the service. This allows the DataflowRunner to support significantly larger job definitions.

R: @swegner

Follow this checklist to help us incorporate your contribution quickly and easily:

• Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

It will help us expedite review of your Pull Request if you tag someone (e.g. @username) to look at it.

Post-Commit Tests Status (on master branch)

Lang	SDK	Apex	Dataflow	Flink	Gearpump	Samza	Spark
Go		---	---	---	---	---	---
Java
Python		---			---	---	---

[swegner]

Thanks for this contribution, and for adding unit tests.

It would also be useful to have an integration test, to ensure that a pipeline submitted to Dataflow service with this option actually runs successfully.

@robertwb do we have Dataflow-specific tests which validate integration between runner harness and service? Searching around a bit I don't see any tests within the beam-runners-google-cloud-dataflow-java project which actually run pipelines.

[swegner]

Beam style rules dictate listing individual imports and not wildcards. See checkstyle failure.

FYI, you can re-run checkstyle directly via ./gradlew checkstyleMain checkstyleTest

[foegler]

Hmmm...I don't think I did that manually. Is this something IntelliJ does automatically and do you happen to know how to disable it?

[swegner]

https://stackoverflow.com/questions/3587071/disable-intellij-starred-package-imports

[foegler]

Thanks!

[swegner]

Will this eventually become the default behavior? Is there any reason to support both?

If this will become default, I suggest using an experiment pipeline option rather than a named option. PipelineOptions become part of the public API and are harder to drop later.

[foegler]

Done.

[foegler]

It's not supported in the service yet. I have that change pending because I'd like to submit it with an integration test before submitting. I can leave this JIRA open until I add an integration test here too once it's available in the service.

[swegner]

LGTM. It seems the Java post-commits are currently red. I'll merge once the tests look healthy.

[swegner]

Run JavaPortabilityApi PreCommit

[swegner]

All green!

[bbhoss]

Is this functional? We just upgraded to 2.9 and are hitting the size limitation on one of the bigger jobs. I have specified the following options on the command line:

 --experiments=upload_graph --tempLocation=gs://mybucket/foo --stagingLocation=gs://mybucket/foo

I have verified the graph is uploaded to the directory, but dataflow immediately fails due to
Failed to retrieve the job graph. Please see the job logs for more details.
The logs that I can see don't have any details (Logs in the Dataflow UI and Stackdriver are blank other than this message)

[bbhoss]

Whoops, the logs do add

Runnable workflow has no steps specified.

Which I don't believe is helpful.

[Jdban]

@bbhoss I'm getting the same error with 2.10.0 and 2.11.0
Runnable workflow has no steps specified.

We enabled it via:

DataflowPipelineOptions options = PipelineOptionsFactory.as(DataflowPipelineOptions.class);
options.setExperiments(Arrays.asList("upload_graph"));

@swegner/@foegler
How do we get this working with GCP dataflow? Has the feature been enabled in the service yet?

[Jdban]

According to GCP support:

After my initial research, I was able to confirm this feature is not yet supported on the Dataflow service side. The Dataflow Engineering team is working on it, but there is no ETA yet. However, it is not likely to be available before the end of Q2.

Good thing we pay for GCP support or we'd never know...

Very confusing that the API library suggests a solution that isn't supported yet...

[bbhoss]

Thanks for the info. It definitely seems like it was merged erroneously, unless there is an EAP we are unaware of.

…

On Mon, Mar 11, 2019 at 11:55 Jason Banich ***@***.***> wrote: According to GCP support: After my initial research, I was able to confirm this feature is not yet supported on the Dataflow service side. The Dataflow Engineering team is working on it, but there is no ETA yet. However, it is not likely to be available before the end of Q2. Good thing we pay for GCP support or we'd never know... Very confusing that the API library suggests a solution that isn't supported yet... — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#7047 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAbhP98U3ELe9GrAh6JPni7mg0F55BMks5vVnxtgaJpZM4Ye6ZN> .

[Jdban]

FYI the feature appears to be live now: https://issuetracker.google.com/issues/139540290