[BEAM-2963] Stage the portable pipeline; put URL in pipeline options

[kennknowles]

Follow this checklist to help us incorporate your contribution quickly and easily:

• Make sure there is a JIRA issue filed for the change (usually before you start working on it). Trivial changes like typos do not require a JIRA issue. Your pull request should address just this issue, without pulling in other changes.
• Each commit in the pull request should have a meaningful subject line and body.
• Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue.
• Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
• Run mvn clean verify to make sure basic checks pass. A more thorough check will be performed on your pull request automatically.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

---

[kennknowles]

R: @lukecwik

Same story, new place to put the info.

[lukecwik]

To my knowledge, arbitrary pipeline options don't make it to the runner harness.
Why the swap away from worker metadata configuration, did using a map -> list of kv conversion not work?

[kennknowles]

The metadata field is a Map. That wouldn't typecheck.

[kennknowles]

You mean hacking it with a custom jackson hook? I'd rather not mess with that, since pipelineoptions is already a pretty arbitrary bag and actually sort of better than VM metadata for this kind of data.

[lukecwik]

Arbitrary pipeline options still don't make it to the runner harness.

[kennknowles]

@herohde can you comment on this? I think that we have adequate control to get this info to the right place noninvasively.

[herohde]

Pipeline options are available to the runner harness in Dataflow, so this approach would work fine. It's trivial to make the runner harness grab what it needs here.

[herohde]

LGTM

[lukecwik]

LGTM