[BEAM-8348] set job_name in portable_runner.py job request

[ibzib]

Post-Commit Tests Status (on master branch)

Lang	SDK	Apex	Dataflow	Flink	Gearpump	Samza	Spark
Go		---	---		---	---
Java
Python		---			---	---
XLang	---	---	---		---	---	---

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website
Non-portable
Portable	---		---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

[tweise]

GoogleCloudOptions doesn't seem appropriate for this. Should we relocate the option?

[ibzib]

That was initially my preferred approach, but I don't see a way to do it without breaking all the existing Dataflow pipelines that set a job name. See #9724

[tweise]

Would it be possible to replicate the option and deprecate it in GoogleCloudOptions?

FWIW this option also exists in the Flink runner and the job_name is already reflected in the pipeline (see #9752 (comment)), so what is the effect of this change going to be?

[ibzib]

PipelineOptions does not allow the same option to belong to multiple subclasses.

This PR changes job_name in the job request, not the pipeline options. e.g. here https://github.com/apache/beam/blob/master/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/jobsubmission/InMemoryJobService.java#L124

[tweise]

I see, what is the user facing impact?

Regarding PipelineOptions, can it not be changed to allow for the co-existence of job_name in both classes for a transition period? @robertwb

[robertwb]

Yes, we could add an explicit @property in GoogleCloudOptions that delegates to the one in PipelineOptions.

[ibzib]

It's a good idea. Unfortunately, it looks like the override of __settatr__ would take precedence over the property's setter. I'm not too familiar with how all these Python internals work, though, so I could be missing something obvious?

beam/sdks/python/apache_beam/options/pipeline_options.py

Line 336 in 16b4afa

def __setattr__(self, name, value):

[robertwb]

Huh. Perhaps override _visible_option_list?

[robertwb]

Any update on this?

[ibzib]

I haven't made any progress on this. It's not a terribly important feature.

[stale]

This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions.