[BEAM-7761] Add transform_name_mapping pipeline option for python sdk#9072
[BEAM-7761] Add transform_name_mapping pipeline option for python sdk#9072aaltay merged 2 commits intoapache:masterfrom
Conversation
y1chi
force-pushed
the
transform_name_mapping
branch
3 times, most recently
from
6454014 to
dfb2704
Compare
|
R: @aaltay Hi ahmet would you mind take a look on this PR? |
y1chi
changed the title
|
R; @dustin12 could you take a look at this? |
tried to add an integration test for dataflow but I guess it's only gonna work after sdk has the option. |
dustin12
left a comment
dustin12
left a comment
There was a problem hiding this comment.
I believe our Java streaming numbers integration test and streaming drain test test this option at least somewhat. Does python have the equivalent tests yet?
| if not view.job_name: | ||
| errors.extend(self._validate_error( | ||
| 'Existing job name must be provided when updating a pipeline.')) | ||
| if view.transform_name_mapping: |
There was a problem hiding this comment.
Can we also confirm in here that the job is streaming and update.
There was a problem hiding this comment.
added streaming option validation below.
| 'See https://cloud.google.com/dataflow/pipelines/' | ||
| 'See https://cloud.google.com/dataflow/docs/guides/' | ||
| 'updating-a-pipeline') | ||
| parser.add_argument('--transform_name_mapping', |
There was a problem hiding this comment.
As far as I can tell python does not have a place for streaming specific options. Is it worth making it and moving this there?
There was a problem hiding this comment.
This option is probably more update related, which is currently only support with dataflow runner I guess?
There was a problem hiding this comment.
That is correct. Update is only supported by Dataflow.
y1chi
force-pushed
the
transform_name_mapping
branch
from
b80c4c8 to
3115bab
Compare
I'm adding similar update integration tests for python, but did not find any tests related to transform name mapping. |
|
Run Portable_Python PreCommit |
|
Run Python_PVR_Flink PreCommit |
3 participants
Add transform_name_mapping pipeline option for python sdk.
It is useful for updating an existing pipeline on Google Cloud Dataflow: https://cloud.google.com/dataflow/docs/guides/updating-a-pipeline
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
R: @username).[BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replaceBEAM-XXXwith the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.Post-Commit Tests Status (on master branch)
Pre-Commit Tests Status (on master branch)
See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.