[BEAM-6955] Support Dataflow --sdk_location with modified version number #8189
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Determine the version tag to use for the Google Container Registry, for the service image versions to use on the Dataflow worker nodes. Users of Dataflow may be using a locally-modified version of Apache Beam, which they submit to Dataflow with the --sdk_location option. Those users would most likely modify the version number of Apache Beam, so they can distinguish it from the public distribution of Apache Beam. However, the remote nodes in Dataflow still need to bootsrap the worker service with a Docker image that a version tag exists for. The most appropriate way for system integrators to modify the Apache Beam version number would be to add a Local Version Identifier: https://www.python.org/dev/peps/pep-0440/#local-version-identifiers If people only use Local Version Identifiers, then we could use the "public" attribute of the pkg_resources version object. If people instead use a post-release version identifier: https://www.python.org/dev/peps/pep-0440/#post-releases then only the "base_version" attribute would work both of these version number changes. Since Dataflow documentation does not specify how to modify version numbers, I am choosing to use "base_version" attribute.
Contributor
|
R: @tvalentyn could you please take a look at this? |
Contributor
|
@dlesco I replied on the Jira, PTAL. |
Contributor
Contributor
Author
|
@tvalentyn Closing this PR... |
Contributor
|
Thanks again for bringing up this issue and suggested solution, @dlesco ! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Determine the version tag to use for the Google Container Registry,
for the service image versions to use on the Dataflow worker nodes.
Users of Dataflow may be using a locally-modified version of
Apache Beam, which they submit to Dataflow with the
--sdk_location option. Those users would most likely modify the
version number of Apache Beam, so they can distinguish it from the
public distribution of Apache Beam.
However, the remote nodes in Dataflow still need to bootsrap the
worker service with a Docker image that a version tag exists for.
The most appropriate way for system integrators to modify the
Apache Beam version number would be to add a Local Version Identifier:
https://www.python.org/dev/peps/pep-0440/#local-version-identifiers
If people only use Local Version Identifiers, then we could use
the "public" attribute of the pkg_resources version object.
If people instead use a post-release version identifier:
https://www.python.org/dev/peps/pep-0440/#post-releases
then only the "base_version" attribute would work both of these
version number changes.
Since Dataflow documentation does not specify how to modify
version numbers, I am choosing to use "base_version" attribute.
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
R: @username).[BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replaceBEAM-XXXwith the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.Post-Commit Tests Status (on master branch)
Pre-Commit Tests Status (on master branch)
See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.