Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BEAM-9510] Fixing version incompatibilities in base_image_requirements.txt #11142

Merged
merged 3 commits into from Mar 17, 2020

Conversation

davidyan74
Copy link
Contributor

Tried this out and it doesn't have any compatibility error messages any more for python3.

R: @tvalentyn

Please add a meaningful description for your change here


Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Choose reviewer(s) and mention them in a comment (R: @username).
  • Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

Post-Commit Tests Status (on master branch)

Lang SDK Apex Dataflow Flink Gearpump Samza Spark
Go Build Status --- --- Build Status --- --- Build Status
Java Build Status Build Status Build Status
Build Status
Build Status
Build Status
Build Status
Build Status Build Status Build Status
Build Status
Build Status
Python Build Status
Build Status
Build Status
Build Status
--- Build Status
Build Status
Build Status
Build Status
Build Status
--- --- Build Status
XLang --- --- --- Build Status --- --- Build Status

Pre-Commit Tests Status (on master branch)

--- Java Python Go Website
Non-portable Build Status Build Status
Build Status
Build Status Build Status
Portable --- Build Status --- ---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

@davidyan74
Copy link
Contributor Author

R: @aaltay

@tvalentyn
Copy link
Contributor

Run Dataflow Python ValidatesContainer

@tvalentyn
Copy link
Contributor

Run Python Dataflow ValidatesContainer

@aaltay aaltay requested a review from tvalentyn March 17, 2020 02:02
Copy link
Member

@aaltay aaltay left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, thank you!

@tvalentyn
Copy link
Contributor

Thanks, @davidyan74! Added a small change to reflect that typing should only be installed on Python 2 (this was recently changed in Beam master).

@tvalentyn
Copy link
Contributor

Run Python Dataflow ValidatesContainer

@tvalentyn
Copy link
Contributor

LGTM
A few things out of scope of this PR:

  1. This PR fixes the incompatibility in dependencies, but the list of dependencies is in base_image_requirements.txt is still not exhaustive. So, container builds may not be repeatable over time, and there is a risk that we will introduce incompatible dependencies down the road and won't notice it.
  2. We can consider switching the base image to python:slim-stretch instead of python:stretch to reduce the size of the images: https://github.com/apache/beam/blob/master/sdks/python/container/Dockerfile#L20
    cc: @Hannah-Jiang

@tvalentyn
Copy link
Contributor

While Jenkins does not show test result on the PR, the tests are actually running: https://builds.apache.org/job/beam_PostCommit_Py_ValCont_PR/122/.
Let's check the console logs that there are no conflicts before the merge.

@tvalentyn
Copy link
Contributor

Run Python Dataflow ValidatesContainer

@tvalentyn
Copy link
Contributor

Test logs LGTM: https://builds.apache.org/job/beam_PostCommit_Py_ValCont_PR/123/timestamps/?time=HH:mm:ss&timeZone=GMT-7&appendLog&locale=en_US

FYI @amaliujia - containers on the latest release branch may also be broken and require changes to fix incompatible dependencies. Do you think you can check this? Otherwise it may come up during release validation. You can make a similar change to the release branch, then have a look at the logs of ValidatesContainer suite against the release branch, and make sure pip does not complain about incompatible dependencies.

@Hannah-Jiang and @davidyan74 also can help if you have questions.

@tvalentyn tvalentyn merged commit c121f73 into apache:master Mar 17, 2020
@davidyan74
Copy link
Contributor Author

Thanks @tvalentyn. We might want to add pip check after pip install in our tests to make sure there is no incompatibility going forward. pip install does not return an error exit code for broken dependencies for historical reasons.

@davidyan74
Copy link
Contributor Author

Fiiled BEAM-9530.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants