[BEAM-7578] add py37 hdfs integration test#8970
Conversation
|
Run Python PostCommit |
|
Run Python PostCommit |
|
@tvalentyn could you help me, I am trying to figure out why this test won't run? It says |
|
I think you'd need to update https://github.com/apache/beam/blob/master/settings.gradle#L135 to include the new project to make this work in current structure, however, since this is a direct runner test, let's keep it within direct runner integration test suites. |
| # By default it runs wordcount against a locally accessible HDFS service. | ||
| # See hdfs_integration_test.sh for example usage. | ||
|
|
||
| FROM python:3.5 |
There was a problem hiding this comment.
Could we make base image a parameter here and pass it externally when we run docker to build the container image? This file could start like:
ARG BASE_IMAGE
FROM $BASE_IMAGE
however we'd need to figure out the gradle bits and the location of the dockerfile so that we could call it with multiple python versions.
There was a problem hiding this comment.
Also, I'd modify the python version in-place and double check that this test passes before doing major changes to the test/config structure. Perhaps you've already done that.
There was a problem hiding this comment.
Thanks for helping @tvalentyn , I made changes to the test structure to pass the python version as an argument, all hdfs tests are passing
|
Run Python PostCommit |
Juta
force-pushed
the
hdfs
branch
3 times, most recently
from
e10b771 to
56b697a
Compare
|
Run Python PostCommit |
|
Run Portable_Python PreCommit |
|
R: @tvalentyn |
|
Run Python PreCommit |
tvalentyn
left a comment
tvalentyn
left a comment
There was a problem hiding this comment.
Thanks Juta, left a minor comment, also, let's change HDFS test to run with Python 3.7 instead of 3.5. If we pick one version I'd go with Python 3.7 at this point. Also as discussed offline, I think one Python 3 suite is sufficient for this test for now.
| trap finally EXIT | ||
|
|
||
| time docker-compose ${COMPOSE_OPT} build | ||
| time docker-compose ${COMPOSE_OPT} build --build-arg BASE_IMAGE=$1 |
There was a problem hiding this comment.
Could you please call out arguments to this script in a comment on top of the file? Thanks!
There was a problem hiding this comment.
Thanks, sorry if I wasn't clear - I actually thought of adding a comment earlier at the beginning of the script:
It could be a description of required arguments or sample usage for someone looking into how to run that script manually.
|
Run Python PostCommit |
|
Run Python PostCommit |
|
Run Python PreCommit |
|
We can also rename the PR to s/py35/py37, a committer or an author could do it. |
Juta
changed the title
|
@udim, could you help squash+merge this please? |
3 participants
Post-Commit Tests Status (on master branch)
Pre-Commit Tests Status (on master branch)
See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.