-
Notifications
You must be signed in to change notification settings - Fork 28.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-28468][INFRA][2.4] Upgrade pip to fix sphinx
install error
#25226
Conversation
@@ -60,6 +60,7 @@ RUN apt-get clean && apt-get update && $APT_INSTALL gnupg ca-certificates apt-tr | |||
$APT_INSTALL nodejs && \ | |||
# Install needed python packages. Use pip for installing packages (for consistency). | |||
$APT_INSTALL libpython2.7-dev libpython3-dev python-pip python3-pip && \ | |||
pip install --upgrade pip && hash -r pip && \ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
hash -r pip
is also required. Without that, we will hit the following.
Traceback (most recent call last):
File "/usr/bin/pip", line 9, in <module>
from pip import main
ImportError: cannot import name main
sphinx
import errorsphinx
import error
sphinx
import errorsphinx
import error
sphinx
import errorsphinx
install error
cc @dbtsai , @srowen , @gatorsmile , @HyukjinKwon |
BTW, @gatorsmile . Did you use |
This comment has been minimized.
This comment has been minimized.
retest this please |
Thank you for retriggering, @HyukjinKwon . |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks fine to me.
Test build #107993 has finished for PR 25226 at commit
|
retest this please |
Test build #108010 has finished for PR 25226 at commit
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for review and approval, @HyukjinKwon and @srowen .
Merged to branch-2.4.
## What changes were proposed in this pull request? Spark 2.4.x should be a LTS version and we should use the release script in `branch-2.4` to avoid the previous mistakes. Currently, `do-release-docker.sh` fails at `sphinx` installation to `Python 2.7` at `branch-2.4` only. This PR aims to upgrade `pip` to handle this. ``` $ dev/create-release/do-release-docker.sh -d /tmp/spark-2.4.4 -n ... = Building spark-rm image with tag latest... Command: docker build -t spark-rm:latest --build-arg UID=501 /Users/dhyun/APACHE/spark-2.4/dev/create-release/spark-rm Log file: docker-build.log // Terminated. ``` ``` $ tail /tmp/spark-2.4.4/docker-build.log Collecting sphinx Downloading https://files.pythonhosted.org/packages/89/1e/64c77163706556b647f99d67b42fced9d39ae6b1b86673965a2cd28037b5/Sphinx-2.1.2.tar.gz (6.3MB) Complete output from command python setup.py egg_info: ERROR: Sphinx requires at least Python 3.5 to run. ---------------------------------------- Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-2tylGA/sphinx/ You are using pip version 8.1.1, however version 19.1.1 is available. You should consider upgrading via the 'pip install --upgrade pip' command. ``` The following is the short reproducible step. ``` $ docker build -t spark-rm-test2 --build-arg UID=501 dev/create-release/spark-rm ``` ## How was this patch tested? Manual. ``` $ docker build -t spark-rm-test2 --build-arg UID=501 dev/create-release/spark-rm ``` Closes #25226 from dongjoon-hyun/SPARK-28468. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
## What changes were proposed in this pull request? Spark 2.4.x should be a LTS version and we should use the release script in `branch-2.4` to avoid the previous mistakes. Currently, `do-release-docker.sh` fails at `sphinx` installation to `Python 2.7` at `branch-2.4` only. This PR aims to upgrade `pip` to handle this. ``` $ dev/create-release/do-release-docker.sh -d /tmp/spark-2.4.4 -n ... = Building spark-rm image with tag latest... Command: docker build -t spark-rm:latest --build-arg UID=501 /Users/dhyun/APACHE/spark-2.4/dev/create-release/spark-rm Log file: docker-build.log // Terminated. ``` ``` $ tail /tmp/spark-2.4.4/docker-build.log Collecting sphinx Downloading https://files.pythonhosted.org/packages/89/1e/64c77163706556b647f99d67b42fced9d39ae6b1b86673965a2cd28037b5/Sphinx-2.1.2.tar.gz (6.3MB) Complete output from command python setup.py egg_info: ERROR: Sphinx requires at least Python 3.5 to run. ---------------------------------------- Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-2tylGA/sphinx/ You are using pip version 8.1.1, however version 19.1.1 is available. You should consider upgrading via the 'pip install --upgrade pip' command. ``` The following is the short reproducible step. ``` $ docker build -t spark-rm-test2 --build-arg UID=501 dev/create-release/spark-rm ``` ## How was this patch tested? Manual. ``` $ docker build -t spark-rm-test2 --build-arg UID=501 dev/create-release/spark-rm ``` Closes apache#25226 from dongjoon-hyun/SPARK-28468. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
## What changes were proposed in this pull request? Spark 2.4.x should be a LTS version and we should use the release script in `branch-2.4` to avoid the previous mistakes. Currently, `do-release-docker.sh` fails at `sphinx` installation to `Python 2.7` at `branch-2.4` only. This PR aims to upgrade `pip` to handle this. ``` $ dev/create-release/do-release-docker.sh -d /tmp/spark-2.4.4 -n ... = Building spark-rm image with tag latest... Command: docker build -t spark-rm:latest --build-arg UID=501 /Users/dhyun/APACHE/spark-2.4/dev/create-release/spark-rm Log file: docker-build.log // Terminated. ``` ``` $ tail /tmp/spark-2.4.4/docker-build.log Collecting sphinx Downloading https://files.pythonhosted.org/packages/89/1e/64c77163706556b647f99d67b42fced9d39ae6b1b86673965a2cd28037b5/Sphinx-2.1.2.tar.gz (6.3MB) Complete output from command python setup.py egg_info: ERROR: Sphinx requires at least Python 3.5 to run. ---------------------------------------- Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-2tylGA/sphinx/ You are using pip version 8.1.1, however version 19.1.1 is available. You should consider upgrading via the 'pip install --upgrade pip' command. ``` The following is the short reproducible step. ``` $ docker build -t spark-rm-test2 --build-arg UID=501 dev/create-release/spark-rm ``` ## How was this patch tested? Manual. ``` $ docker build -t spark-rm-test2 --build-arg UID=501 dev/create-release/spark-rm ``` Closes apache#25226 from dongjoon-hyun/SPARK-28468. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
What changes were proposed in this pull request?
Spark 2.4.x should be a LTS version and we should use the release script in
branch-2.4
to avoid the previous mistakes. Currently,do-release-docker.sh
fails atsphinx
installation toPython 2.7
atbranch-2.4
only. This PR aims to upgradepip
to handle this.The following is the short reproducible step.
How was this patch tested?
Manual.