Skip to content

Commit

Permalink
[SPARK-33927][BUILD] Fix Dockerfile for Spark release to work
Browse files Browse the repository at this point in the history
This PR proposes to fix the `Dockerfile` for Spark release.

- Port apache@b135db3 to `Dockerfile`
- Upgrade Ubuntu 18.04 -> 20.04 (because of porting b135db3)
- Remove Python 2 (because of Ubuntu upgrade)
- Use built-in Python 3.8.5 (because of Ubuntu upgrade)
- Node.js 11 -> 12 (because of Ubuntu upgrade)
- Ruby 2.5 -> 2.7 (because of Ubuntu upgrade)
- Python dependencies and Jekyll + plugins upgrade to the latest as it's used in GitHub Actions build (unrelated to the issue itself)

To make a Spark release :-).

No, dev-only.

Manually tested via:

```bash
cd dev/create-release/spark-rm
docker build -t spark-rm --build-arg UID=$UID .
```

```
...
Successfully built 516d7943634f
Successfully tagged spark-rm:latest
```

Closes apache#30971 from HyukjinKwon/SPARK-33927.

Lead-authored-by: Hyukjin Kwon <gurwls223@apache.org>
Co-authored-by: HyukjinKwon <gurwls223@apache.org>
Signed-off-by: HyukjinKwon <gurwls223@apache.org>
  • Loading branch information
HyukjinKwon authored and okumin committed Jan 2, 2021
1 parent 2b147c4 commit c80f23f
Showing 1 changed file with 14 additions and 14 deletions.
28 changes: 14 additions & 14 deletions dev/create-release/spark-rm/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -15,16 +15,20 @@
# limitations under the License.
#

# Image for building Spark releases. Based on Ubuntu 18.04.
# Image for building Spark releases. Based on Ubuntu 20.04.
#
# Includes:
# * Java 8
# * Ivy
# * Python (2.7.15/3.6.7)
# * R-base/R-base-dev (4.0.2)
# * Ruby 2.3 build utilities
# * Python (3.8.5)
# * R-base/R-base-dev (4.0.3)
# * Ruby (2.7.0)
#
# You can test it as below:
# cd dev/create-release/spark-rm
# docker build -t spark-rm --build-arg UID=$UID .

FROM ubuntu:18.04
FROM ubuntu:20.04

# For apt to be noninteractive
ENV DEBIAN_FRONTEND noninteractive
Expand All @@ -43,41 +47,37 @@ ARG GEM_PKGS="jekyll:4.0.0 jekyll-redirect-from:0.16.0 rouge:3.15.0"
# This is all in a single "RUN" command so that if anything changes, "apt update" is run to fetch
# the most current package versions (instead of potentially using old versions cached by docker).
RUN apt-get clean && apt-get update && $APT_INSTALL gnupg ca-certificates && \
echo 'deb https://cloud.r-project.org/bin/linux/ubuntu bionic-cran40/' >> /etc/apt/sources.list && \
echo 'deb https://cloud.r-project.org/bin/linux/ubuntu focal-cran40/' >> /etc/apt/sources.list && \
gpg --keyserver keyserver.ubuntu.com --recv-key E298A3A825C0D65DFD57CBB651716619E084DAB9 && \
gpg -a --export E084DAB9 | apt-key add - && \
apt-get clean && \
rm -rf /var/lib/apt/lists/* && \
apt-get clean && \
apt-get update && \
$APT_INSTALL software-properties-common && \
apt-add-repository -y ppa:brightbox/ruby-ng && \
apt-get update && \
# Install openjdk 8.
$APT_INSTALL openjdk-8-jdk && \
update-alternatives --set java /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java && \
# Install build / source control tools
$APT_INSTALL curl wget git maven ivy subversion make gcc lsof libffi-dev \
pandoc pandoc-citeproc libssl-dev libcurl4-openssl-dev libxml2-dev && \
curl -sL https://deb.nodesource.com/setup_11.x | bash && \
curl -sL https://deb.nodesource.com/setup_12.x | bash && \
$APT_INSTALL nodejs && \
# Install needed python packages. Use pip for installing packages (for consistency).
$APT_INSTALL libpython3-dev python3-pip python3-setuptools && \
$APT_INSTALL python3-pip python3-setuptools && \
# qpdf is required for CRAN checks to pass.
$APT_INSTALL qpdf jq && \
# Change default python version to python3.
update-alternatives --install /usr/bin/python python /usr/bin/python2.7 1 && \
update-alternatives --install /usr/bin/python python /usr/bin/python3.6 2 && \
update-alternatives --set python /usr/bin/python3.6 && \
pip3 install $PIP_PKGS && \
# Install R packages and dependencies used when building.
# R depends on pandoc*, libssl (which are installed above).
$APT_INSTALL r-base r-base-dev && \
$APT_INSTALL libcurl4-openssl-dev libgit2-dev libssl-dev libxml2-dev && \
$APT_INSTALL texlive-latex-base texlive texlive-fonts-extra texinfo qpdf && \
Rscript -e "install.packages(c('curl', 'xml2', 'httr', 'devtools', 'testthat', 'knitr', 'rmarkdown', 'roxygen2', 'e1071', 'survival'), repos='https://cloud.r-project.org/')" && \
Rscript -e "devtools::install_github('jimhester/lintr')" && \
# Install tools needed to build the documentation.
$APT_INSTALL ruby2.5 ruby2.5-dev && \
$APT_INSTALL ruby2.7 ruby2.7-dev && \
gem install --no-document $GEM_PKGS

WORKDIR /opt/spark-rm/output
Expand Down

0 comments on commit c80f23f

Please sign in to comment.