Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[ZEPPELIN-4978] Build separate Zeppelin server and interpreter images
### What is this PR for? This PR provides new Dockerfiles to build a Zeppelin server and interpreter image. This PR tries to solve #3769 (comment) I agree with alexott that multiple interpreter images are difficult to handle. Currently neither the docker launcher nor the K8s launcher support switching to other interpreter images. Some motivations: - conda is used to install all needed python and R packages - conda is able to manage a whole environment and both languages - files are used for installing python and R dependency - it's easy to customize the content of files. No need to change the Dockerfile itself. - a file has more readability - reduce dependency calculation - a zeppelin-distribution image is needed - you need to build Zeppelin only once, if you want to build new images in parallel on a build cluster ### What type of PR is it? Improvement ### Todos * [ ] - Task ### What is the Jira issue? * https://jira.apache.org/jira/browse/ZEPPELIN-4978 ### How should this be tested? * Build the images ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Philipp Dallig <philipp.dallig@gmail.com> Closes #3859 from Reamer/three_images and squashes the following commits: 300fc0c [Philipp Dallig] Give the Zeppelin interpreter more rights for folders 175566c [Philipp Dallig] Correct Zeppelin home d221181 [Philipp Dallig] Update miniconda to py38_4.8.3 9ffb77c [Philipp Dallig] Disallow the modification of conda packages 336ffd7 [Philipp Dallig] Use only one conda file to resolve dependencies only once 38c3d57 [Philipp Dallig] Using tini from ubuntu package repository b8d3143 [Philipp Dallig] Correct typo 8fa6f80 [Philipp Dallig] Update to Ubuntu 20.04 e095587 [Philipp Dallig] Fix download of additional interpreter packages f0585b9 [Philipp Dallig] Allow to edit conda packages 1a1211e [Philipp Dallig] Build only three images
- Loading branch information
Showing
11 changed files
with
321 additions
and
7 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,32 @@ | ||
# | ||
# Licensed to the Apache Software Foundation (ASF) under one or more | ||
# contributor license agreements. See the NOTICE file distributed with | ||
# this work for additional information regarding copyright ownership. | ||
# The ASF licenses this file to You under the Apache License, Version 2.0 | ||
# (the "License"); you may not use this file except in compliance with | ||
# the License. You may obtain a copy of the License at | ||
# | ||
# http://www.apache.org/licenses/LICENSE-2.0 | ||
# | ||
# Unless required by applicable law or agreed to in writing, software | ||
# distributed under the License is distributed on an "AS IS" BASIS, | ||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
# See the License for the specific language governing permissions and | ||
# limitations under the License. | ||
# | ||
FROM maven:3.5-jdk-8 as builder | ||
ADD . /workspace/zeppelin | ||
WORKDIR /workspace/zeppelin | ||
# Allow npm and bower to run with root privileges | ||
RUN echo "unsafe-perm=true" > ~/.npmrc && \ | ||
echo '{ "allow_root": true }' > ~/.bowerrc && \ | ||
mvn -B package -DskipTests -Pbuild-distr -Pspark-3.0 -Pinclude-hadoop -Phadoop3 -Pspark-scala-2.12 -Pweb-angular && \ | ||
# Example with doesn't compile all interpreters | ||
# mvn -B package -DskipTests -Pbuild-distr -Pspark-3.0 -Pinclude-hadoop -Phadoop3 -Pspark-scala-2.12 -Pweb-angular -pl '!groovy,!submarine,!livy,!hbase,!pig,!file,!flink,!ignite,!kylin,!lens' && \ | ||
mv /workspace/zeppelin/zeppelin-distribution/target/zeppelin-*/zeppelin-* /opt/zeppelin/ && \ | ||
# Removing stuff saves time, because docker creates a temporary layer | ||
rm -rf ~/.m2 && \ | ||
rm -rf /workspace/zeppelin/* | ||
|
||
FROM ubuntu:18.04 | ||
COPY --from=builder /opt/zeppelin /opt/zeppelin |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,83 @@ | ||
# Licensed to the Apache Software Foundation (ASF) under one or more | ||
# contributor license agreements. See the NOTICE file distributed with | ||
# this work for additional information regarding copyright ownership. | ||
# The ASF licenses this file to You under the Apache License, Version 2.0 | ||
# (the "License"); you may not use this file except in compliance with | ||
# the License. You may obtain a copy of the License at | ||
# | ||
# http://www.apache.org/licenses/LICENSE-2.0 | ||
# | ||
# Unless required by applicable law or agreed to in writing, software | ||
# distributed under the License is distributed on an "AS IS" BASIS, | ||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
# See the License for the specific language governing permissions and | ||
# limitations under the License. | ||
|
||
FROM zeppelin-distribution:latest AS zeppelin-distribution | ||
|
||
FROM ubuntu:20.04 | ||
|
||
LABEL maintainer="Apache Software Foundation <dev@zeppelin.apache.org>" | ||
|
||
ARG version="0.9.0-SNAPSHOT" | ||
|
||
ENV VERSION="${version}" \ | ||
Z_HOME="/opt/zeppelin" | ||
|
||
RUN set -ex && \ | ||
apt-get -y update && \ | ||
DEBIAN_FRONTEND=noninteractive apt-get install -y openjdk-8-jre-headless wget && \ | ||
# Cleanup | ||
rm -rf /var/lib/apt/lists/* && \ | ||
apt-get autoclean && \ | ||
apt-get clean | ||
|
||
COPY --from=zeppelin-distribution /opt/zeppelin/bin ${Z_HOME}/bin | ||
COPY log4j.properties ${Z_HOME}/conf/ | ||
COPY log4j_yarn_cluster.properties ${Z_HOME}/conf/ | ||
# Decide: | ||
## 1) Build a huge image with all interpreters (default) | ||
COPY --from=zeppelin-distribution /opt/zeppelin/interpreter ${Z_HOME}/interpreter | ||
## 2) Build an image with only some interpreters | ||
#### Copy interpreter-shaded JAR, needed for all interpreters | ||
### COPY --from=zeppelin-distribution /opt/zeppelin/interpreter/zeppelin-interpreter-shaded-${VERSION}.jar ${Z_HOME}/interpreter/zeppelin-interpreter-shaded-${VERSION}.jar | ||
#### Copy specific interpreters, replace "${interpreter_name}" with your interpreter. Of course you can repeat the line with defferent interpreter | ||
### COPY --from=zeppelin-distribution /opt/zeppelin/interpreter/${interpreter_name} ${Z_HOME}/interpreter/${interpreter_name} | ||
|
||
|
||
# Decide: Install conda to manage python and R packages. Maybe adjust the packages in pip_packages.txt or conda_packages.txt | ||
ARG miniconda_version="py38_4.8.3" | ||
ARG miniconda_sha256="879457af6a0bf5b34b48c12de31d4df0ee2f06a8e68768e5758c3293b2daf688" | ||
# Install python and R packages via conda | ||
COPY conda_packages.txt /conda_packages.txt | ||
# Some python packages are not available via conda, so we are using pip | ||
COPY pip_packages.txt /pip_packages.txt | ||
RUN set -ex && \ | ||
wget -nv https://repo.anaconda.com/miniconda/Miniconda3-${miniconda_version}-Linux-x86_64.sh -O miniconda.sh && \ | ||
echo "${miniconda_sha256} miniconda.sh" > anaconda.sha256 && \ | ||
sha256sum --strict -c anaconda.sha256 && \ | ||
bash miniconda.sh -b -p /opt/conda && \ | ||
export PATH=/opt/conda/bin:$PATH && \ | ||
conda config --set always_yes yes --set changeps1 no && \ | ||
conda info -a && \ | ||
conda config --add channels conda-forge && \ | ||
conda install -y --quiet --file /conda_packages.txt && \ | ||
pip install -q -r /pip_packages.txt && \ | ||
# Cleanup | ||
rm -v miniconda.sh anaconda.sha256 && \ | ||
# Cleanup based on https://github.com/ContinuumIO/docker-images/commit/cac3352bf21a26fa0b97925b578fb24a0fe8c383 | ||
find /opt/conda/ -follow -type f -name '*.a' -delete && \ | ||
find /opt/conda/ -follow -type f -name '*.js.map' -delete && \ | ||
conda clean -ay | ||
# Allow to modify conda packages. This allows malicious code to be injected into other interpreter sessions, therefore it is disabled by default | ||
# chmod -R ug+rwX /opt/conda | ||
ENV PATH /opt/conda/bin:$PATH | ||
|
||
RUN mkdir -p "${Z_HOME}/logs" "${Z_HOME}/run" "${Z_HOME}/local-repo" && \ | ||
# Allow process to edit /etc/passwd, to create a user entry for zeppelin | ||
chgrp root /etc/passwd && chmod ug+rw /etc/passwd && \ | ||
# Give access to some specific folders | ||
chmod -R 775 "${Z_HOME}/logs" "${Z_HOME}/run" "${Z_HOME}/local-repo" | ||
|
||
USER 1000 | ||
WORKDIR ${Z_HOME} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,23 @@ | ||
# python packages | ||
pycodestyle | ||
numpy | ||
pandas | ||
scipy | ||
grpcio | ||
hvplot | ||
protobuf | ||
pandasql | ||
ipython | ||
matplotlib | ||
ipykernel | ||
jupyter_client | ||
bokeh | ||
apache-beam | ||
|
||
# R packages | ||
r-evaluate | ||
r-base64enc | ||
r-knitr | ||
r-ggplot2 | ||
r-shiny | ||
r-googlevis |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,22 @@ | ||
# | ||
# Licensed to the Apache Software Foundation (ASF) under one or more | ||
# contributor license agreements. See the NOTICE file distributed with | ||
# this work for additional information regarding copyright ownership. | ||
# The ASF licenses this file to You under the Apache License, Version 2.0 | ||
# (the "License"); you may not use this file except in compliance with | ||
# the License. You may obtain a copy of the License at | ||
# | ||
# http://www.apache.org/licenses/LICENSE-2.0 | ||
# | ||
# Unless required by applicable law or agreed to in writing, software | ||
# distributed under the License is distributed on an "AS IS" BASIS, | ||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
# See the License for the specific language governing permissions and | ||
# limitations under the License. | ||
# | ||
|
||
log4j.rootLogger = INFO, stdout | ||
|
||
log4j.appender.stdout = org.apache.log4j.ConsoleAppender | ||
log4j.appender.stdout.layout = org.apache.log4j.PatternLayout | ||
log4j.appender.stdout.layout.ConversionPattern=%5p [%d] ({%t} %F[%M]:%L) - %m%n |
23 changes: 23 additions & 0 deletions
23
scripts/docker/zeppelin-interpreter/log4j_yarn_cluster.properties
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,23 @@ | ||
# | ||
# Licensed to the Apache Software Foundation (ASF) under one or more | ||
# contributor license agreements. See the NOTICE file distributed with | ||
# this work for additional information regarding copyright ownership. | ||
# The ASF licenses this file to You under the Apache License, Version 2.0 | ||
# (the "License"); you may not use this file except in compliance with | ||
# the License. You may obtain a copy of the License at | ||
# | ||
# http://www.apache.org/licenses/LICENSE-2.0 | ||
# | ||
# Unless required by applicable law or agreed to in writing, software | ||
# distributed under the License is distributed on an "AS IS" BASIS, | ||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
# See the License for the specific language governing permissions and | ||
# limitations under the License. | ||
# | ||
|
||
log4j.rootLogger = INFO, stdout | ||
|
||
log4j.appender.stdout = org.apache.log4j.ConsoleAppender | ||
log4j.appender.stdout.layout = org.apache.log4j.PatternLayout | ||
log4j.appender.stdout.layout.ConversionPattern=%5p [%d] ({%t} %F[%M]:%L) - %m%n | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
bkzep==0.6.1 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,83 @@ | ||
# Licensed to the Apache Software Foundation (ASF) under one or more | ||
# contributor license agreements. See the NOTICE file distributed with | ||
# this work for additional information regarding copyright ownership. | ||
# The ASF licenses this file to You under the Apache License, Version 2.0 | ||
# (the "License"); you may not use this file except in compliance with | ||
# the License. You may obtain a copy of the License at | ||
# | ||
# http://www.apache.org/licenses/LICENSE-2.0 | ||
# | ||
# Unless required by applicable law or agreed to in writing, software | ||
# distributed under the License is distributed on an "AS IS" BASIS, | ||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
# See the License for the specific language governing permissions and | ||
# limitations under the License. | ||
|
||
FROM zeppelin-distribution:latest AS zeppelin-distribution | ||
|
||
# Prepare all interpreter settings for Zeppelin server | ||
# This steps are not needed, if you you add only specific interpreters settings to your image | ||
FROM alpine:3.11 AS interpreter-settings | ||
COPY --from=zeppelin-distribution /opt/zeppelin/interpreter /tmp/interpreter | ||
RUN mkdir -p /opt/zeppelin/interpreter && \ | ||
cd /tmp/interpreter && \ | ||
find . -name 'interpreter-setting.json' -exec cp --parents \{\} /opt/zeppelin/interpreter \; | ||
|
||
FROM ubuntu:20.04 | ||
LABEL maintainer="Apache Software Foundation <dev@zeppelin.apache.org>" | ||
|
||
RUN set -ex && \ | ||
apt-get -y update && \ | ||
# Install language and other base packages | ||
DEBIAN_FRONTEND=noninteractive apt-get install -y language-pack-en openjdk-8-jre-headless tini wget && \ | ||
# Cleanup | ||
rm -rf /var/lib/apt/lists/* && \ | ||
apt-get autoclean && \ | ||
apt-get clean | ||
|
||
ARG version="0.9.0-SNAPSHOT" | ||
|
||
ENV LANG=en_US.UTF-8 \ | ||
LC_ALL=en_US.UTF-8 \ | ||
JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 \ | ||
VERSION="${version}" \ | ||
Z_HOME="/opt/zeppelin" \ | ||
ZEPPELIN_ADDR="0.0.0.0" \ | ||
ZEPPELIN_WAR_TEMPDIR="/tmp/webapps" | ||
|
||
# Copy Zeppelin related files | ||
COPY --from=zeppelin-distribution /opt/zeppelin/zeppelin-web-${VERSION}.war ${Z_HOME}/ | ||
COPY --from=zeppelin-distribution /opt/zeppelin/zeppelin-web-angular-${VERSION}.war ${Z_HOME}/ | ||
COPY --from=zeppelin-distribution /opt/zeppelin/conf ${Z_HOME}/conf | ||
COPY --from=zeppelin-distribution /opt/zeppelin/bin ${Z_HOME}/bin | ||
COPY --from=zeppelin-distribution /opt/zeppelin/lib ${Z_HOME}/lib | ||
COPY --from=zeppelin-distribution /opt/zeppelin/plugins ${Z_HOME}/plugins | ||
COPY --from=zeppelin-distribution /opt/zeppelin/interpreter/zeppelin-interpreter-shaded-${VERSION}.jar ${Z_HOME}/interpreter/zeppelin-interpreter-shaded-${VERSION}.jar | ||
# copy example notebooks | ||
COPY --from=zeppelin-distribution /opt/zeppelin/notebook ${Z_HOME}/notebook | ||
# copy k8s files | ||
COPY --from=zeppelin-distribution /opt/zeppelin/k8s ${Z_HOME}/k8s | ||
|
||
# Decide | ||
## 1) Copy and activate all interpreters (default) | ||
COPY --from=interpreter-settings /opt/zeppelin/interpreter ${Z_HOME}/interpreter | ||
## 2) Copy and activate only a specific set of interpreter | ||
# COPY --from=zeppelin-distribution /opt/zeppelin/interpreter/spark/interpreter-setting.json ${Z_HOME}/interpreter/spark/interpreter-setting.json | ||
# COPY --from=zeppelin-distribution /opt/zeppelin/interpreter/jdbc/interpreter-setting.json ${Z_HOME}/interpreter/jdbc/interpreter-setting.json | ||
# COPY --from=zeppelin-distribution /opt/zeppelin/interpreter/md/interpreter-setting.json ${Z_HOME}/interpreter/md/interpreter-setting.json | ||
|
||
COPY log4j.properties ${Z_HOME}/conf/ | ||
|
||
RUN mkdir -p "${Z_HOME}/logs" "${Z_HOME}/run" "${Z_HOME}/notebook" "${Z_HOME}/local-repo" && \ | ||
# Allow process to edit /etc/passwd, to create a user entry for zeppelin | ||
chgrp root /etc/passwd && chmod ug+rw /etc/passwd && \ | ||
# Give access to some specific folders | ||
chmod -R 775 "${Z_HOME}/logs" "${Z_HOME}/run" "${Z_HOME}/conf" "${Z_HOME}/notebook" "${Z_HOME}/local-repo" | ||
|
||
USER 1000 | ||
|
||
EXPOSE 8080 | ||
|
||
ENTRYPOINT [ "/usr/bin/tini", "--" ] | ||
WORKDIR ${Z_HOME} | ||
CMD ["bin/zeppelin.sh"] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,22 @@ | ||
# | ||
# Licensed to the Apache Software Foundation (ASF) under one or more | ||
# contributor license agreements. See the NOTICE file distributed with | ||
# this work for additional information regarding copyright ownership. | ||
# The ASF licenses this file to You under the Apache License, Version 2.0 | ||
# (the "License"); you may not use this file except in compliance with | ||
# the License. You may obtain a copy of the License at | ||
# | ||
# http://www.apache.org/licenses/LICENSE-2.0 | ||
# | ||
# Unless required by applicable law or agreed to in writing, software | ||
# distributed under the License is distributed on an "AS IS" BASIS, | ||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
# See the License for the specific language governing permissions and | ||
# limitations under the License. | ||
# | ||
|
||
log4j.rootLogger = INFO, stdout | ||
|
||
log4j.appender.stdout = org.apache.log4j.ConsoleAppender | ||
log4j.appender.stdout.layout = org.apache.log4j.PatternLayout | ||
log4j.appender.stdout.layout.ConversionPattern=%5p [%d] ({%t} %F[%M]:%L) - %m%n |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters