ORC-694: Update docker files for adding Java11 support by yuokada · Pull Request #655 · apache/orc

yuokada · 2021-03-15T16:27:20Z

What changes were proposed in this pull request?

Add jdk argument in Dockerfile to support JDK11

Why are the changes needed?

Since github actions for the repository use java 11, we should provide the images using JDK 11

How was this patch tested?

Confirmed that Dockerfiles are built successfully with reinit.sh
bash -xe ./run-one.sh yuokada ORC-694 ubuntu20

dongjoon-hyun

Thank you for working on this, @yuokada .

BTW, do we need to change MAINTAINER in this PR?

yuokada · 2021-03-15T23:50:35Z

@dongjoon-hyun
MAINTAINER command is deprecated since docker 1.13 or later. So I replaced it with the LABEL command.
https://docs.docker.com/engine/reference/builder/#maintainer-deprecated

But, we don't need to change it in this PR.

Should we create another ticket to replace MAINTAINER command?

dongjoon-hyun · 2021-03-16T00:26:55Z

Shall we do that in a separate PR because it's irrelevant to Java?

yuokada · 2021-03-16T00:35:57Z

Created: #656

yuokada · 2021-03-16T03:05:30Z


    cd docker/$os
-    docker build -t orc-$os .
+    docker build -t "orc-$os-jdk8" --build-arg jdk=8 . OR docker build -t "orc-$os-jdk11" --build-arg jdk=11 .


Is this change easy to understand?

Lets make it more explicit.
For jdk8: docker build -t "orc-$os-jdk8" --build-arg jdk=8 .
For jdk11: docker build -t "orc-$os-jdk11" --build-arg jdk=11 .

dongjoon-hyun

Could you minimize the size of patch at CentOS Dockerfile? Technically, if statement should be related to java-xx-openjdk. For the other part git/maven/make/..., we don't need to duplicate.

dongjoon-hyun · 2021-03-17T18:55:55Z

-  ( cd $os && docker build -t "orc-$os" . ) > logs/$os-build.log 2>&1 || exit 1
+for jdk in 8 11; do
+    for os in `cat os-list.txt`; do
+        echo "Building $os"


Could you show $jdk together?

dongjoon-hyun

Thank you for updating, @yuokada . The PR looks good mostly. I left one minor comment about log message.

pgaref · 2021-03-17T19:18:44Z


    cd docker/$os
-    docker build -t orc-$os .
+    docker build -t "orc-$os-jdk8" --build-arg jdk=8 . OR docker build -t "orc-$os-jdk11" --build-arg jdk=11 .


Lets make it more explicit.
For jdk8: docker build -t "orc-$os-jdk8" --build-arg jdk=8 .
For jdk11: docker build -t "orc-$os-jdk11" --build-arg jdk=11 .

pgaref · 2021-03-17T19:22:11Z

@@ -21,7 +21,7 @@ The scripts are:
 A base image for each OS is built using:


Lets add above that:
run-all.sh, run-one.sh and reinit.sh tests both on jdk8 and 11 across OSes

pgaref · 2021-03-17T19:40:25Z

Thanks @yuokada and @dongjoon-hyun for the work on this! Left some minor comments.
Seems to work well locally!

dongjoon-hyun

I built and ran this PR.

$ for i in $(docker images | grep orc | awk '{print $1}'); do echo $i; docker run -it --rm $i java -version; done
orc-ubuntu20-clang-jdk11
openjdk version "11.0.10" 2021-01-19
OpenJDK Runtime Environment (build 11.0.10+9-Ubuntu-0ubuntu1.20.04)
OpenJDK 64-Bit Server VM (build 11.0.10+9-Ubuntu-0ubuntu1.20.04, mixed mode, sharing)
orc-ubuntu20-jdk11
openjdk version "11.0.10" 2021-01-19
OpenJDK Runtime Environment (build 11.0.10+9-Ubuntu-0ubuntu1.20.04)
OpenJDK 64-Bit Server VM (build 11.0.10+9-Ubuntu-0ubuntu1.20.04, mixed mode, sharing)
orc-ubuntu18-jdk11
openjdk version "11.0.10" 2021-01-19
OpenJDK Runtime Environment (build 11.0.10+9-Ubuntu-0ubuntu1.18.04)
OpenJDK 64-Bit Server VM (build 11.0.10+9-Ubuntu-0ubuntu1.18.04, mixed mode, sharing)
orc-centos8-jdk11
openjdk version "11.0.9.1" 2020-11-04 LTS
OpenJDK Runtime Environment 18.9 (build 11.0.9.1+1-LTS)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.9.1+1-LTS, mixed mode, sharing)
orc-centos7-jdk11
openjdk version "11.0.10" 2021-01-19 LTS
OpenJDK Runtime Environment 18.9 (build 11.0.10+9-LTS)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.10+9-LTS, mixed mode, sharing)
orc-ubuntu20-clang-jdk8
openjdk version "1.8.0_282"
OpenJDK Runtime Environment (build 1.8.0_282-8u282-b08-0ubuntu1~20.04-b08)
OpenJDK 64-Bit Server VM (build 25.282-b08, mixed mode)
orc-ubuntu20-jdk8
openjdk version "1.8.0_282"
OpenJDK Runtime Environment (build 1.8.0_282-8u282-b08-0ubuntu1~20.04-b08)
OpenJDK 64-Bit Server VM (build 25.282-b08, mixed mode)
orc-ubuntu18-jdk8
openjdk version "1.8.0_282"
OpenJDK Runtime Environment (build 1.8.0_282-8u282-b08-0ubuntu1~18.04-b08)
OpenJDK 64-Bit Server VM (build 25.282-b08, mixed mode)
orc-ubuntu16-jdk11
openjdk version "1.8.0_282"
OpenJDK Runtime Environment (build 1.8.0_282-8u282-b08-0ubuntu1~16.04-b08)
OpenJDK 64-Bit Server VM (build 25.282-b08, mixed mode)
orc-ubuntu16-jdk8
openjdk version "1.8.0_282"
OpenJDK Runtime Environment (build 1.8.0_282-8u282-b08-0ubuntu1~16.04-b08)
OpenJDK 64-Bit Server VM (build 25.282-b08, mixed mode)
orc-debian10-jdk11
openjdk version "11.0.9.1" 2020-11-04
OpenJDK Runtime Environment (build 11.0.9.1+1-post-Debian-1deb10u2)
OpenJDK 64-Bit Server VM (build 11.0.9.1+1-post-Debian-1deb10u2, mixed mode, sharing)
orc-debian10-jdk8
openjdk version "11.0.9.1" 2020-11-04
OpenJDK Runtime Environment (build 11.0.9.1+1-post-Debian-1deb10u2)
OpenJDK 64-Bit Server VM (build 11.0.9.1+1-post-Debian-1deb10u2, mixed mode, sharing)
orc-debian9-jdk11
openjdk version "1.8.0_275"
OpenJDK Runtime Environment (build 1.8.0_275-8u275-b01-1~deb9u1-b01)
OpenJDK 64-Bit Server VM (build 25.275-b01, mixed mode)
orc-debian9-jdk8
openjdk version "1.8.0_275"
OpenJDK Runtime Environment (build 1.8.0_275-8u275-b01-1~deb9u1-b01)
OpenJDK 64-Bit Server VM (build 25.275-b01, mixed mode)
orc-centos8-jdk8
openjdk version "1.8.0_275"
OpenJDK Runtime Environment (build 1.8.0_275-b01)
OpenJDK 64-Bit Server VM (build 25.275-b01, mixed mode)
orc-centos7-jdk8
openjdk version "1.8.0_282"
OpenJDK Runtime Environment (build 1.8.0_282-b08)
OpenJDK 64-Bit Server VM (build 25.282-b08, mixed mode)

In short, the following three look suspicious to me. Could you double-check it, @yuokada ?

orc-ubuntu16-jdk11
openjdk version "1.8.0_282"
OpenJDK Runtime Environment (build 1.8.0_282-8u282-b08-0ubuntu1~16.04-b08)
OpenJDK 64-Bit Server VM (build 25.282-b08, mixed mode)
...
orc-debian10-jdk8
openjdk version "11.0.9.1" 2020-11-04
OpenJDK Runtime Environment (build 11.0.9.1+1-post-Debian-1deb10u2)
OpenJDK 64-Bit Server VM (build 11.0.9.1+1-post-Debian-1deb10u2, mixed mode, sharing)
...
orc-debian9-jdk11
openjdk version "1.8.0_275"
OpenJDK Runtime Environment (build 1.8.0_275-8u275-b01-1~deb9u1-b01)
OpenJDK 64-Bit Server VM (build 25.275-b01, mixed mode)

yuokada · 2021-03-17T23:54:53Z

@dongjoon-hyun
Just questions.
For debian 10, the OS image already use ~~jdk10~~ jdk11. Should we support jdk8 on it?

For debian 9 and ubuntu16, the openjdk-11 deb package isn't provided for both OSes.
Do we need to support JDK 11 in both?

Or should we skip building an image on those OSes?

dongjoon-hyun · 2021-03-18T01:29:31Z

If there is no JDK11 package, we had better document it and skip JDK11 for old OSes inside the above script like run-all.sh.

dongjoon-hyun · 2021-03-18T22:02:36Z

Thank you, @yuokada and @omalley .

yuokada force-pushed the ORC-694 branch from 16e884d to 4279af8 Compare March 15, 2021 16:37

dongjoon-hyun reviewed Mar 15, 2021

View reviewed changes

yuokada force-pushed the ORC-694 branch from 4279af8 to cd6549a Compare March 16, 2021 00:08

yuokada added 3 commits March 16, 2021 12:01

Improve Dockerfile per os to support jdk argument

89a2cb8

Update shell scripts to support docker images for JDK 11

e9d4f5b

Update README.md for docker

a85bd62

yuokada force-pushed the ORC-694 branch from 33794b8 to a85bd62 Compare March 16, 2021 03:01

yuokada commented Mar 16, 2021

View reviewed changes

fixup! Improve Dockerfile per os to support jdk argument

3475e65

yuokada marked this pull request as ready for review March 16, 2021 07:21

dongjoon-hyun requested changes Mar 16, 2021

View reviewed changes

dongjoon-hyun reviewed Mar 16, 2021

View reviewed changes

Comment thread docker/ubuntu18/Dockerfile

dongjoon-hyun reviewed Mar 16, 2021

View reviewed changes

Comment thread docker/ubuntu20-clang/Dockerfile

dongjoon-hyun reviewed Mar 16, 2021

View reviewed changes

Comment thread docker/ubuntu20/Dockerfile

Improve Dockerfile for centos7/centos8

28dbb4d

dongjoon-hyun reviewed Mar 17, 2021

View reviewed changes

pgaref reviewed Mar 17, 2021

View reviewed changes

dongjoon-hyun requested changes Mar 17, 2021

View reviewed changes

yuokada added 2 commits March 18, 2021 08:27

Improve message in run-all.sh

e08d96e

Update README

f7329c0

yuokada added 2 commits March 18, 2021 11:51

Skip the building operations with specific combinations

27a8570

Update README.md

415a3bd

yuokada force-pushed the ORC-694 branch from 1fddd15 to 415a3bd Compare March 18, 2021 02:51

omalley closed this in 75726b1 Mar 18, 2021

		@@ -21,7 +21,7 @@ The scripts are:
		A base image for each OS is built using:

Uh oh!

Conversation

yuokada commented Mar 15, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

How was this patch tested?

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

yuokada commented Mar 15, 2021

Uh oh!

dongjoon-hyun commented Mar 16, 2021

Uh oh!

yuokada commented Mar 16, 2021

Uh oh!

yuokada Mar 16, 2021

Choose a reason for hiding this comment

Uh oh!

pgaref Mar 17, 2021

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dongjoon-hyun Mar 17, 2021

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

pgaref Mar 17, 2021

Choose a reason for hiding this comment

Uh oh!

pgaref Mar 17, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pgaref commented Mar 17, 2021

Uh oh!

dongjoon-hyun left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yuokada commented Mar 17, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongjoon-hyun commented Mar 18, 2021

Uh oh!

dongjoon-hyun commented Mar 18, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

yuokada commented Mar 15, 2021 •

edited

Loading

dongjoon-hyun left a comment •

edited

Loading

pgaref Mar 17, 2021 •

edited

Loading

dongjoon-hyun left a comment •

edited

Loading

yuokada commented Mar 17, 2021 •

edited

Loading