BIGTOP-3852: Java 11 to bigtop-toolchain to use it in docker container during packaging #1043

yoda-mon · 2022-10-19T06:16:31Z

Description of PR

This PR aims bigtop users can use Java 11 when they package some projects on a docker container.

The behavior of bigtop is not changed.
If user set an option, bigtop pass a flag to the docker container by environment variable. If build target project can receive the flag, bigtop switch JAVA_HOME to change the Java version.

How was this patch tested?

Installation of JDK 11

I checked below on aarch 64 machine

# build bigtop/slave
cd docker/bigtop-slaves/
OS_LIST=( ubuntu-18.04 ubuntu-20.04 ubuntu-22.04 debian-10 debian-11 centos-7 rockylinux-8 fedora-35 fedora-36 )
for OS in ${OS_LIST[@]} ; do
  ./build.sh trunk-${OS}
done

# check the installation
for OS in ${OS_LIST[@]} ; do
  echo $OS; 
  docker run -i --rm bigtop/slaves:trunk-$OS-aarch64 ls /usr/lib/jvm;
  echo ""
done

The default JAVA_HOME is set to 8 by /etc/profile.d/bigtop.sh so installation does not affect on it.

Build Kafka 2.8.1 by Java 11

I checked below on aarch 64 machine.

./gradlew zookeeper-pkg-ind kafka-pkg-ind -Ppreferred-java-version=11 -POS=rockylinux-8

I checked Java version by inserting mvn --version or gradle --version to do-component-build.
Zookeeper was build by 8 and Kafka was by 11.

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'BIGTOP-3638. Your PR title ...')?
Make sure that newly added files do not have any licensing issues. When in doubt refer to https://www.apache.org/licenses/

- both 8 and 11 are installed 8 is used as default - add an option to pass the preferred java version to the build container - add an option to swich jdk to kafka's do-component-build

yoda-mon · 2022-10-19T06:26:40Z

I chose the way that has small impact, I think there are other ways to tell the selection of JDK to the container (like -P{project}-java-version=... and swich JAVA_HOME by groovy script), I would like to ask other's opinion.

iwasakims

LGTM. It worked on Ubuntu 18.04 aarch64 and CentOS 7 aarch64.

iwasakims · 2022-10-19T08:29:52Z

I chose the way that has small impact, I think there are other ways to tell the selection of JDK to the container (like -P{project}-java-version=... and swich JAVA_HOME by groovy script), I would like to ask other's opinion.

We can not yet use Java 11 for building all products. Tweaking Java version in do-component-build for specific product should be fine now.

bigtop_toolchain/manifests/installer.pp

guyuqi · 2022-10-19T09:15:20Z

@yoda-mon
Thanks for working on it.
As far as I know, JDK 11 hasn't been totally supported in Apache world (some components don't support JDK11).
IMHO, it is better to retain the JDK8 and make JDK8 the default jvm/jdk in Bigtop, and JDK11 could be installed the other separate path as the optional tool for users. :-)

yoda-mon · 2022-10-20T03:22:57Z

@iwasakims

Tweaking Java version in do-component-build for specific product should be fine now.

Yes, I think so too. This way does not harm other projects.

@guyuqi

it is better to retain the JDK8 and make JDK8 the default jvm/jdk in Bigtop, and JDK11 could be installed the other separate path as the optional tool for users.

I agree. The main JDK on Bigtop should be 8 because some project don't have compatibly for >8.
However it is tough thing that several projects seem deprecating to use Java 8 on current versions, such as Kafka or Flink...
To catch up them, we have to look around balanced way.

- Add an order to jdk11 -> jdk8 - Set java 8 as default explicitly using update-java-alternatives or update-alternatives

yoda-mon · 2022-10-24T07:36:11Z

I added several procedures to ensure to use java 8, not java 11.

Add dependency to the class between jdk.pp and jdk11.pp for ordering. packaging systems uses alternatives for java, and later installed version has higher priority to be default (cf https://linux.die.net/man/8/update-alternatives).
Ordering is not enough for some OSs, in this case, Debian/Ubuntu and Fedora 35, because other libraries require java 8 before installing java 11. So I ensure to use java 8 by using update-java-alternatives or update-alternatives.
We can use update-java-alternatives on java that from apt packaging, it is simple. We can only use update-alternatives on java that RHEL based OS's packaging system, we have to set both java and javac (Other commands are hierarchically controlled under these two, you can check by update-alternatives --display java.

Here shows the script for a checking.

OS_LIST=(centos-7 rockylinux-8 fedora-35 fedora-36 debian-10 debian-11 ubuntu-18.04 ubuntu-20.04 ubuntu-22.04)
for OS in "${OS_LIST[@]}"
do
  echo $OS
  docker run --rm -itd --name bigtop-test-${OS} bigtop/slaves:trunk-${OS}-aarch64 /bin/bash
  docker exec -it bigtop-test-${OS} java -version
  docker exec -it bigtop-test-${OS} javac -version
  docker exec -it bigtop-test-${OS} bash -cl 'echo "$JAVA_HOME"'
  docker stop bigtop-test-${OS}
  echo ""
done

bigtop_toolchain/manifests/installer.pp

yoda-mon · 2022-10-27T09:06:59Z

I modified to use uname. In addition, I found dpkg --print-architecture shows arm64 or amd64 which is used as suffix of the java alternatives. So I add it and checked it worked on both CPU architectures.

iwasakims

Thanks for the update @yoda-mon. +1.

timyuer · 2024-04-18T08:57:25Z

@yoda-mon
Sorry to bother you, I found that there is already a BIGTOP_JDK variable in the bigtop.bom file that can switch JDK versions. Why do we need to create a new variable called preferred-java-version here ?

@iwasakims @guyuqi
Can help take a look together?

yoda-mon · 2024-04-18T09:19:09Z

@timyuer
Several years ago some packages do not compatible with Java 11 (For example, Hadoop only supported running on 11 and did not support building with 11) . However, some pakages started to omit suporting 8.
I intended to mitigate these situations with adding another options here(I think these situations have been getting well).

timyuer · 2024-04-18T10:54:23Z

@timyuer Several years ago some packages do not compatible with Java 11 (For example, Hadoop only supported running on 11 and did not support building with 11) . However, some pakages started to omit suporting 8. I intended to mitigate these situations with adding another options here(I think these situations have been getting well).

@yoda-mon
I agree that most components of BigTop have started or completed supporting JDK17, so I think this scenario is unnecessary. Moreover, adding a separate JDK option can easily cause ambiguity. I suggest that we use the BIGTOP_JDK variable to uniformly control the JDK version.
What do you think?

timyuer · 2024-04-18T11:23:34Z

@timyuer Several years ago some packages do not compatible with Java 11 (For example, Hadoop only supported running on 11 and did not support building with 11) . However, some pakages started to omit suporting 8. I intended to mitigate these situations with adding another options here(I think these situations have been getting well).

In fact, using the BIGTOP_JDK variable can also solve the above problems.

yoda-mon · 2024-04-18T13:50:28Z

@timyuer

I agree that most components of BigTop have started or completed supporting JDK17, so I think this scenario is unnecessary.

I think Hadoop would be the blocker, wouldn't it ?
https://issues.apache.org/jira/browse/HADOOP-17177

using the BIGTOP_JDK variable can also solve the above problems.

I think this is partially correct. If you build multi packages with

same JDK at the same time
or
different JDK separatelly
it's enough only to use BIGTOP_JDK
However,
different JDK at the same time
it's not enough.

timyuer · 2024-04-18T14:02:46Z

@timyuer

I agree that most components of BigTop have started or completed supporting JDK17, so I think this scenario is unnecessary.

I think Hadoop would be the blocker, wouldn't it ? https://issues.apache.org/jira/browse/HADOOP-17177

using the BIGTOP_JDK variable can also solve the above problems.

I think this is partially correct. If you build multi packages with

same JDK at the same time
or

different JDK separatelly
it's enough only to use BIGTOP_JDK
However,

different JDK at the same time
it's not enough.

In the third point, using different jdk at the same time can be subdivided into the following two types,

Using two different versions of JDK at the same time, BIGTOP_JDK and preferred-java-version can be satisfied.
Using three or more different versions of JDK at the same time, neither of the above two can be satisfied.

timyuer · 2024-04-18T14:06:26Z

This leads to the following discussion:

Is it necessary to use two different JDK compilers at the same time? Will this complicate deployment?
If supported, should it also support three or more different JDK compilations at the same time?

Add Java 11 to bigtop-toolchain

e892c88

- both 8 and 11 are installed 8 is used as default - add an option to pass the preferred java version to the build container - add an option to swich jdk to kafka's do-component-build

iwasakims approved these changes Oct 19, 2022

View reviewed changes

iwasakims reviewed Oct 19, 2022

View reviewed changes

bigtop_toolchain/manifests/installer.pp Show resolved Hide resolved

iwasakims reviewed Oct 19, 2022

View reviewed changes

bigtop_toolchain/manifests/installer.pp Show resolved Hide resolved

Ensure Java 8 is set as default

e1b488d

- Add an order to jdk11 -> jdk8 - Set java 8 as default explicitly using update-java-alternatives or update-alternatives

iwasakims reviewed Oct 25, 2022

View reviewed changes

bigtop_toolchain/manifests/installer.pp Outdated Show resolved Hide resolved

select java version more appropriate way

1fa5b08

iwasakims approved these changes Oct 27, 2022

View reviewed changes

iwasakims merged commit 2b8203f into apache:master Oct 27, 2022

BIGTOP-3852: Java 11 to bigtop-toolchain to use it in docker container during packaging #1043

BIGTOP-3852: Java 11 to bigtop-toolchain to use it in docker container during packaging #1043

Uh oh!

Conversation

yoda-mon commented Oct 19, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of PR

How was this patch tested?

Installation of JDK 11

Build Kafka 2.8.1 by Java 11

For code changes:

Uh oh!

yoda-mon commented Oct 19, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

iwasakims left a comment

Choose a reason for hiding this comment

Uh oh!

iwasakims commented Oct 19, 2022

Uh oh!

Uh oh!

Uh oh!

guyuqi commented Oct 19, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yoda-mon commented Oct 20, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yoda-mon commented Oct 24, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

yoda-mon commented Oct 27, 2022

Uh oh!

iwasakims left a comment

Choose a reason for hiding this comment

Uh oh!

timyuer commented Apr 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yoda-mon commented Apr 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

timyuer commented Apr 18, 2024

Uh oh!

timyuer commented Apr 18, 2024

Uh oh!

yoda-mon commented Apr 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

timyuer commented Apr 18, 2024

Uh oh!

timyuer commented Apr 18, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

yoda-mon commented Oct 19, 2022 •

edited

Loading

yoda-mon commented Oct 19, 2022 •

edited

Loading

guyuqi commented Oct 19, 2022 •

edited

Loading

yoda-mon commented Oct 20, 2022 •

edited

Loading

yoda-mon commented Oct 24, 2022 •

edited

Loading

timyuer commented Apr 18, 2024 •

edited

Loading

yoda-mon commented Apr 18, 2024 •

edited

Loading

yoda-mon commented Apr 18, 2024 •

edited

Loading