Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Security] Use ubuntu:20.04 base image for Pulsar docker images #11026

Merged
merged 6 commits into from
Aug 2, 2021
Merged

[Security] Use ubuntu:20.04 base image for Pulsar docker images #11026

merged 6 commits into from
Aug 2, 2021

Conversation

lhotari
Copy link
Member

@lhotari lhotari commented Jun 22, 2021 •
edited
Loading

Motivation

  • Pulsar docker images currently use openjdk:11-jdk-slim as the base image.

  • openjdk:11-jdk-slim/openjdk:11-jdk images are based on Debian 10 which contains a lot of unfixed vulnerabilities.

    • this causes the Pulsar docker images to get flagged in Docker image vulnerability
      scanning with docker image vulnerability scanning tools such as Clair

Modifications

  • Switch the base image to ubuntu:20.04. Install OpenJDK with apt-get.
  • Ubuntu fixes all critical and high severity security vulnerabilities. New security vulnerabilities get quickly patched in Ubuntu.
  • Python version is 3.8 in adoptopenjdk:11-jdk base image and that's why the switch requires changes to the Python & Pulsar Python client installation.

@lhotari lhotari added this to the 2.9.0 milestone Jun 22, 2021
@lhotari lhotari self-assigned this Jun 22, 2021
@lhotari lhotari force-pushed the lh-change-docker-baseimage branch 2 times, most recently from 8af5b5b to 26ac6ea Compare June 23, 2021 11:39
eolivelli
eolivelli approved these changes Jun 23, 2021
View reviewed changes
Copy link
Contributor

@eolivelli eolivelli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

I believe that it is super important to provide up-to-date base docker images for enterprise users.

It is good to commit this patch early at the beginning of 2.9 release cycle, this way we can get feedback from people who are testing from master branch

@eolivelli
Copy link
Contributor

@michaeljmarshall @sijie @rdhabalia @jerrypeng @codelipenghui
PTAL

michaeljmarshall
michaeljmarshall approved these changes Jun 23, 2021
View reviewed changes
Copy link
Member

@michaeljmarshall michaeljmarshall left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. This is great change, and it'll decrease the overall size of our pulsar docker images by a good bit. I just pulled these images and here is the basic docker images info:

REPOSITORY TAG IMAGE ID CREATED SIZE
openjdk 11-jdk bd97a328b384 8 hours ago 647MB
adoptopenjdk 11-jdk 20d606fe6719 5 days ago 437MB

@lhotari lhotari marked this pull request as draft June 23, 2021 17:51
@lhotari lhotari force-pushed the lh-change-docker-baseimage branch from 26ac6ea to 38b9e1e Compare June 23, 2021 18:49
@lhotari lhotari marked this pull request as ready for review June 23, 2021 18:49
sijie
sijie requested changes Jun 24, 2021
View reviewed changes
Copy link
Member

@sijie sijie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

who maintains adoptopenjdk?

@lhotari
Copy link
Member Author

lhotari commented Jun 24, 2021

who maintains adoptopenjdk?

The JDK, AdoptOpenJDK itself has moved to Eclipse Foundation and has been renamed to Adoptium. The GitHub organization is adoptium and the transition from AdoptOpenJDK to Adoptium is still in progress.

The adoptopenjdk docker images are maintained in 2 locations:

Both images use the same underlying Java binaries from AdoptOpenJDK, but they support different OS flavours as below. Also the official DockerHub images are updated for underlying OS updates as and when the OS fixes are available. The AdoptOpenJDK maintained images are rebuilt everyday.

@lhotari
Copy link
Member Author

lhotari commented Jun 24, 2021

Another choice for the adoptopenjdk base image would be to use the official ubuntu:20.04 base image and install the openjdk-11-jdk package in the container. This would also resolve the security issues that come from using the Debian based openjdk docker base image.

Switching to other than Ubuntu based OS would be more work since current Dockerfiles use apt to install packages and thus work for Debian/Ubuntu.

@sijie Please provide advice how to proceed.

@lhotari lhotari requested a review from sijie June 24, 2021 06:29
@sijie
Copy link
Member

sijie commented Jul 12, 2021

Another choice for the adoptopenjdk base image would be to use the official ubuntu:20.04 base image and install the openjdk-11-jdk package in the container.

I think this is a better approach.

@lhotari
[Security] Use ubuntu:20.04 base image for Pulsar docker images
cf5b047 
- Ubuntu fixes critical and high security vulnerabilities.
  - openjdk:11-jdk-slim/openjdk:11-jdk images are based on
    Debian 10 which contains a lot of unfixed vulnerabilities.
    - this causes the Pulsar docker images to get flagged in Docker image vulnerability
      scanning with docker image vulnerability scanning tools such as Clair
@lhotari
Install Ubuntu updates
b91b672
@lhotari lhotari force-pushed the lh-change-docker-baseimage branch from 38b9e1e to b91b672 Compare July 23, 2021 12:54
@lhotari lhotari changed the title [Security] Use adoptopenjdk:11-jdk base image for Pulsar docker images [Security] Use ubuntu:20.04 base image for Pulsar docker images Jul 23, 2021
@lhotari lhotari force-pushed the lh-change-docker-baseimage branch from 43f4407 to 70e02ab Compare July 23, 2021 17:22
eolivelli
eolivelli approved these changes Jul 24, 2021
View reviewed changes
Copy link
Contributor

@eolivelli eolivelli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lgtm

@lhotari lhotari merged commit f989512 into apache:master Aug 2, 2021
@timmyyuan timmyyuan mentioned this pull request Aug 10, 2021
Closed
@sijie sijie mentioned this pull request Aug 10, 2021
Closed
hangc0276 pushed a commit that referenced this pull request Aug 26, 2021
@lhotari @hangc0276
[Security] Use ubuntu:20.04 base image for Pulsar docker images (#11026)
3aa1f1d 
* [Security] Use ubuntu:20.04 base image for Pulsar docker images

- Ubuntu fixes critical and high security vulnerabilities.
  - openjdk:11-jdk-slim/openjdk:11-jdk images are based on
    Debian 10 which contains a lot of unfixed vulnerabilities.
    - this causes the Pulsar docker images to get flagged in Docker image vulnerability
      scanning with docker image vulnerability scanning tools such as Clair

* Install Ubuntu updates

* Set DEBIAN_FRONTEND=noninteractive so that dist-upgrade doesn't wait for input

* Set JAVA_HOME

* Fix configuring networkaddress.cache.ttl

- JAVA_HOME set by ENV isn't available for RUN commands

* Configure networkaddress.cache.ttl after installing OpenJDK

(cherry picked from commit f989512)
@hangc0276 hangc0276 added the cherry-picked/branch-2.8 Archived: 2.8 is end of life label Aug 26, 2021
@timmyyuan timmyyuan mentioned this pull request Aug 26, 2021
Closed
@hangc0276
Copy link
Contributor

Due to python3.7 is not available in openjdk:11-jdk image, this build exception is: https://github.com/apache/pulsar/runs/3428234664

So i cherry picked this PR to branch-2.8, and release in Pulsar 2.8.1 to solve docker image build failed problem.

@eolivelli
Copy link
Contributor

@hangc0276 please advertise this on dev@

This is an important change, we must be sure that the community is up to date

@hangc0276
Copy link
Contributor

@hangc0276 please advertise this on dev@

This is an important change, we must be sure that the community is up to date

@eolivelli Ok, I have sent an email to dev@ mail list, Thanks.

@hangc0276 hangc0276 mentioned this pull request Aug 31, 2021
Merged
hangc0276 added a commit that referenced this pull request Aug 31, 2021
@hangc0276
fix pulsar_standalone docker image build failed (#11862)
78a155b 
### Motivation
When build the pulsar-standalone docker image, it throw the following exception

[INFO] + sudo -u postgres /usr/lib/postgresql/11/bin/initdb /data/
[INFO]
[INFO] sudo: /usr/lib/postgresql/11/bin/initdb: command not found
[INFO]
[ERROR] The command '/bin/sh -c /pulsar/django/init-postgres.sh' returned a non-zero code: 1
[WARNING] An attempt failed, will retry 1 more times
org.apache.maven.plugin.MojoExecutionException: Could not build image
    at com.spotify.plugin.dockerfile.BuildMojo.buildImage (BuildMojo.java:247)
    at com.spotify.plugin.dockerfile.BuildMojo.execute (BuildMojo.java:135)
    at com.spotify.plugin.dockerfile.AbstractDockerMojo.tryExecute (AbstractDockerMojo.java:265)
    at com.spotify.plugin.dockerfile.AbstractDockerMojo.execute (AbstractDockerMojo.java:254)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:956)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:498)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
The root cause is when we use ubuntu 20.04 use base docker image instead of openjdk:11-jdk image introduce by #11026 , the ubuntu 20.04 will install postgresql 12 instead of postgresql 11 by default. However, the init and start script has been hard code with postgresql 11 install path, which will lead to command not found exception.

### Modification
1. add postgresql 11 resource address for ubuntu 20.04 and install postgresql 11
gaoran10 added a commit to gaoran10/pulsar that referenced this pull request Sep 10, 2021
@gaoran10
1. Use ubuntu:20.04 base image for Pulsar docker images
ebc2b97 
2. Build python client by Python3.8

cherry-pick:
1. apache#11026
2. apache#11623
3. apache#11862
This was referenced Sep 10, 2021
Closed
Merged
codelipenghui pushed a commit that referenced this pull request Sep 14, 2021
@gaoran10
[Branch-2.7][Build] Use ubuntu 20.04 as docker image base (#12017)
259c698 
### Motivation

Currently, the docker is from `openjdk:8-jdk-slim `, it has a new release a few days ago and the Linux version was changed to `Impish`, it didn't support install `python3.7` by apt-get tool, the minimum Python version is 3.9, so we need to change the python version or use a different Linux release, I cherry-pick some commits from branch master to build docker image from the `ubuntu:20.04`.

Mainly related PRs:

1. #11026
2. #11623
3. #11862

### Modifications

Build docker image from the `ubuntu:20.04`.

upgrade pulsar-go-client from `0.2.0` to `0.6.0` and rever #9124 from branch-2.7
bharanic-dev pushed a commit to bharanic-dev/pulsar that referenced this pull request Mar 18, 2022
@lhotari @ciaocloud
[Security] Use ubuntu:20.04 base image for Pulsar docker images (apac…
1510443 
…he#11026)

* [Security] Use ubuntu:20.04 base image for Pulsar docker images

- Ubuntu fixes critical and high security vulnerabilities.
  - openjdk:11-jdk-slim/openjdk:11-jdk images are based on
    Debian 10 which contains a lot of unfixed vulnerabilities.
    - this causes the Pulsar docker images to get flagged in Docker image vulnerability
      scanning with docker image vulnerability scanning tools such as Clair

* Install Ubuntu updates

* Set DEBIAN_FRONTEND=noninteractive so that dist-upgrade doesn't wait for input

* Set JAVA_HOME

* Fix configuring networkaddress.cache.ttl

- JAVA_HOME set by ENV isn't available for RUN commands

* Configure networkaddress.cache.ttl after installing OpenJDK
nicoloboschi pushed a commit to datastax/pulsar that referenced this pull request Jul 22, 2022
@gaoran10 @nicoloboschi
[Branch-2.7][Build] Use ubuntu 20.04 as docker image base (apache#12017)
13e0bc4 
Currently, the docker is from `openjdk:8-jdk-slim `, it has a new release a few days ago and the Linux version was changed to `Impish`, it didn't support install `python3.7` by apt-get tool, the minimum Python version is 3.9, so we need to change the python version or use a different Linux release, I cherry-pick some commits from branch master to build docker image from the `ubuntu:20.04`.

Mainly related PRs:

1. apache#11026
2. apache#11623
3. apache#11862

Build docker image from the `ubuntu:20.04`.

upgrade pulsar-go-client from `0.2.0` to `0.6.0` and rever apache#9124 from branch-2.7

(cherry picked from commit 259c698)
nicoloboschi pushed a commit to datastax/pulsar that referenced this pull request Jul 22, 2022
@gaoran10 @nicoloboschi
[Branch-2.7][Build] Use ubuntu 20.04 as docker image base (apache#12017)
d313a0b 
Currently, the docker is from `openjdk:8-jdk-slim `, it has a new release a few days ago and the Linux version was changed to `Impish`, it didn't support install `python3.7` by apt-get tool, the minimum Python version is 3.9, so we need to change the python version or use a different Linux release, I cherry-pick some commits from branch master to build docker image from the `ubuntu:20.04`.

Mainly related PRs:

1. apache#11026
2. apache#11623
3. apache#11862

Build docker image from the `ubuntu:20.04`.

upgrade pulsar-go-client from `0.2.0` to `0.6.0` and rever apache#9124 from branch-2.7

(cherry picked from commit 259c698)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sijie sijie sijie approved these changes

@eolivelli eolivelli eolivelli approved these changes

@michaeljmarshall michaeljmarshall michaeljmarshall approved these changes

@aahmed-se aahmed-se Awaiting requested review from aahmed-se

@merlimat merlimat Awaiting requested review from merlimat

@codelipenghui codelipenghui Awaiting requested review from codelipenghui

Assignees

@lhotari lhotari

Labels
area/security cherry-picked/branch-2.8 Archived: 2.8 is end of life release/2.8.1
Projects
None yet
Milestone
2.9.0
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@lhotari @eolivelli @sijie @hangc0276 @michaeljmarshall