Ubuntu transition #157

brosenberg42 · 2021-11-02T19:41:19Z

Issues:

Related PRs:

This change is

jrobble · 2022-01-07T18:14:23Z

activemq/Dockerfile, line 43 at r1 (raw file):

RUN wget -O- https://archive.apache.org/dist/activemq/5.16.3/apache-activemq-5.16.3-bin.tar.gz \
        | tar --extract --gzip --directory /opt; \
    ln --symbolic /opt/apache-activemq-5.16.3 /opt/activemq; \

Dropping a note here that we need to either update to ActiveMQ 5.17.0 to address Trivy scan issues, or get sign-off by the end users.

jrobble

Reviewed 17 of 17 files at r1, all commit messages.
Reviewable status: all files reviewed, 9 unresolved discussions (waiting on @brosenberg42)

docker-compose.core.yml, line 46 at r1 (raw file):

    image: ${REGISTRY}openmpf_activemq:${TAG}
    build: activemq
    init: true

Why add this?

Jenkinsfile, line 428 at r1 (raw file):

                def service = composeYaml.services[serviceName]
                if (!service.build || serviceName == 'kibana') {
                    echo "Not scanning $service.image since we didn't build it"

I think we should scan all of the images, including those we didn't build (or at least provide an option to enable that). That way we can know of all of the issues before they do scans on the production side. For example, I don't want ActiveMQ issues like the log4j 1 ones to come as a surprise.

Jenkinsfile, line 441 at r1 (raw file):
I think we should alter the status of the job to reflect these findings. How about we mark the build as UNSTABLE, as shown here?

From the Jenkins API for UNSTABLE:

The build had some errors but they were not fatal. For example, some tests failed.

If a user is actively deciding to enable Trivy scans, I think they would want the job status to reflect the result of the scans.

I could see how changing the status might be an annoyance if the scan is done as part of a normal build.

activemq/Dockerfile, line 34 at r1 (raw file):

SHELL ["/bin/bash", "-o", "errexit", "-o", "pipefail", "-c"]

ENV LANG en_US.UTF-8

When is this needed, and why? I see it in other Dockerfiles too.

components/java_component_build/Dockerfile, line 50 at r1 (raw file):

COPY --from=openmpf_build /usr/local/bin/ffmpeg /usr/local/bin/ffprobe /usr/local/bin/

RUN wget -O- 'https://archive.apache.org/dist/maven/maven-3/3.3.3/binaries/apache-maven-3.3.3-bin.tar.gz' \

Why switch from curl to wget?

components/python/Dockerfile, line 82 at r1 (raw file):

RUN python3 -m venv "$COMPONENT_VIRTUALENV"; \
    pip3 install --no-cache-dir --upgrade pip;

Will pip3 be updated at the same time as installing python3?

openmpf_build/Dockerfile, line 43 at r1 (raw file):

ENV PIP_COMPILE=0

ENV DEBIAN_FRONTEND=noninteractive

Maybe we should set this as an ARG? See this.

workflow_manager/docker-entrypoint.sh, line 80 at r1 (raw file):

    [ ! "$cert" ] && continue

    "$JAVA_HOME/bin/keytool" -import -alias "mpf_imported_$((cert_counter++))" -file "$cert" -cacerts \

We don't need to update the Java keystore anymore? Or does update-ca-certificates take care of that?

brosenberg42

Reviewable status: 16 of 17 files reviewed, 9 unresolved discussions (waiting on @brosenberg42 and @jrobble)

docker-compose.core.yml, line 46 at r1 (raw file):

Previously, jrobble (Jeff Robble) wrote…

Why add this?

It is so that it will respond to signals. Without it, it doesn't respond to SIGTERM, which then makes Docker kill it after 10 seconds.

Jenkinsfile, line 428 at r1 (raw file):

Previously, jrobble (Jeff Robble) wrote…

I think we should scan all of the images, including those we didn't build (or at least provide an option to enable that). That way we can know of all of the issues before they do scans on the production side. For example, I don't want ActiveMQ issues like the log4j 1 ones to come as a surprise.

This will create a ton of useless logging. If we find a vulnerability in in something like Redis, there isn't anything we can do about it.

Jenkinsfile, line 441 at r1 (raw file):

Previously, jrobble (Jeff Robble) wrote…

I think we should alter the status of the job to reflect these findings. How about we mark the build as UNSTABLE, as shown here?

From the Jenkins API for UNSTABLE:

The build had some errors but they were not fatal. For example, some tests failed.

If a user is actively deciding to enable Trivy scans, I think they would want the job status to reflect the result of the scans.

I could see how changing the status might be an annoyance if the scan is done as part of a normal build.

There are almost always going to be vulnerabilities in dependencies we use, so your suggestions will result in every build using Trivy to be unstable.

activemq/Dockerfile, line 34 at r1 (raw file):

Previously, jrobble (Jeff Robble) wrote…

When is this needed, and why? I see it in other Dockerfiles too.

I added it when I got build errors related to locale or encoding. From what I remember, I think boost-regex and the Maven assembly plugin need it.

components/java_component_build/Dockerfile, line 50 at r1 (raw file):

Previously, jrobble (Jeff Robble) wrote…

Why switch from curl to wget?

The examples I saw for Ubuntu all used it. Neither is installed by default and wget is a tiny bit smaller.

components/python/Dockerfile, line 82 at r1 (raw file):

Previously, jrobble (Jeff Robble) wrote…

Will pip3 be updated at the same time as installing python3?

We want to use the version of pip that comes from apt-get. It may lag behind the latest version of pip by a little bit.

openmpf_build/Dockerfile, line 43 at r1 (raw file):

Previously, jrobble (Jeff Robble) wrote…

Maybe we should set this as an ARG? See this.

Done.

workflow_manager/docker-entrypoint.sh, line 80 at r1 (raw file):

Previously, jrobble (Jeff Robble) wrote…

We don't need to update the Java keystore anymore? Or does update-ca-certificates take care of that?

On Ubuntu, Java will also check the system's certificates.

jrobble

Reviewable status: 16 of 17 files reviewed, 3 unresolved discussions (waiting on @brosenberg42 and @jrobble)

Jenkinsfile, line 428 at r1 (raw file):

Previously, brosenberg42 wrote…

This will create a ton of useless logging. If we find a vulnerability in in something like Redis, there isn't anything we can do about it.

We still need to report it or get it waived.

Jenkinsfile, line 441 at r1 (raw file):

Previously, brosenberg42 wrote…

There are almost always going to be vulnerabilities in dependencies we use, so your suggestions will result in every build using Trivy to be unstable.

I hear what you're saying. I think it boils down to the use case.

I certainly wouldn't want to enable Trivy on develop and master builds that automatically get kicked off, or on pull request builds; however, if I'm initiating a build for the sole purpose of looking at the Trivy results, then I probably want the state of the build to reflect the results.

If we're going to enable Trivy scans on the master builds that we tag for GA release then I agree that we don't want to mark those as unstable.

When do you think it makes the most sense to kick off a build with Trivy scans enabled? Do you think they should be part of the GA release build, or a separate build altogether?

Another, additional, option is to consider adding Trivy scans to a nightly master build. If nightly, I don't want to have to look at the logs every time to determine that there is a new vuln. We could consider whitelisting things that are waived.

jrobble

Reviewed 1 of 1 files at r2.
Reviewable status: all files reviewed (commit messages unreviewed), 3 unresolved discussions (waiting on @brosenberg42)

jrobble · 2022-01-31T19:19:48Z

Jenkinsfile, line 441 at r1 (raw file):

adding Trivy scans to a nightly master build

This could be on the clean build, since that pulls in new dependencies.

jrobble · 2022-02-01T18:07:29Z

Jenkinsfile, line 441 at r1 (raw file):

Previously, jrobble (Jeff Robble) wrote…

adding Trivy scans to a nightly master build

This could be on the clean build, since that pulls in new dependencies.

We decided to do the Trivy scans as part of the release builds. We also considered doing them proactively / periodically, but decided instead to do them on-demand.

brosenberg42

Reviewable status: 16 of 18 files reviewed, 2 unresolved discussions (waiting on @brosenberg42 and @jrobble)

Jenkinsfile, line 428 at r1 (raw file):

Previously, jrobble (Jeff Robble) wrote…

We still need to report it or get it waived.

Done.

jrobble

Reviewed 2 of 2 files at r3, 1 of 1 files at r4, all commit messages.
Reviewable status: complete! all files reviewed, all discussions resolved (waiting on @brosenberg42)

activemq/Dockerfile, line 43 at r1 (raw file):

Previously, jrobble (Jeff Robble) wrote…

Dropping a note here that we need to either update to ActiveMQ 5.17.0 to address Trivy scan issues, or get sign-off by the end users.

Resolved by updating to 5.16.4.

brosenberg42 added 3 commits October 28, 2021 11:22

Ubuntu transition

9372c98

Merge branch 'develop' into feature/ubuntu

e4a931e

Pull ubuntu in Jenkinsfile

2060b47

brosenberg42 self-assigned this Nov 2, 2021

This was referenced Nov 2, 2021

Ubuntu transition openmpf/openmpf#1447

Merged

Ubuntu transition openmpf/openmpf-components#287

Merged

Ubuntu transition openmpf/openmpf-contrib-components#76

Merged

Ubuntu transition openmpf/openmpf-cpp-component-sdk#108

Merged

brosenberg42 requested a review from jrobble November 2, 2021 19:46

brosenberg42 added 4 commits December 17, 2021 09:03

Merge branch 'develop' into feature/ubuntu

c5adf7c

Update ActiveMQ

6820665

Add Trivy scan to Jenkinsfile

faf91cb

Update trivy scan code

a5b50cf

brosenberg42 mentioned this pull request Jan 18, 2022

Ubuntu Transition openmpf/openmpf-java-component-sdk#56

Merged

Merge branch 'develop' into feature/ubuntu

c6f1ecc

jrobble requested changes Jan 26, 2022

View reviewed changes

Move DEBIAN_FRONTEND=noninteractive

601170b

brosenberg42 commented Jan 31, 2022

View reviewed changes

jrobble requested changes Jan 31, 2022

View reviewed changes

jrobble reviewed Jan 31, 2022

View reviewed changes

brosenberg42 added 3 commits February 1, 2022 08:19

Add trivyignore.txt

139027e

Use absolute path for trivyignore

53cedd3

Increase trivvy timeout

8018007

brosenberg42 commented Feb 2, 2022

View reviewed changes

Update to ActiveMQ 5.16.4

1a5ad4e

jrobble approved these changes Feb 17, 2022

View reviewed changes

Switch to docker compose plugin

59c5b94

brosenberg42 added 2 commits March 28, 2022 09:39

Update ActiveMQ

cc4f154

Update ActiveMQ

cb28869

brosenberg42 merged commit 807983f into develop Mar 30, 2022

brosenberg42 deleted the feature/ubuntu branch March 30, 2022 18:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ubuntu transition #157

Ubuntu transition #157

brosenberg42 commented Nov 2, 2021 •

edited by jrobble

jrobble commented Jan 7, 2022

jrobble left a comment

brosenberg42 left a comment

jrobble left a comment

jrobble left a comment

jrobble commented Jan 31, 2022

jrobble commented Feb 1, 2022

brosenberg42 left a comment

jrobble left a comment

Ubuntu transition #157

Ubuntu transition #157

Conversation

brosenberg42 commented Nov 2, 2021 • edited by jrobble

jrobble commented Jan 7, 2022

jrobble left a comment

Choose a reason for hiding this comment

brosenberg42 left a comment

Choose a reason for hiding this comment

jrobble left a comment

Choose a reason for hiding this comment

jrobble left a comment

Choose a reason for hiding this comment

jrobble commented Jan 31, 2022

jrobble commented Feb 1, 2022

brosenberg42 left a comment

Choose a reason for hiding this comment

jrobble left a comment

Choose a reason for hiding this comment

brosenberg42 commented Nov 2, 2021 •

edited by jrobble