Use Debian bullseye on CI/CD/Dockerfile #1005

thiagokokada · 2021-09-09T21:24:31Z

Fix issue #866.

Also, many QoL changes on Dockerfile/.cicleci/config.yml.

Also, some QoL changes on the `Dockerfile`: - Parametrize GRAALVM_VERSION to make it easier to update - Remove installing programs already included on base image - Remove `deps.edn` from `.dockerignore` since it is necessary to build

Also some modifications necessary to make it work: - Migrate from `circleci` images to `clojure` since they're out-of-date and also seem to be considered legacy: https://circleci.com/docs/2.0/circleci-images/#legacy-language-images - Remove unnecessary usage of `sudo`, since all commands runs as root - Sync packages with `Dockerfile`. This will make easier to test locally if everything will work (of course, it is not 100% guarantee)

.circleci/config.yml

thiagokokada · 2021-09-09T21:27:25Z

.dockerignore

@@ -12,7 +12,6 @@ test/
 .gitmodules
 appveyor.yml
 CHANGES.md
-deps.edn


I don't know how this worked before, but this file is necessary to build using the Dockerfile.

thiagokokada · 2021-09-09T21:27:56Z

Dockerfile


 ENV DEBIAN_FRONTEND=noninteractive
 RUN apt update
-RUN apt install --no-install-recommends -yy curl unzip build-essential zlib1g-dev sudo


curl, unzip are already included on clojure:openjdk-11-lein-2.9.6-bullseye. sudo is not used anywhere in the Dockerfile (and also, all commands already run as root).

thiagokokada · 2021-09-09T21:28:26Z

script/setup-musl

-# This script sets up the latest available musl-tools using apt pinning from Debian bullseye (11).
-# The one available in stretch (Debian 9) is outdated and this ensures we get the latest improvements
-# This explictly installs musl from bullseye and keeps the others at a higher priority
-# TODO: remove after clojure image is updated to bullseye
-
-cat >> /etc/apt/sources.list <<eof
-deb http://ftp.us.debian.org/debian bullseye main non-free contrib
-deb http://non-us.debian.org/debian-non-us bullseye/non-us main contrib non-free
-eof
-
-cat >> /etc/apt/preferences <<eof
-Package: *
-Pin: release a=stable
-Pin-Priority: 700
-Package: *
-Pin: release a=testing
-Pin-Priority: 650
-Package: *
-pin: release a=bullseye
-pin-priority: 600
-eof
-
-apt-get update -y && apt-get install musl-tools/bullseye -y


Not necessary to backport anymore since we are already using bullseye.

thiagokokada · 2021-09-09T21:31:32Z

.circleci/config.yml

-      - run:
-          name: Install lsof
-          command: |
-            sudo apt-get install lsof


Not used anywhere and not available on clojure:openjdk-11-lein-2.9.6-bullseye by default.

Already included on the base image.

.circleci/config.yml

thiagokokada · 2021-09-09T21:38:01Z

Huh... I thought that CircleCI built everything as root, but it seems it builds with a non-root user. So this approach will not work, sadly.

borkdude · 2021-09-09T21:41:19Z

@thiagokokada You can use sudo though.

thiagokokada · 2021-09-09T21:42:33Z

@thiagokokada You can use sudo though.

It is not available on clojure:openjdk-11-lein-2.9.6-bullseye by default, and we have a chicken-egg problem for installing it (I can't use sudo apt install sudo without sudo).

thiagokokada · 2021-09-09T21:44:11Z

Actually, it does seem that the commands without sudo worked, so the problem seems something else. Needs more investigation...

This reverts commit 5e2a615.

thiagokokada · 2021-09-09T22:37:49Z

.circleci/config.yml

@@ -7,7 +7,7 @@ jobs:
  jvm:
    docker:
      # specify the version you desire here
-      - image: circleci/clojure:lein-2.9.1
+      - image: circleci/clojure:lein-2.9.6


jvm needs PostgreSQL to run, so using the circleci/clojure:lein-2.9.6 that already includes it by default.

thiagokokada · 2021-09-09T23:27:36Z

@borkdude Well, it is working now (sorry for all the runs on CircleCI). BTW, what I said above was wrong, the error was purely my fault.

Not entirely happy with this PR, will wait some days to see if CircleCI releases their updated image with more tooling included. However if they do not, I will open the PR for review later.

borkdude · 2021-09-10T07:46:02Z

Thanks so far, I'll wait.

borkdude · 2021-09-10T09:15:22Z

@thiagokokada Could you perhaps not use force push? I don't think it adds anything, but it makes following what's going on a lot harder for me. The Github UI shows "this isn't here anymore"-like things when I click on the notifications.

thiagokokada · 2021-09-10T09:15:55Z

Sorry @borkdude, will avoid this from now on.

This reverts commit 3cfad03.

thiagokokada · 2021-09-10T09:24:37Z

I think for clean-ups is pretty much it, I will just wait to see if CircleCI will release a new version of their Clojure image.

thiagokokada · 2021-09-13T21:20:23Z

@borkdude It is ready now.

borkdude · 2021-09-13T21:28:16Z

@thiagokokada Beautiful, thank you!

thiagokokada added 2 commits September 9, 2021 18:18

Migrate Dockerfile to use a bullseye-based image

8c5c1bb

Also, some QoL changes on the `Dockerfile`: - Parametrize GRAALVM_VERSION to make it easier to update - Remove installing programs already included on base image - Remove `deps.edn` from `.dockerignore` since it is necessary to build

thiagokokada commented Sep 9, 2021

View reviewed changes

.circleci/config.yml Outdated Show resolved Hide resolved

thiagokokada commented Sep 9, 2021

View reviewed changes

Remove lsof

4f869a2

thiagokokada commented Sep 9, 2021

View reviewed changes

thiagokokada marked this pull request as draft September 9, 2021 21:32

Remove "Install {Clojure,Leiningen}" steps

e407762

Already included on the base image.

thiagokokada commented Sep 9, 2021

View reviewed changes

.circleci/config.yml Outdated Show resolved Hide resolved

thiagokokada added 3 commits September 9, 2021 18:47

Do not change directory when downloading GraalVM

8248eea

Move "Download GraalVM" to script/download-graalvm

d4d4737

Set GRAALVM_HOME correctly

5e2a615

thiagokokada force-pushed the use-bullseye branch from 6c750d9 to 5e2a615 Compare September 9, 2021 21:57

thiagokokada added 6 commits September 9, 2021 18:58

Unbreak mac build

52e2a72

Revert "Set GRAALVM_HOME correctly"

a3bf357

This reverts commit 5e2a615.

Set GRAALVM_HOME correctly, take 2

8df29c6

Improve download-graalvm script

94fbf59

Re-added "Install Clojure" steps

fcc23b7

Set amd64 as default GRAALVM_ARCH

cb4ed7e

thiagokokada force-pushed the use-bullseye branch from b1bab9a to cb4ed7e Compare September 9, 2021 22:25

thiagokokada added 4 commits September 9, 2021 19:31

Unbreak linux-aarch64

96870db

Ubreak jvm

3dcbd81

Do not change directory

3b05fed

Fix yaml

3594647

thiagokokada commented Sep 9, 2021

View reviewed changes

Fix mac/linux-aarch64

a435c36

thiagokokada added 2 commits September 9, 2021 19:40

Add missing Clojure

4eb1378

Fix cache

1f45e0d

Move GraalVM installation to /tmp

0ed3f61

thiagokokada force-pushed the use-bullseye branch from 9b53844 to 0ed3f61 Compare September 10, 2021 08:44

Use script/install-clojure

873841c

thiagokokada force-pushed the use-bullseye branch 3 times, most recently from 0ed8639 to 7951a21 Compare September 10, 2021 09:12

Use /opt instead of /tmp to install GraalVM

3cfad03

thiagokokada force-pushed the use-bullseye branch from 7951a21 to 3cfad03 Compare September 10, 2021 09:15

thiagokokada closed this Sep 10, 2021

thiagokokada reopened this Sep 10, 2021

Revert "Use /opt instead of /tmp to install GraalVM"

2b2831e

This reverts commit 3cfad03.

thiagokokada added 2 commits September 13, 2021 17:47

Use CircleCI's Clojure images again

d321fa8

Go back to installing GraalVM on $HOME

2235377

thiagokokada marked this pull request as ready for review September 13, 2021 21:20

borkdude merged commit 9ca3a3b into babashka:master Sep 13, 2021

thiagokokada deleted the use-bullseye branch September 15, 2021 19:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use Debian bullseye on CI/CD/Dockerfile #1005

Use Debian bullseye on CI/CD/Dockerfile #1005

thiagokokada commented Sep 9, 2021

thiagokokada Sep 9, 2021

thiagokokada Sep 9, 2021

thiagokokada Sep 9, 2021

thiagokokada Sep 9, 2021

thiagokokada commented Sep 9, 2021

borkdude commented Sep 9, 2021

thiagokokada commented Sep 9, 2021

thiagokokada commented Sep 9, 2021

thiagokokada Sep 9, 2021

thiagokokada commented Sep 9, 2021

borkdude commented Sep 10, 2021

borkdude commented Sep 10, 2021

thiagokokada commented Sep 10, 2021

thiagokokada commented Sep 10, 2021

thiagokokada commented Sep 13, 2021

borkdude commented Sep 13, 2021

Use Debian bullseye on CI/CD/Dockerfile #1005

Use Debian bullseye on CI/CD/Dockerfile #1005

Conversation

thiagokokada commented Sep 9, 2021

thiagokokada Sep 9, 2021

Choose a reason for hiding this comment

thiagokokada Sep 9, 2021

Choose a reason for hiding this comment

thiagokokada Sep 9, 2021

Choose a reason for hiding this comment

thiagokokada Sep 9, 2021

Choose a reason for hiding this comment

thiagokokada commented Sep 9, 2021

borkdude commented Sep 9, 2021

thiagokokada commented Sep 9, 2021

thiagokokada commented Sep 9, 2021

thiagokokada Sep 9, 2021

Choose a reason for hiding this comment

thiagokokada commented Sep 9, 2021

borkdude commented Sep 10, 2021

borkdude commented Sep 10, 2021

thiagokokada commented Sep 10, 2021

thiagokokada commented Sep 10, 2021

thiagokokada commented Sep 13, 2021

borkdude commented Sep 13, 2021