Overhaul releasing process #779

TriplEight · 2021-03-01T13:21:39Z

-> Due to the long image builds we have to use the self-hosted runner.
-> A self-hosted runner can't buildah
Have to update to the newer version of the deprecated official docker action.
This is, in fact, not bad. Now the action uses BuildX.
-> BuildX has an ugly bug

Overhaul of publishing

remove multistage build as it is a CI recursion
upgrade to ubuntu:20.04 base
CI: move to use buildah actions
chore

Dockerfile

TriplEight · 2021-03-02T20:43:41Z

Got these images building sending them to the public runners with buildah action.

Caching is not helpful if it gets removed every day, invalidated every day with the nightly rust update, executes every time on a fresh VM, etc. Also, there was a lot of duplicated processes and esp when launched 4 times in parallel it doesn't make anything faster. Besides, --package and other flags are not yet working in cargo-chef.

Still, they take a surprisingly long time to build (the last image gets built in 46min), normally it's 3.5 min for each without any cache.

I'd suggest an optimization - produce the binaries separately (it's faster and probs you'll want the binaries somewhere anyway) in CI and on the images stage just put them in the containers.

TriplEight · 2021-03-02T20:44:45Z

Oh btw I've updated the images and now they run off ubuntu:20.04 please check if they are working properly.

TriplEight · 2021-03-02T20:45:52Z

The next step is I'll steal your CI dockerfile to build it nightly with the other images in #paritytech/scripts.

TriplEight · 2021-03-02T21:10:10Z

Regarding

Besides, --package and other flags are not yet working in cargo-chef.

I lied, just found that they've addressed my issue.

TriplEight · 2021-03-04T23:00:16Z

please check if these images work correctly before merging:
paritytech/millau-bridge-node:ubuntu
paritytech/ethereum-poa-relay:ubuntu
paritytech/rialto-bridge-node:ubuntu
paritytech/substrate-relay:ubuntu
paritytech/bridge-dependencies:latest

tomusdrw · 2021-03-05T13:03:45Z

please check if these images work correctly before merging:

We will learn that on our testnet deployment :)

HCastano

Looks good!

HCastano · 2021-03-05T16:41:37Z

deployments/BridgeDeps.Dockerfile

@@ -2,9 +2,10 @@
 #
 # This image is meant to be used as a building block when building images for
 # the various components in the bridge repo, such as nodes and relayers.
-FROM ubuntu:xenial
+FROM ubuntu:20.04

 ENV LAST_DEPS_UPDATE 2020-12-21


We should bump this and the one on line 21.

The idea behind these lines is that we can use them to break the cache and update dependencies if needed.

HCastano · 2021-03-05T16:42:14Z

Dockerfile

@@ -8,51 +8,33 @@
 #
 # See the `deployments/README.md` for all the available `PROJECT` values.

-# This first stage prepares our dependencies to be built by `cargo-chef`.
-FROM rust as planner
+FROM docker.io/paritytech/bridge-dependencies as builder


Is it not good enough to just do paritytech/bridge-dependencies?

it's better to have the full names when building with something that's not docker, we use buildah.

HCastano · 2021-03-05T16:45:05Z

Dockerfile


 ARG PROJECT=ethereum-poa-relay
 RUN cargo build --release --verbose -p ${PROJECT}
 RUN strip ./target/release/${PROJECT}

 # In this final stage we copy over the final binary and do some checks
 # to make sure that everything looks good.
-FROM ubuntu:xenial as runtime
+FROM docker.io/ubuntu:20.04 as runtime


Same question here, why do we need to specify docker.io?

HCastano · 2021-03-05T16:48:51Z

Dockerfile

+		libssl-dev curl && \
+	groupadd -g 1000 user && \
+	useradd -u 1000 -g user -s /bin/sh -m user && \
+# apt clean up


Suggested change

# apt clean up

# apt clean up

HCastano · 2021-03-05T16:50:18Z

.github/workflows/publish-docker.yml

+          echo ::set-output name=DATE::$(date +%d-%m-%Y)
+
+      - name:                                       Build image for ${{ matrix.project }}
+        uses:                                       redhat-actions/buildah-build@v2.2


Why did you choose buildah over build-push-action?

HCastano · 2021-03-08T14:20:53Z

Pinging @TriplEight since I've got some questions here

HCastano · 2021-03-08T21:19:28Z

I fixed some of my nitpicks since I want this to get used on tonight's testnet deployment.

CI: instead of migrating docker's action braking changes move to buildah

7727baf

tomusdrw reviewed Mar 1, 2021

View reviewed changes

Dockerfile Outdated Show resolved Hide resolved

TriplEight added 10 commits March 1, 2021 16:35

CI: try docker v2 action

93d9b87

CI: correct user var

5a1ddac

CI: copypasta

3afdf49

CI: debug

9ade53a

CI: weird action

dcd8759

CI: hates quotes

802d8e8

CI: hates all quotes

8b5a41d

CI: masterpiece (debug)

ad2434a

CI: migrate to docker/build-push-action@v2

17941b7

CI: very heavy compilation, gotta be selfhosted

deb3cb0

HCastano marked this pull request as draft March 1, 2021 19:28

HCastano added the P-Devops label Mar 1, 2021

HCastano changed the title ~~wip: CI: move to buildah to build images~~ Move to buildah to build images Mar 1, 2021

TriplEight self-assigned this Mar 1, 2021

TriplEight requested a review from rcny March 1, 2021 19:41

TriplEight changed the title ~~Move to buildah to build images~~ Migrate to the new docker action Mar 1, 2021

TriplEight changed the title ~~Migrate to the new docker action~~ Migrate to the new docker action version Mar 1, 2021

TriplEight changed the title ~~Migrate to the new docker action version~~ WIP: Migrate to the new docker action version Mar 1, 2021

TriplEight added 10 commits March 1, 2021 21:43

CI: now image cache ruins the build

8933e39

CI: tuning the self-hosted build

fb1e395

CI: pin checkout

9aa8d07

CI: strip dockerfile; use buildah

a2095bf

dockerfile: qualify the registry

67b46fb

dockerfile: fogot to remove chef

18aa0aa

dockerfile: nit

128734c

dockerfile: fryin to guess

5f0f6ea

CI: fix tags ang images name

5f47250

CI: space separated, right

60daea6

CI: ad quoted

ad42d3a

TriplEight added 3 commits March 4, 2021 21:28

CI: masterpiece (truly)

e07ddaa

CI: noninteractive frontend

485fbe6

CI: thre's no g++

4aef4e3

TriplEight marked this pull request as ready for review March 5, 2021 09:08

TriplEight changed the title ~~WIP: Migrate to the new docker action version~~ CI: Migrate to the new docker action version Mar 5, 2021

TriplEight changed the title ~~CI: Migrate to the new docker action version~~ CI: Overhaul releasing process Mar 5, 2021

tomusdrw enabled auto-merge (squash) March 5, 2021 13:03

tomusdrw approved these changes Mar 5, 2021

View reviewed changes

HCastano changed the title ~~CI: Overhaul releasing process~~ Overhaul releasing process Mar 5, 2021

HCastano disabled auto-merge March 5, 2021 16:54

HCastano approved these changes Mar 5, 2021

View reviewed changes

HCastano added 2 commits March 8, 2021 16:14

Bump LAST_UPDATE dates

7fa8270

Address some small style nitpicks

88852a5

HCastano enabled auto-merge (squash) March 8, 2021 21:16

HCastano merged commit a363ec0 into master Mar 8, 2021

HCastano deleted the 3x8_use_buildah branch March 8, 2021 22:07

tomusdrw mentioned this pull request Mar 9, 2021

Migrate docker images to use focal #626

Closed

HCastano mentioned this pull request Mar 12, 2021

Update docker/build-push-action to V2 #630

Closed

svyatonik pushed a commit that referenced this pull request Jul 17, 2023

Add entry in CODEOWNERS (#779)

146a0a6

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Overhaul releasing process #779

Overhaul releasing process #779

TriplEight commented Mar 1, 2021 •

edited

TriplEight commented Mar 2, 2021 •

edited

TriplEight commented Mar 2, 2021

TriplEight commented Mar 2, 2021

TriplEight commented Mar 2, 2021 •

edited

TriplEight commented Mar 4, 2021

tomusdrw commented Mar 5, 2021

HCastano left a comment

HCastano Mar 5, 2021

HCastano Mar 5, 2021

TriplEight Mar 9, 2021

HCastano Mar 5, 2021

HCastano Mar 5, 2021

HCastano Mar 5, 2021

HCastano commented Mar 8, 2021

HCastano commented Mar 8, 2021 •

edited

Overhaul releasing process #779

Overhaul releasing process #779

Conversation

TriplEight commented Mar 1, 2021 • edited

TriplEight commented Mar 2, 2021 • edited

TriplEight commented Mar 2, 2021

TriplEight commented Mar 2, 2021

TriplEight commented Mar 2, 2021 • edited

TriplEight commented Mar 4, 2021

tomusdrw commented Mar 5, 2021

HCastano left a comment

Choose a reason for hiding this comment

HCastano Mar 5, 2021

Choose a reason for hiding this comment

HCastano Mar 5, 2021

Choose a reason for hiding this comment

TriplEight Mar 9, 2021

Choose a reason for hiding this comment

HCastano Mar 5, 2021

Choose a reason for hiding this comment

HCastano Mar 5, 2021

Choose a reason for hiding this comment

HCastano Mar 5, 2021

Choose a reason for hiding this comment

HCastano commented Mar 8, 2021

HCastano commented Mar 8, 2021 • edited

TriplEight commented Mar 1, 2021 •

edited

TriplEight commented Mar 2, 2021 •

edited

TriplEight commented Mar 2, 2021 •

edited

HCastano commented Mar 8, 2021 •

edited