new(ci): gha master and release workflows #2501

FedeDP · 2023-04-18T08:38:28Z

What type of PR is this?

/kind feature

Any specific area of the project related to this PR?

/area CI

What this PR does / why we need it:

This PR tries to enable master and release build + packages + docker images CI on gha instead of circleci.
It surely needs some tests hence the wip.

To do so, it uses reusable workflows that are shared between master and release; therefore testing CI on master should be enough to test also the release one, since they are exactly the same.
Moreover, people could also reuse our provided workflows in their own CI.

Which issue(s) this PR fixes:

Fixes #1876

Special notes for your reviewer:

Steps:

fix arm64 build
fix arm64 job build Falco version (it sees 0.0.0 somehow...)
add GPG_KEY as gha variable (needs repo admin)
fix master and release yaml issues
understand how to docker in docker (reusable_build_docker workflow) -> just go on host. Tested that docker and awscli can be used in those jobs
Fix packages publishing (see new(config): added iam s3 access roles for Falco (dev and release). test-infra#1102)
Make reusable_publish_docker the only docker publishing job (ie: build_docker today is publishing images too, while publish_docker is using the docker manifest API to merge architectures specific image into a multi arch tag)
What about ecr docker publishing?
test master :)

Other notes:

reusable_build_packages job was tested with a fake job in the PR ci on the arm64 node: https://github.com/falcosecurity/falco/actions/runs/4734095566
The operation was canceled, but it worked just fine.
See the commit with the code: 75759c8

Also, as you can see, reusable_build_packages is not using the falco modern builder image; instead it is directly building in a centos:7 container (that is the base image for the falco modern builder). This is needed because we cannot run docker in docker on our prow nodes (ie: the arm64 ones).
Instead, the skeleton building phase is made on a fedora:latest image, because it already ships bpftool in its repo with a recent version.

reusable_build_docker workflow has been tested through a fake job in the PR ci on both arm and x86: https://github.com/falcosecurity/falco/actions/runs/4743816658
It was failing because {{ github.ref_name }} variable was not master nor a tag name, instead it was 2501/merge:

ERROR: invalid tag "falcosecurity/falco-no-driver:aarch64-2501/merge": invalid reference format

But you see it was failing with the same error at the same stage on both x86 and arm; therefore i expect it to work just fine on master.
See the commit with the code: dc37254

Does this PR introduce a user-facing change?:

new(ci): ported master and release artifacts publishing CI to gha

…es + docker images publish). Signed-off-by: Federico Di Pierro <nierro92@gmail.com>

Renamed `dev.yaml` to `master.yaml`. Signed-off-by: Federico Di Pierro <nierro92@gmail.com>

Signed-off-by: Federico Di Pierro <nierro92@gmail.com>

FedeDP · 2023-04-18T08:39:06Z

.circleci/config.yml

-          requires:
-            - "build-docker-dev"
-            - "build-docker-dev-arm64"
+      # - "rpm-sign":


Avoid conflicts; disable circleCI jobs.

FedeDP · 2023-04-18T08:39:23Z

.github/workflows/ci.yml

@@ -2,17 +2,71 @@ name: CI Build
 on:
  pull_request:
    branches: [master]
-  push:


We now have a dedicated master.yaml for that.

FedeDP · 2023-04-18T08:39:32Z

.github/workflows/ci.yml

 concurrency:
  group: ${{ github.head_ref || github.run_id }}
  cancel-in-progress: true  

 jobs:
+  test-arm:


This is just to test.

Will be removed before removing wip.

FedeDP · 2023-04-18T08:40:22Z

.github/workflows/master.yaml

+    branches: [master]
+
+# Checks if any concurrent jobs under the same pull request or branch are being executed
+concurrency:


This fixes #1876

.github/workflows/reusable_publish_packages.yaml

FedeDP · 2023-04-18T08:41:43Z

.github/workflows/reusable_publish_packages.yaml

+          ~/sign $RUNNER_TEMP/falco-*.rpm
+          rpm --qf %{SIGPGP:pgpsig} -qp $RUNNER_TEMP/falco-*.rpm | grep SHA256
+
+      - name: Publish rpm


To publish, we also need S3 credentials and access.

.github/workflows/master.yaml

FedeDP · 2023-04-18T08:46:52Z

Remaining steps have been added to PR body.

FedeDP · 2023-04-18T08:47:09Z

/milestone 0.35.0

FedeDP · 2023-04-18T09:01:22Z

.github/workflows/master.yaml

+      version: ${{ needs.build-dev-packages.outputs.version }}
+    secrets: inherit
+
+  build-dev-docker:


Both build-dev-docker and its arm64 counterpart require build-dev-packages because they use its output.

Same goes for the release CI.

FedeDP · 2023-04-18T09:48:27Z

Also, we definitely need more powerful gha self-hosted runners for arm (and more replicas).

FedeDP · 2023-04-18T10:23:29Z

Uh this time build-arm64 is much faster. Don't know what happened :/

.github/workflows/reusable_publish_packages.yaml

Signed-off-by: Federico Di Pierro <nierro92@gmail.com> Co-authored-by: Luca Guerra <luca@guerra.sh>

LucaGuerra · 2023-04-27T10:03:33Z

LGTM, I think we can test it on master :)

FedeDP · 2023-04-27T10:53:47Z

/hold

leogr

Couldn't try it in detail, but this is 🤩

I think we should give it a try, then fix any issues that will arise (as usual with the CI).

Andreagit97

/approve

poiana · 2023-04-27T12:23:39Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Andreagit97, FedeDP, leogr

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [Andreagit97,FedeDP,leogr]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

FedeDP · 2023-04-27T12:24:22Z

/unhold

FedeDP added 4 commits April 17, 2023 12:16

new(ci): added github action workflows for dev and release CI (packag…

9790887

…es + docker images publish). Signed-off-by: Federico Di Pierro <nierro92@gmail.com>

chore(ci): add proper concurrency groups for master and release CI.

3b7b5a5

Renamed `dev.yaml` to `master.yaml`. Signed-off-by: Federico Di Pierro <nierro92@gmail.com>

chore(ci): disable on push: master trigger for ci.yml.

f9333ef

Signed-off-by: Federico Di Pierro <nierro92@gmail.com>

chore(ci): commented out circleCI master and release jobs.

ca345ff

Signed-off-by: Federico Di Pierro <nierro92@gmail.com>

poiana added release-note do-not-merge/work-in-progress kind/feature dco-signoff: yes area/ci approved labels Apr 18, 2023

poiana requested review from Kaizhe and leogr April 18, 2023 08:38

poiana added the size/XXL label Apr 18, 2023

FedeDP commented Apr 18, 2023

View reviewed changes

.github/workflows/reusable_publish_packages.yaml Outdated Show resolved Hide resolved

FedeDP commented Apr 18, 2023

View reviewed changes

.github/workflows/master.yaml Outdated Show resolved Hide resolved

poiana added this to the 0.35.0 milestone Apr 18, 2023

FedeDP force-pushed the new/gha_dev_release_workflows branch from 2f4110e to b882b9d Compare April 18, 2023 08:48

FedeDP commented Apr 18, 2023

View reviewed changes

FedeDP force-pushed the new/gha_dev_release_workflows branch from 2cab6b7 to 3cdc306 Compare April 18, 2023 10:21

FedeDP force-pushed the new/gha_dev_release_workflows branch 2 times, most recently from 83bb5f0 to ae7256c Compare April 18, 2023 10:55

LucaGuerra reviewed Apr 27, 2023

View reviewed changes

.github/workflows/reusable_publish_packages.yaml Outdated Show resolved Hide resolved

chore(ci): improved GPG_KEY management through an env.

337b9e6

Signed-off-by: Federico Di Pierro <nierro92@gmail.com> Co-authored-by: Luca Guerra <luca@guerra.sh>

poiana added the do-not-merge/hold label Apr 27, 2023

leogr approved these changes Apr 27, 2023

View reviewed changes

poiana assigned leogr Apr 27, 2023

poiana added the lgtm label Apr 27, 2023

Andreagit97 approved these changes Apr 27, 2023

View reviewed changes

poiana assigned Andreagit97 Apr 27, 2023

poiana removed the do-not-merge/hold label Apr 27, 2023

poiana merged commit 86d33ae into master Apr 27, 2023
16 checks passed

poiana deleted the new/gha_dev_release_workflows branch April 27, 2023 12:25

LucaGuerra mentioned this pull request May 12, 2023

fix(ci): simplify and fix multi-arch image publishing process #2542

Merged

FedeDP mentioned this pull request May 12, 2023

fix(ci): correctly tag slim manifest #2545

Merged

LucaGuerra mentioned this pull request May 15, 2023

fix(ci): load falco image before building falco-driver-loader #2549

Merged

FedeDP mentioned this pull request May 19, 2023

fix(ci): use normal docker to build docker images, instead of buildx. #2556

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

new(ci): gha master and release workflows #2501

new(ci): gha master and release workflows #2501

FedeDP commented Apr 18, 2023 •

edited

FedeDP Apr 18, 2023

FedeDP Apr 18, 2023

FedeDP Apr 18, 2023

FedeDP Apr 18, 2023

FedeDP Apr 18, 2023

FedeDP Apr 18, 2023

FedeDP commented Apr 18, 2023 •

edited

FedeDP commented Apr 18, 2023

FedeDP Apr 18, 2023

FedeDP Apr 18, 2023

FedeDP commented Apr 18, 2023

FedeDP commented Apr 18, 2023

LucaGuerra commented Apr 27, 2023

FedeDP commented Apr 27, 2023

leogr left a comment

Andreagit97 left a comment

poiana commented Apr 27, 2023

FedeDP commented Apr 27, 2023

new(ci): gha master and release workflows #2501

new(ci): gha master and release workflows #2501

Conversation

FedeDP commented Apr 18, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

FedeDP commented Apr 18, 2023 • edited

FedeDP commented Apr 18, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

FedeDP commented Apr 18, 2023

FedeDP commented Apr 18, 2023

LucaGuerra commented Apr 27, 2023

FedeDP commented Apr 27, 2023

leogr left a comment

Choose a reason for hiding this comment

Andreagit97 left a comment

Choose a reason for hiding this comment

poiana commented Apr 27, 2023

FedeDP commented Apr 27, 2023

FedeDP commented Apr 18, 2023 •

edited

FedeDP commented Apr 18, 2023 •

edited