feat!: Docker CI - Security added & Tagging updates - 2 #1327

jrmanes · 2023-02-02T07:15:27Z

Overview

ℹ️ Same content as: 1320, I had an issue and the PR was closed...

Hello team,

Hope you’re doing well

The following PR contains some changes in the Docker CI, build & publish.

I’ve added some features:

File renamed: amd64-docker-build.yml -> docker-build-publish.yml
We will be able to use a matrix for building the container in different architectures.
This CI is triggered when
- When push to any branch
- When creating PRs
- When there's a new hotfix (when push to main branch)
- When push tags (v0.0.0/v0.0.0-alpha/v0.0.0-beta/v0.0.0-rcX)
- Only push new Docker images when they are in main or v* (blocking forks for security)
Security:
Trivy added to the CI, this is the first step during the build, it scans the image and provide us a table summary in case we have any CVE, if everything goes well, it continues with the next step.
It’s not going to block the CI in case of bugs, though I think it will be good for us, stop it and fix them.
Docker tagging:
- Git short SHA => for example: a594b69
- Tag ID => for example: 0.0.1
Provided some additional metadata to the images:
- Maintainer => "maintainer": "CelestiaOrg"
- Description => "CelestiaOrg repository celestiaorg/celestia-app"
- URL to the specific commit => "commit_url": "a594b69"
- Docker pull command => "docker_pull_command": "docker pull ghcr.io/celestiaorg/celestia-app:a594b691"

Checklist

Required CI checks are passing
Linked issues closed with keywords

Blockers

Hello team!
I'll need to add some permissions to allow the CI to have the access to the packages.

This is an error that I'm having: link

ERROR: failed to solve: failed to push ghcr.io/celestiaorg/celestia-app:pr-1320: unexpected status: 403 Forbidden
Error: buildx failed with: ERROR: failed to solve: failed to push ghcr.io/celestiaorg/celestia-app:pr-1320: unexpected status: 403 Forbidden

Thanks in advance!

cc: @evan-forbes @rootulp @Bidon15 @sysrex

Please, ping me when you'll going to merge it, just to check that everything goes fine 😊
Thank you team!

Closes Issue: 37

rach-id

👍 👍

Can you resolve the conflicts please?

.github/workflows/docker-build-publish.yml

.github/workflows/amd64-docker-build.yml

evan-forbes

buildx failed with: ERROR: failed to solve: failed to push ghcr.io/celestiaorg/celestia-app:pr-1327: unexpected status: 403 Forbidden

are we still seeing the issues that we saw with #1320 with the failed docker build CI?

besides fixing the linters, this LGTM

.github/workflows/docker-build-publish.yml

evan-forbes · 2023-02-02T17:46:04Z

@jrmanes suggested me trying this since PR since I write permissions, so I'm closing here for now to let me open up a PR from the same branch

jrmanes · 2023-02-06T11:04:31Z

hello @MSevey @Bidon15

I've added the changes that we discuss last Friday for fixing the issues with the forks and so far the CI looks better.

Split the build & publish
- build: always, it's ok
- publish: only when the branch is main or when releases v*

Let me know if you find anything that you would like to add :)

thanks in advance!

Bidon15 · 2023-02-06T11:16:30Z

publish: only when the branch is main or when releases v*

Does it mean that latest tag is always == to what is merged on main, right? @jrmanes
cc: @smuu

jrmanes · 2023-02-06T11:21:33Z

Yes, we'll build it alway, but only push the new images when the branch is main or v*, meaning that the tag latest will be updated during that process.
We have to test it when we'll merge it, but it should works in that way

jrmanes · 2023-02-06T14:37:28Z

Yes, we'll build it alway, but only push the new images when the branch is main or v*, meaning that the tag latest will be updated during that process. We have to test it when we'll merge it, but it should works in that way

Update:
After discuss it:

We postpone the latest topic for future

.github/workflows/docker-build-publish.yml

evan-forbes

there are a lot of commits, what was the thing that allowed this to work now 😅 ?

evan-forbes · 2023-02-09T13:14:43Z

.github/workflows/docker-build-publish.yml

+          # refer to it later
+          tags: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:local
+
+      - name: Run Trivy vulnerability scanner


can we run this locally if it fails?

https://github.com/aquasecurity/trivy#installation

@evan-forbes

## Overview ℹ️ Same content as: [1320](#1320), I had an issue and the PR was closed... Hello team, Hope you’re doing well The following PR contains some changes in the Docker CI, build & publish. I’ve added some features: - File renamed: `amd64-docker-build.yml` -> `docker-build-publish.yml` *We will be able to use a matrix for building the container in different architectures.* - This CI is triggered when - When push to any branch - When creating `PRs` - When there's a new hotfix (when push to main branch) - When push tags `(v0.0.0/v0.0.0-alpha/v0.0.0-beta/v0.0.0-rcX)` - Only push new Docker images when they are in `main` or `v*` (blocking forks for security) - Security: [Trivy](https://www.aquasec.com/products/trivy/) added to the CI, this is the first step during the build, it scans the image and provide us a table summary in case we have any CVE, if everything goes well, it continues with the next step. *It’s not going to block the CI in case of bugs, though I think it will be good for us, stop it and fix them.* - Docker tagging: - Git short SHA => for example: `a594b69` - Tag ID => for example: `0.0.1` - Provided some additional metadata to the images: - Maintainer => "maintainer": "CelestiaOrg" - Description => "CelestiaOrg repository celestiaorg/celestia-app" - URL to the specific commit => "commit_url": "a594b69" - Docker pull command => "docker_pull_command": "docker pull ghcr.io/celestiaorg/celestia-app:a594b691" --- ## Checklist - [x] Required CI checks are passing - [x] Linked issues closed with keywords --- ## Blockers Hello team! I'll need to add some permissions to allow the CI to have the access to the packages. This is an error that I'm having: [link](https://github.com/celestiaorg/celestia-app/actions/runs/4058197292/jobs/6984898432) ``` ERROR: failed to solve: failed to push ghcr.io/celestiaorg/celestia-app:pr-1320: unexpected status: 403 Forbidden Error: buildx failed with: ERROR: failed to solve: failed to push ghcr.io/celestiaorg/celestia-app:pr-1320: unexpected status: 403 Forbidden ``` Thanks in advance! cc: @evan-forbes @rootulp @Bidon15 @sysrex Please, ping me when you'll going to merge it, just to check that everything goes fine 😊 Thank you team! --- Closes Issue: [37](celestiaorg/devops#37)

@evan-forbes

…1327) ## Overview ℹ️ Same content as: [1320](celestiaorg#1320), I had an issue and the PR was closed... Hello team, Hope you’re doing well The following PR contains some changes in the Docker CI, build & publish. I’ve added some features: - File renamed: `amd64-docker-build.yml` -> `docker-build-publish.yml` *We will be able to use a matrix for building the container in different architectures.* - This CI is triggered when - When push to any branch - When creating `PRs` - When there's a new hotfix (when push to main branch) - When push tags `(v0.0.0/v0.0.0-alpha/v0.0.0-beta/v0.0.0-rcX)` - Only push new Docker images when they are in `main` or `v*` (blocking forks for security) - Security: [Trivy](https://www.aquasec.com/products/trivy/) added to the CI, this is the first step during the build, it scans the image and provide us a table summary in case we have any CVE, if everything goes well, it continues with the next step. *It’s not going to block the CI in case of bugs, though I think it will be good for us, stop it and fix them.* - Docker tagging: - Git short SHA => for example: `a594b69` - Tag ID => for example: `0.0.1` - Provided some additional metadata to the images: - Maintainer => "maintainer": "CelestiaOrg" - Description => "CelestiaOrg repository celestiaorg/celestia-app" - URL to the specific commit => "commit_url": "celestiaorg@a594b69" - Docker pull command => "docker_pull_command": "docker pull ghcr.io/celestiaorg/celestia-app:a594b691" --- ## Checklist - [x] Required CI checks are passing - [x] Linked issues closed with keywords --- ## Blockers Hello team! I'll need to add some permissions to allow the CI to have the access to the packages. This is an error that I'm having: [link](https://github.com/celestiaorg/celestia-app/actions/runs/4058197292/jobs/6984898432) ``` ERROR: failed to solve: failed to push ghcr.io/celestiaorg/celestia-app:pr-1320: unexpected status: 403 Forbidden Error: buildx failed with: ERROR: failed to solve: failed to push ghcr.io/celestiaorg/celestia-app:pr-1320: unexpected status: 403 Forbidden ``` Thanks in advance! cc: @evan-forbes @rootulp @Bidon15 @sysrex Please, ping me when you'll going to merge it, just to check that everything goes fine 😊 Thank you team! --- Closes Issue: [37](celestiaorg/devops#37) (cherry picked from commit 67c90ed)

@evan-forbes

## Overview ℹ️ Same content as: [1320](#1320), I had an issue and the PR was closed... Hello team, Hope you’re doing well The following PR contains some changes in the Docker CI, build & publish. I’ve added some features: - File renamed: `amd64-docker-build.yml` -> `docker-build-publish.yml` *We will be able to use a matrix for building the container in different architectures.* - This CI is triggered when - When push to any branch - When creating `PRs` - When there's a new hotfix (when push to main branch) - When push tags `(v0.0.0/v0.0.0-alpha/v0.0.0-beta/v0.0.0-rcX)` - Only push new Docker images when they are in `main` or `v*` (blocking forks for security) - Security: [Trivy](https://www.aquasec.com/products/trivy/) added to the CI, this is the first step during the build, it scans the image and provide us a table summary in case we have any CVE, if everything goes well, it continues with the next step. *It’s not going to block the CI in case of bugs, though I think it will be good for us, stop it and fix them.* - Docker tagging: - Git short SHA => for example: `a594b69` - Tag ID => for example: `0.0.1` - Provided some additional metadata to the images: - Maintainer => "maintainer": "CelestiaOrg" - Description => "CelestiaOrg repository celestiaorg/celestia-app" - URL to the specific commit => "commit_url": "a594b69" - Docker pull command => "docker_pull_command": "docker pull ghcr.io/celestiaorg/celestia-app:a594b691" --- ## Checklist - [x] Required CI checks are passing - [x] Linked issues closed with keywords --- ## Blockers Hello team! I'll need to add some permissions to allow the CI to have the access to the packages. This is an error that I'm having: [link](https://github.com/celestiaorg/celestia-app/actions/runs/4058197292/jobs/6984898432) ``` ERROR: failed to solve: failed to push ghcr.io/celestiaorg/celestia-app:pr-1320: unexpected status: 403 Forbidden Error: buildx failed with: ERROR: failed to solve: failed to push ghcr.io/celestiaorg/celestia-app:pr-1320: unexpected status: 403 Forbidden ``` Thanks in advance! cc: @evan-forbes @rootulp @Bidon15 @sysrex Please, ping me when you'll going to merge it, just to check that everything goes fine 😊 Thank you team! --- Closes Issue: [37](celestiaorg/devops#37) (cherry picked from commit 67c90ed)

jrmanes added app CI item that directly relates to or modify the CI warn:blocked item is not currently being worked on but is still blocked labels Feb 2, 2023

jrmanes requested review from Bidon15 and rach-id February 2, 2023 07:23

jrmanes marked this pull request as ready for review February 2, 2023 07:25

jrmanes requested a review from evan-forbes as a code owner February 2, 2023 07:25

MSevey requested a review from a team February 2, 2023 07:30

jrmanes mentioned this pull request Feb 2, 2023

feat!: Docker CI - Security added & Tagging updates #1320

Closed

3 tasks

rach-id reviewed Feb 2, 2023

View reviewed changes

.github/workflows/docker-build-publish.yml Show resolved Hide resolved

jrmanes force-pushed the jose/docker-ci-images branch from 706910c to 2339ae7 Compare February 2, 2023 11:53

MSevey requested a review from a team February 2, 2023 11:53

Bidon15 reviewed Feb 2, 2023

View reviewed changes

.github/workflows/amd64-docker-build.yml Outdated Show resolved Hide resolved

evan-forbes assigned jrmanes Feb 2, 2023

evan-forbes requested changes Feb 2, 2023

View reviewed changes

MSevey requested a review from a team February 2, 2023 13:52

MSevey reviewed Feb 2, 2023

View reviewed changes

.github/workflows/docker-build-publish.yml Outdated Show resolved Hide resolved

evan-forbes closed this Feb 2, 2023

evan-forbes mentioned this pull request Feb 2, 2023

feat!: Docker CI - Security added & Tagging updates #1332

Closed

5 tasks

evan-forbes reopened this Feb 2, 2023

MSevey requested a review from a team February 2, 2023 18:07

jrmanes requested a review from evan-forbes February 6, 2023 14:30

jrmanes requested review from MSevey and Bidon15 February 6, 2023 14:42

jrmanes added 15 commits February 9, 2023 08:13

use the main-docker-build format

c4ff8c2

trigger old one when PRs

7014b33

undo main-docker-build

b008241

use action to bulid the container

ac387a4

disable yamllint for tags

8330d46

undo current build - pull_request not needed

d162c52

feat: split build & publish - only push when branch is main or v*

470da1b

fix: yammlint - length lines

794d7ae

feat: specify the Dockerfile path - remove latest

968b366

fix: revert yamlllint

19311bf

fix: update besed on the comments

4a20252

add v to version (output v1.2.3) - test in push

5d3e0ce

fix: solve length rule (pending to increase it to 120)

866712f

fix: Cleaning up not needed block

940dc48

feat: Commented line

1da23e8

jrmanes force-pushed the jose/docker-ci-images branch from 70512d2 to 1da23e8 Compare February 9, 2023 07:13

MSevey requested a review from a team February 9, 2023 07:14

Bidon15 mentioned this pull request Feb 9, 2023

docker/tagging: Strategy options with future docker namings celestiaorg/devops#55

Open

smuu reviewed Feb 9, 2023

View reviewed changes

.github/workflows/docker-build-publish.yml Outdated Show resolved Hide resolved

fix: Add prefix v

9bda076

jrmanes dismissed stale reviews from MSevey, smuu, and Bidon15 via 9bda076 February 9, 2023 12:02

Bidon15 approved these changes Feb 9, 2023

View reviewed changes

smuu approved these changes Feb 9, 2023

View reviewed changes

evan-forbes approved these changes Feb 9, 2023

View reviewed changes

evan-forbes merged commit 67c90ed into celestiaorg:main Feb 9, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat!: Docker CI - Security added & Tagging updates - 2 #1327

feat!: Docker CI - Security added & Tagging updates - 2 #1327

jrmanes commented Feb 2, 2023 •

edited

rach-id left a comment

evan-forbes left a comment

evan-forbes commented Feb 2, 2023

jrmanes commented Feb 6, 2023

Bidon15 commented Feb 6, 2023

jrmanes commented Feb 6, 2023

jrmanes commented Feb 6, 2023

evan-forbes left a comment

evan-forbes Feb 9, 2023

smuu Feb 9, 2023

feat!: Docker CI - Security added & Tagging updates - 2 #1327

feat!: Docker CI - Security added & Tagging updates - 2 #1327

Conversation

jrmanes commented Feb 2, 2023 • edited

Overview

Checklist

Blockers

rach-id left a comment

Choose a reason for hiding this comment

evan-forbes left a comment

Choose a reason for hiding this comment

evan-forbes commented Feb 2, 2023

jrmanes commented Feb 6, 2023

Bidon15 commented Feb 6, 2023

jrmanes commented Feb 6, 2023

jrmanes commented Feb 6, 2023

evan-forbes left a comment

Choose a reason for hiding this comment

evan-forbes Feb 9, 2023

Choose a reason for hiding this comment

smuu Feb 9, 2023

Choose a reason for hiding this comment

jrmanes commented Feb 2, 2023 •

edited