Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

publish packages to S3 (download.falco.org) #1577

Merged
merged 9 commits into from
Mar 26, 2021
Merged

Conversation

leogr
Copy link
Member

@leogr leogr commented Mar 11, 2021

What type of PR is this?

/kind feature

Any specific area of the project related to this PR?

/area build

What this PR does / why we need it:

This PR introduces scripts to deal with updates of Debian and RPM repositories and modify CI jobs to publish packages (ie. bin-dev, deb-dev, rpm-dev, bin, deb, rpm) to our S3 bucket which serves those files at https://download.falco.org/

Which issue(s) this PR fixes:

Fixes #1567

Special notes for your reviewer:

If this works as expected, a follow-up PR will update the configuration to publish stable packages too (ie. bin, deb, rpm) in the same way.
I'm confident that scripts work as expected, so I decided to include the stable packages too in this PR. A follow-up PR might come in case of unexpected issues.

The artifact cleanup job has been disabled since it is intended to work with Bintray only. If needed, we can refactor or clean up it in a follow-up PR.

Does this PR introduce a user-facing change?:

build: now Falco packages are published at https://download.falco.org
BREAKING CHANGE:  Bintray is deprecated, no new packages will be published at https://dl.bintray.com/falcosecurity/

@leogr
Copy link
Member Author

leogr commented Mar 11, 2021

Update:

I did some tests in a simulated environment:

publish-deb is still not tested, wip.
cicleci config is untested too, ofc.

Early feedback is welcome.

/cc @leodido
/cc @fntlnz

@poiana poiana requested review from fntlnz and leodido March 11, 2021 17:00
Copy link
Contributor

@fntlnz fntlnz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good job leo! Just a quick note while this is still wip

.circleci/config.yml Outdated Show resolved Hide resolved
@leogr
Copy link
Member Author

leogr commented Mar 12, 2021

Update:
I have added AWS_CLOUDFRONT_DIST_ID to the test-infra context on CircleCI.
It's required for the CloudFront cache invalidation.

@leogr
Copy link
Member Author

leogr commented Mar 12, 2021

publish-deb worked, results 👉 https://download.falco.org/?prefix=packages/deb-dev/

Signed-off-by: Leonardo Grasso <me@leonardograsso.com>
Signed-off-by: Leonardo Grasso <me@leonardograsso.com>
Signed-off-by: Leonardo Grasso <me@leonardograsso.com>
Signed-off-by: Leonardo Grasso <me@leonardograsso.com>
Signed-off-by: Leonardo Grasso <me@leonardograsso.com>
Signed-off-by: Leonardo Grasso <me@leonardograsso.com>
Signed-off-by: Leonardo Grasso <me@leonardograsso.com>
@leogr leogr added this to the 0.28.0 milestone Mar 15, 2021
Signed-off-by: Leonardo Grasso <me@leonardograsso.com>
@leogr leogr changed the title wip: publish dev packages pkg to S3 (download.falco.org) publish dev packages pkg to S3 (download.falco.org) Mar 16, 2021
@poiana
Copy link

poiana commented Mar 25, 2021

LGTM label has been added.

Git tree hash: d6034ccca6f5247e0178feb667e58dd36c18a896

jfrog bt u /build/release/falco-${FALCO_VERSION}-x86_64.deb falcosecurity/deb-dev/falco/${FALCO_VERSION} stable/ --deb stable/main/amd64 --user poiana --key ${BINTRAY_SECRET} --publish --override
yum update -y
yum install createrepo gpg -y
echo $GPG_KEY | base64 -d | gpg --import
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's remember in the future (when we'll move these jobs to ProwJobs) that echo -e $GPG_KEY works ok (no need to base64 encode/decode).

README.md Show resolved Hide resolved
README.md Show resolved Hide resolved
README.md Show resolved Hide resolved
README.md Show resolved Hide resolved
README.md Show resolved Hide resolved
| rpm | [![rpm-dev](https://img.shields.io/bintray/v/falcosecurity/rpm-dev/falco?label=Falco&color=%2300aec7&style=flat-square)][1] | [![rpm](https://img.shields.io/bintray/v/falcosecurity/rpm/falco?label=Falco&color=%23005763&style=flat-square)][2] |
| deb | [![deb-dev](https://img.shields.io/bintray/v/falcosecurity/deb-dev/falco?label=Falco&color=%2300aec7&style=flat-square)][3] | [![deb](https://img.shields.io/bintray/v/falcosecurity/deb/falco?label=Falco&color=%23005763&style=flat-square)][4] |
| binary | [![bin-dev](https://img.shields.io/bintray/v/falcosecurity/bin-dev/falco?label=Falco&color=%2300aec7&style=flat-square)][5] | [![bin](https://img.shields.io/bintray/v/falcosecurity/bin/falco?label=Falco&color=%23005763&style=flat-square)][6] |
| rpm | [![rpm-dev](https://img.shields.io/badge/dynamic/xml?color=%2300aec7&style=flat-square&label=Falco&query=substring-before%28substring-after%28%28%2F%2A%5Bname%28%29%3D%27ListBucketResult%27%5D%2F%2A%5Bname%28%29%3D%27Contents%27%5D%29%5Blast%28%29%5D%2F%2A%5Bname%28%29%3D%27Key%27%5D%2C%22falco-%22%29%2C%22.asc%22%29&url=https%3A%2F%2Ffalco-distribution.s3-eu-west-1.amazonaws.com%2F%3Fprefix%3Dpackages%2Frpm-dev%2Ffalco-)][1] | [![rpm](https://img.shields.io/badge/dynamic/xml?color=%2300aec7&style=flat-square&label=Falco&query=substring-before%28substring-after%28%28%2F%2A%5Bname%28%29%3D%27ListBucketResult%27%5D%2F%2A%5Bname%28%29%3D%27Contents%27%5D%29%5Blast%28%29%5D%2F%2A%5Bname%28%29%3D%27Key%27%5D%2C%22falco-%22%29%2C%22.asc%22%29&url=https%3A%2F%2Ffalco-distribution.s3-eu-west-1.amazonaws.com%2F%3Fprefix%3Dpackages%2Frpm%2Ffalco-)][2] |
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great job :D

README.md Show resolved Hide resolved
Copy link
Member

@leodido leodido left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

There are some cleanup things but we'll fix them in a follow-up PR, nothing top priority.

@poiana
Copy link

poiana commented Mar 26, 2021

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: fntlnz, leodido

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@poiana poiana merged commit 40edfe6 into master Mar 26, 2021
@poiana poiana deleted the build/publish-pkg-on-s3 branch March 26, 2021 11:55
@leogr leogr mentioned this pull request Mar 26, 2021
leogr added a commit that referenced this pull request Mar 26, 2021
Applying suggestions from
#1577

Co-authored-by: Leonardo Di Donato <leodidonato@gmail.com>
Signed-off-by: Leonardo Grasso <me@leonardograsso.com>
leogr added a commit that referenced this pull request Mar 26, 2021
Applying suggestions from #1577

Co-authored-by: Leonardo Di Donato <leodidonato@gmail.com>
Signed-off-by: Leonardo Grasso <me@leonardograsso.com>
leogr added a commit that referenced this pull request Mar 29, 2021
Applying suggestions from
#1577

Co-authored-by: Leonardo Di Donato <leodidonato@gmail.com>
Signed-off-by: Leonardo Grasso <me@leonardograsso.com>
leogr added a commit that referenced this pull request Mar 29, 2021
Applying suggestions from #1577

Co-authored-by: Leonardo Di Donato <leodidonato@gmail.com>
Signed-off-by: Leonardo Grasso <me@leonardograsso.com>
@leogr leogr mentioned this pull request Mar 29, 2021
poiana pushed a commit that referenced this pull request Apr 7, 2021
Applying suggestions from
#1577

Co-authored-by: Leonardo Di Donato <leodidonato@gmail.com>
Signed-off-by: Leonardo Grasso <me@leonardograsso.com>
poiana pushed a commit that referenced this pull request Apr 7, 2021
Applying suggestions from #1577

Co-authored-by: Leonardo Di Donato <leodidonato@gmail.com>
Signed-off-by: Leonardo Grasso <me@leonardograsso.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Switch from Bintray to download.falco.org
4 participants