Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docker-based packaging workflow in GitHub Actions. #9964

Merged
merged 4 commits into from Mar 15, 2021

Conversation

Ferroin
Copy link
Member

@Ferroin Ferroin commented Sep 21, 2020

Summary

This implements a new Docker-based packaging CI workflow in GitHub Actions. It still uses the same trigger commits as the existing LXC-based workflow we have in Travis so that minimal changes are needed to have it work correctly.

This requires the addition of two secrets to the repository itself:

  • PACKAGE_CLOUD_REPO: This specifies which PackageCloud repository to push the packages to.
  • PACKAGE_CLOUD_API_KEY: This specifies the PackageCloud API key to use when uploading packages.

This leverages GitHub Actions' matrix build feature to keep overall code in the workflow file to a minimum.

Component Name

area/ci
area/packaging

Additional Information

Fixes: #7346
Fixes: #7822

@Ferroin
Copy link
Member Author

Ferroin commented Mar 9, 2021

OK, this is finally ready for a round of preliminary review. It still is not quite ready to be merged (we still need more testing of the upload handling, as well as needing to move the repo cleanup process from Travis to GHA), but the code that is here is unlikely to change significantly unless we run into serious complications.

@Ferroin
Copy link
Member Author

Ferroin commented Mar 9, 2021

For those who want to verify the packages that would be produced by this locally, here are the steps required to use the same package building process locally:

  1. Pull the correct Docker image for your distribution and architecture from the netdata/package-builders repository on Docker Hub. There is one tag there for each supported distribution.
  2. From your local copy of the Netdata sources, run: docker run -it --rm -v $PWD:/netdata netdata/package-builders:<tag>, replacing <tag> with the tag for your distribution (for example debian10 for a Debian 10 package).

This will create an artifacts directory in the Netdata source tree on the host system which will contain the package files, which you can then manually install on your system by passing them directly to your package manager. The generated packages will list a version of 0.1 instead of the current version, but are otherwise the current version of the code provided by the Netdata sources used for the build.

The package building process itself takes about 5-10 minutes on a reasonably fast system.

Note that we currently only build for 64-bit x86 on most of these platforms (and the handful we don’t only include 32-bit x86 beyond that).

knatsakis
knatsakis previously approved these changes Mar 9, 2021
kaskavel
kaskavel previously approved these changes Mar 10, 2021
knatsakis
knatsakis previously approved these changes Mar 11, 2021
@netdata-community-bot
Copy link

This pull request has been mentioned on Netdata Community. There might be relevant details there:

https://community.netdata.cloud/t/centos-8-rpms-missing-from-cloudpackage/1077/2

This greatly simplifies the package handling code, significantly reduces
the amount of things we are running in Travis, adds better CI for package
builds for PRs, finally eliminates LXC from any of our CI, and enables
us to make further major improvements much easier.
@Ferroin Ferroin merged commit e54982b into netdata:master Mar 15, 2021
@Ferroin Ferroin deleted the gha-packaging branch March 15, 2021 12:33
MrZammler pushed a commit to MrZammler/netdata that referenced this pull request Mar 16, 2021
* Add a new GHA workflow for building and publishing binary packages.

This greatly simplifies the package handling code, significantly reduces
the amount of things we are running in Travis, adds better CI for package
builds for PRs, finally eliminates LXC from any of our CI, and enables
us to make further major improvements much easier.

* Migrate repo cleanup to GHA and remove dead code.

* Fix RPM package builds.

* Fix DEB installation.
Saruspete pushed a commit to Saruspete/netdata that referenced this pull request Mar 17, 2021
* Add a new GHA workflow for building and publishing binary packages.

This greatly simplifies the package handling code, significantly reduces
the amount of things we are running in Travis, adds better CI for package
builds for PRs, finally eliminates LXC from any of our CI, and enables
us to make further major improvements much easier.

* Migrate repo cleanup to GHA and remove dead code.

* Fix RPM package builds.

* Fix DEB installation.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/ci area/packaging Packaging and operating systems support
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Refactor scripts that publish to packagecloud for nightlies Eliminate LXC from package generation pipeline
4 participants