From 5bff6168f6b0dce2f77383104e4fdf68ff6aa757 Mon Sep 17 00:00:00 2001 From: Carlos Alexandro Becker Date: Tue, 16 Jan 2024 08:33:16 -0300 Subject: [PATCH] docs(blog): import more posts (#4544) refs #3503 importing using `mediumexporter` and fixing up manually what's wrong. --------- Signed-off-by: Carlos Alexandro Becker --- www/docs/blog/.authors.yml | 9 +- .../posts/2022-03-26-reproducible-builds.md | 66 ++++ .../blog/posts/2022-03-28-supply-chain-sec.md | 189 +++++++++++ .../blog/posts/2022-05-22-nightly-actions.md | 98 ++++++ .../blog/posts/2023-01-10-azure-devops.md | 315 ++++++++++++++++++ .../blog/posts/2023-01-30-goreleaser-ko.md | 191 +++++++++++ 6 files changed, 867 insertions(+), 1 deletion(-) create mode 100644 www/docs/blog/posts/2022-03-26-reproducible-builds.md create mode 100644 www/docs/blog/posts/2022-03-28-supply-chain-sec.md create mode 100644 www/docs/blog/posts/2022-05-22-nightly-actions.md create mode 100644 www/docs/blog/posts/2023-01-10-azure-devops.md create mode 100644 www/docs/blog/posts/2023-01-30-goreleaser-ko.md diff --git a/www/docs/blog/.authors.yml b/www/docs/blog/.authors.yml index abda921df6a..f7bbc03c38b 100644 --- a/www/docs/blog/.authors.yml +++ b/www/docs/blog/.authors.yml @@ -11,4 +11,11 @@ authors: name: Furkan Turkal description: Author avatar: https://avatars.githubusercontent.com/u/16493751 - + dirien: + name: Engin Diri + description: Author + avatar: https://avatars.githubusercontent.com/u/38325136 + hariso: + name: Haris Osmanagić + description: Author + avatar: https://avatars.githubusercontent.com/u/2166300 diff --git a/www/docs/blog/posts/2022-03-26-reproducible-builds.md b/www/docs/blog/posts/2022-03-26-reproducible-builds.md new file mode 100644 index 00000000000..a844ff06962 --- /dev/null +++ b/www/docs/blog/posts/2022-03-26-reproducible-builds.md @@ -0,0 +1,66 @@ +--- +date: 2022-03-26 +slug: reproducible-builds +categories: + - tutorials +authors: + - caarlos0 +--- + +GoReleaser can help you, to some extent, to have reproducible builds. + +![](https://carlosbecker.com/posts/goreleaser-reproducible-buids/c4824165-c6e2-40df-b4b5-8abe443195ce.png) + + + +## **What are reproducible builds?** + +According to [Reproducible-Builds.org](https://reproducible-builds.org/docs/definition/): + +> A build is reproducible if given the same source code, build environment and build instructions, any party can recreate bit-by-bit identical copies of all specified artifacts. + +So, things we need to pay attention here are: + +- the source is the same +- the dependencies are the same, in the same versions +- `chtimes` are the same +- build path is the same +- any other tools needed to compile must be the same, in the same versions + +While this might sound complicated, rest assured GoReleaser can help you with most of these items! + +## **Reproducible Builds with GoReleaser** + +GoReleaser will by default inject a `ldflag` with the current timestamp as `main.date`, which you can use to display build time information. We will want to change that to use some fixed date, for instance, the date of the commit being built. + +To avoid changes from one machine to another, we will also want to use `-trimpath`. + +Finally, we'll want to make sure the repo code haven't changed, e.g., when building a tag, we want to make sure it wasn't deleted and pushed again (i.e., moved). + +We can achieve that with a config that looks like this: + +```yaml +builds: + - env: + - CGO_ENABLED=0 + goos: + - linux + - darwin + goarch: + - amd64 + - arm64 + mod_timestamp: "{{ .CommitTimestamp }}" + flags: + - -trimpath + ldflags: + - -s -w -X main.version={{.Version}} -X main.commit={{.Commit}} -X main.date={{ .CommitDate }} + +gomod: + proxy: true +``` + +From now on, we basically only need to ensure the Go version is the same. + +That's out of the scope of GoReleaser's scope, but easy enough to do in GitHub Actions by pinning to a specific version of Go. + +So, there you have it: reproducible Go binary builds with GoReleaser! diff --git a/www/docs/blog/posts/2022-03-28-supply-chain-sec.md b/www/docs/blog/posts/2022-03-28-supply-chain-sec.md new file mode 100644 index 00000000000..5943a99dd34 --- /dev/null +++ b/www/docs/blog/posts/2022-03-28-supply-chain-sec.md @@ -0,0 +1,189 @@ +--- +date: 2022-03-28 +slug: supply-chain-security +categories: + - tutorials +authors: + - developerguy +--- + +# GoReleaser And Software Supply Chain Security + +Before talking about the security of the software supply chains, we should mention what should come to our minds first when we are talking about software supply chains. +In most basic terms, you can think of **software supply chains are anything that's needed to deliver your product — including all the components you use**, for example, **your codebase**, **packages**, ** libs, your CI/CD pipeline**, **third-party services you use**, **anything that goes into or affects your code from development to gets deployed into production systems.** + + + +![[https://security.googleblog.com/2021/06/introducing-slsa-end-to-end-framework.html](https://security.googleblog.com/2021/06/introducing-slsa-end-to-end-framework.html)](https://cdn-images-1.medium.com/max/2000/1*fYWJfKAY5tdAvTMJnbsQPQ.png)[https://security.googleblog.com/2021/06/introducing-slsa-end-to-end-framework.html](https://security.googleblog.com/2021/06/introducing-slsa-end-to-end-framework.html) + +The picture above, taken from [SLSA (Supply Chain Levels for Software +Artifacts)](https://slsa.dev), **is a security framework**, **a check-list of +standards and controls** to **prevent tampering**, **improve the integrity**, +and secure packages and infrastructure in your projects, businesses, or enterprises, shows us the importance of protecting our workflows that delivers software we built to the customer because there are many places that attackers can attack and gain access to our system. + +Unfortunately, the **new threats** in software development are not only related to the specific company itself. Thanks to [CNCF Security Technical Advisory Group](https://github.com/cncf/tag-security), they made a repository to list all the companies' compromises against supply chain attacks. Trends show that [supply chain attacks are increasing](https://www.sonatype.com/hubfs/Q3%202021-State%20of%20the%20Software%20Supply%20Chain-Report/SSSC-Report-2021_0913_PM_2.pdf?hsLang=en-us) at an **exponential rate of 4–5x per year**, with several thousand last year, the most common being related to dependency confusion or [typosquatting](https://sysdig.com/blog/malicious-python-libraries-jeilyfish-dateutil/), followed by **malicious source code injection**, so, with the rise of software supply chain attacks, it becomes more critical to secure our software supply chains. + +![[https://www.memesmonkey.com/images/memesmonkey/43/43b3e5ab9f90a6266a163278c025cba5.jpeg](https://www.memesmonkey.com/images/memesmonkey/43/43b3e5ab9f90a6266a163278c025cba5.jpeg)](https://cdn-images-1.medium.com/max/2000/1*tqXzGy9XAM90BUN7cilkiw.jpeg)[https://www.memesmonkey.com/images/memesmonkey/43/43b3e5ab9f90a6266a163278c025cba5.jpeg](https://www.memesmonkey.com/images/memesmonkey/43/43b3e5ab9f90a6266a163278c025cba5.jpeg) + +Securing the software supply chain is not an easy task. +So many people and organizations have already working on this specific problem, but luckily we have some tools to make that process a bit easier to deal with for us. + +Today, we'll be talking about the GoReleaser project and its features that can help us mitigate the risk of compromises in software supply chains. Then, at the end of this guide, we'll be demoing everything we talked about to give more practical information about how you can make the supply chain more secure for your Go projects by using GoReleaser. + +[GoReleaser](https://goreleaser.com/intro/) is a release automation tool for Go projects. +The goal is to simplify the build, release, and publish steps while providing variant customization options for all steps. **Although GoReleaser is built with CI tools in mind**, you only need to download and execute it in your build script. But, of course, you can still install it locally if you want. This is because many projects have already been using GoReleaser for a long time in the open-source world. Also, with the announcement [GitHub Actions](https://github.com/features/actions) platform, GoReleaser's popularity increased, so it takes firm steps towards becoming a defacto standard for releasing Go projects, especially in the GitHub ecosystem. + +**The first step in securing your supply chain is to create an inventory of the software and libraries being used during your build and deployment cycle.** + +This is where an [SBOM](https://www.linuxfoundation.org/blog/what-is-an-sbom/) comes into the picture. + +**More technically**, An SBOM is a structured list of components, modules, and libraries that are included in a given piece of software. However, there are many different meanings of an SBOM in the ecosystem. In the most basic form, think of them as a list of ingredients that evolves throughout the software development lifecycle as you add new code or components. Knowing about our software's dependencies, we can first determine if we are impacted when a new security vulnerability is found. If so, we can apply security patches to mitigate the risk of that vulnerability before affecting our software. + +Organizations can and do create and publish a software bill of materials in several different formats. In addition to these common formats, several methods are explicitly designed for delivering SBOMs, including [SPDX (Software Package Data Exchange)](https://spdx.dev), [Software Identification (SWID) Tags](https://csrc.nist.gov/projects/Software-Identification-SWID), and [Cyclone DX](https://cyclonedx.org), many open-source tools exist, such as [Syft](https://github.com/anchore/syft) from [Anchore](https://anchore.com), which is what we are going to talk about in this blog post, [kubernetes-sigs/bom](https://github.com/kubernetes-sigs/bom) from [Kubernetes SIG Release](https://github.com/kubernetes/sig-release), [cyclonedx-cli](https://github.com/CycloneDX/cyclonedx-cli) from [CycloneDX](https://cyclonedx.org), [spdx-sbom-generator](https://github.com/opensbom-generator/spdx-sbom-generator), [Tern](https://github.com/tern-tools/tern), and many more... + +Let's look at what we have to do successfully to generate an SBOM in GoReleaser using Syft. We'll be setting up the demo on the GitHub Actions platform so that some examples might be specific to that platform. + +> 🚨 **TLDR**; you can find all the source code what we are going to show you as an example on GitHub [here](https://github.com/goreleaser/supply-chain-example). + +**Since GoReleaser uses Syft by calling it's binary**, we ensure that Syft binary exists before running GoReleaser. We can download the binary and move it to the executable's path while running the job, but there is a better way of doing this, [anchore/sbom-action](https://github.com/anchore/sbom-action). A sbom-action is a GitHub Action for creating a software bill of materials using Syft, but we can use its sub-action called [anchore/sbom-action/download-syft](https://github.com/anchore/sbom-action/blob/main/download-syft/action.yml) to download the executable only. + +To install Syft, you need to add the following line to our GitHub Action workflow. + +```yaml +- uses: anchore/sbom-action/download-syft@v0.7.0 # installs syft +``` + +Next, we need to add a [setting](https://goreleaser.com/customization/sbom/) +specific to configure SBOM generation to the GoReleaser configuration file +`.goreleaser.yml`. +I said configure because GoReleaser's SBOM generation support is highly configurable. +After all, it accepts commands to be run to generate an SBOM and arguments that will pass to the command, which makes GoReleaser can work with any SBOM generation tool, as we mentioned earlier, seamlessly. + +```yaml +# creates SBOMs of all archives and the source tarball using syft +# https://goreleaser.com/customization/sbom +# Two different sbom configurations need two different IDs +sboms: + - id: archive + artifacts: archive + - id: source + artifacts: source +``` + +When you do not specify any command, it will use syftas a command by default, as you can see [here](https://github.com/goreleaser/goreleaser/blob/7671dab291483b2733e871abff379d07e74dfc6c/internal/pipe/sbom/sbom.go#L64-L73). + +GoReleaser lets you cross-compile your Go binaries and package them in various formats, including container images or tarballs. +Then, using some public services such as GitHub Releases and DockerHub to distribute them to customers or production systems +With the rise of software supply chain attacks, ensuring the integrity between the artifacts (container images, blobs, etc.) we produce and consume becomes more critical. +**Integrity** means ensuring an artifact is what it says it is. +It also indicates the artifact we consume has not been tampered with since it was created and comes from a valid source, a trusted identity (a public key, a person, a company, or some other entity). +Leveraging this workflow gives your users confidence that the container images from their container registry were the trusted code you built and published. +One of the best ways of checking the integrity of an artifact and ensuring that if the artifact has been tampered with is to use a utility called [cosign](https://github.com/sigstore/cosign) from [Sigstore](https://sigstore.dev). + +Sigstore is an open-source security project now sponsored by the [OpenSSF](https://openssf.org), **the Open Software Security Foundation**, allowing developers to build, distribute, and verify signed software artifacts securely. +**Sigstore** provides a cosign tool, enabling developers to build, distribute, +and verify signed software artifacts securely. + +Cosign supports **several types of signing keys**, such as **text-based keys**, **cloud KMS-based keys** or **even keys generated on hardware tokens**, and **Kubernetes Secrets**, which can all be generated directly with the tool itself, and also supports adding **key-value annotations** to the signature. + +Luckily, GoReleaser has built-in support for signing blobs, container images by using cosign. +The same rule for Syft applies here, too. +GoReleaser uses cosign by calling it's binary, which means that we should ensure that cosign binary exists before running GoReleaser. +Thanks to the [cosign-installer](https://github.com/sigstore/cosign-installer), a GitHub Action lets you download cosign binary. + +To install cosign, you need to add the following line to our GitHub Action workflow. + +```yaml +- uses: sigstore/cosign-installer@v2.1.0 # installs cosign +``` + +> To install cosign into your environment, please follow the installation link +> from [official website](https://docs.sigstore.dev/cosign/installation/). +> But if you are Mac user, you can start installing cosign via HomeBrew 👇 +> +> ```bash +> $ brew install cosign +> ``` + +You can start signing your artifacts by creating public/private key pairs with the **generate-key-pair** command. Then, you need to run the **sign** command with the private key you generated. But in today's blog post, we'll be talking about a unique concept in cosign called [Keyless Signing](https://github.com/sigstore/cosign/blob/main/KEYLESS.md), which means that we no longer need to generate public/private key pairs. + +> For more background on **"keyless signing"**, see blog posts on the Chainguard blog on [Fulcio](https://chainguard.dev/posts/2021-11-12-fulcio-deep-dive) and [keyless signing with EKS](https://chainguard.dev/posts/2021-11-03-zero-friction-keyless-signing). + +It's important to note that another part of sigstore is [Fulcio](https://github.com/sigstore/fulcio), +a root CA that issues signing certificates from OIDC tokens, and [Rekor](https://github.com/sigstore/rekor), +a transparency log for certificates issued by Fulcio. In October, we announced that +[Actions runs can get OIDC tokens from GitHub for use with cloud providers](https://github.blog/changelog/2021-10-27-github-actions-secure-cloud-deployments-with-openid-connect/), including the public Fulcio and Rekor servers run by the sigstore project. You can sign your container images with the GitHub-provided OIDC token in Actions without provisioning or managing your private key. +This is critically what makes signing so easy. + +> [https://github.blog/2021-12-06-safeguard-container-signing-capability-actions/](https://github.blog/2021-12-06-safeguard-container-signing-capability-actions/) + +To enable signing container images in GoReleaser, all you need to do is add these lines below. + +> Keyless signing is still an experimental feature in cosign, so, we should use a special environment variable to enable that support named `COSIGN_EXPERIMENTAL`. + +```yaml +# signs our docker image +# https://goreleaser.com/customization/docker_sign +docker_signs: + - cmd: cosign + env: + - COSIGN_EXPERIMENTAL=1 + artifacts: images + output: true + args: + - "sign" + - "${artifact}" +``` + +On the other hand, you must add these lines below to enable signing container images in GoReleaser. + +```yaml +# signs the checksum file +# all files (including the sboms) are included in the checksum, so we don't need to sign each one if we don't want to +# https://goreleaser.com/customization/sign +signs: + - cmd: cosign + env: + - COSIGN_EXPERIMENTAL=1 + certificate: "${artifact}.pem" + args: + - sign-blob + - "--output-certificate=${certificate}" + - "--output-signature=${signature}" + - "${artifact}" + artifacts: checksum + output: true +``` + +Once you have all of these, you will end up having something like the following picture for your release: + +![](https://cdn-images-1.medium.com/max/2004/1*cXC_RXowFPRJEIW41olNlg.png) + +Also, a successful release pipeline: + +![[https://github.com/goreleaser/supply-chain-example/runs/4618694011?check_suite_focus=true](https://github.com/goreleaser/supply-chain-example/runs/4618694011?check_suite_focus=true)](https://cdn-images-1.medium.com/max/5084/1*LUmE7iOj-HLkYT-yGoJMnQ.png)[https://github.com/goreleaser/supply-chain-example/runs/4618694011?check_suite_focus=true](https://github.com/goreleaser/supply-chain-example/runs/4618694011?check_suite_focus=true) + +If you verify the container image you pushed to the ghcr.io, a **verify** command of cosign might help you verify the image's signature. + +```bash +$ COSIGN_EXPERIMENTAL=1 cosign verify ghcr.io/goreleaser/supply-chain-example:v1.2.0 +``` + +If you verify the blob, checksums.txt, in this case, you need to download the signature, the certificate, and checksums.txt itself first. + +```bash +$ COSIGN_EXPERIMENTAL=1 cosign verify-blob \ + --cert checksums.txt.pem \ + --signature checksums.txt.sig + checksums.txt \ +tlog entry verified with uuid: "e42743bbbc1d06058ff7705a00bdf5046d920ede73e1fec7f313d19f5f3513b8" index: 977012 +Verified OK +``` + +## Conclusion + +GoReleaser always cares about the security of the artifacts it produces this is why it integrates with tools like cosign, Syft, etc., to mitigate the risks happening in software supply chains. As you can see from the examples we gave, it does that effortlessly by simply adding a bunch of new settings to your GoReleaser configuration file, which all happens behind the scenes without making it even more complex. + +> 🍭 BONUS: Another important topic that gives you a confidence about the software’s integrity is [Reproducible Builds](http://reproducible-builds.org), are a set of software development practices that create an independently-verifiable path from source to binary code, thanks to [Carlos A. Becker](undefined), wrote a blogpost to explain it in detail, so, please do not forget to checkout it his blogpost to learn more about how GoReleaser can help you to achieve reproducibility👇 +> +> [Here's the link](https://medium.com/goreleaser/reproducible-build-with-goreleaser-6de2763458a5). diff --git a/www/docs/blog/posts/2022-05-22-nightly-actions.md b/www/docs/blog/posts/2022-05-22-nightly-actions.md new file mode 100644 index 00000000000..af8c0357067 --- /dev/null +++ b/www/docs/blog/posts/2022-05-22-nightly-actions.md @@ -0,0 +1,98 @@ +--- +date: 2022-05-22 +slug: nightly-actions +categories: + - tutorials +authors: + - hariso +--- + +# Nightly builds with GoReleaser and GitHub Actions + +![](https://miro.medium.com/v2/resize:fit:4800/format:webp/1*MKGwITFSobVveZhnemzKcA.png) +Our flow for nightly builds + + + +Nightly builds offer other benefits like insight into a project’s activity. They also help to highlight the continuous delivery process. In this blog, I’ll go over how my team and I set up nightly builds using [GoReleaser](https://goreleaser.com) and [GitHub](https://github.com). I’ll explain it using a bottom-up approach, so you can better understand what drove our decision and the design of the release procedure. + +Before we dive in here’s a quick overview of how the [Conduit](https://github.com/ConduitIO/conduit) team actually performs releases. A release consists of: + +1. a GitHub release containing binaries for different platforms and a changelog amongst other artifacts. +1. a [Docker](https://www.docker.com) image (pushed to GitHub’s Container Registry) + +## Requirements + +One of Conduit’s primary drivers is being developer-friendly, which also means that we want developers (and Conduit users generally) to be able to try out all the latest features and fixes in any way they want: be it a binary they run directly or a Docker image. We also want to be very precise about what exactly is new in a build. + +The above means that nightly builds and regular builds should have the same structure. For example, if our regular release contains a changelog, a binary for Linux and a Docker image, then a nightly release should also contain a changelog, a binary for Linux and a Docker image. + +Ideally, any change in the release structure should be automatically reflected in both, the regular builds and the nightly builds. Continuing the previous example: if we decide to start supporting Plan9, we ideally change the configuration in one place, and see the change in regular and nightly builds alike. The release structure is defined through a GoReleaser configuration, so for that reason we would like to use a single GoReleaser configuration for both types of builds. + +We have a new contribution almost every (work)day, so we want the nightly builds to be scheduled around them. We may get dependency upgrades on weekends, but we’re fine not having a nightly build only for those. And, of course, we want the older releases to be cleaned up. We keep the builds for at most 7 days, so our full list of requirements is as follows: + +1. Nightly builds are “full releases” (i.e. include everything a “normal” release includes) +1. The existing GoReleaser configuration is used. +1. Nightly builds are scheduled on each working day +1. Nightly builds older than 7 days are removed. + +## Our Process Before Nightly Releases + +We chose GoReleaser to automate building the binaries, create a GitHub release, etc. To build the Docker image we use Docker’s GitHub actions and not GoReleaser, because Conduit comes with a built-in UI. This allows for multi-stage Docker builds, while GoRelease only supports single-stage builds. + +The two parts we had prior to nightly builds were: + +1. A GoReleaser config: [.goreleaser.yml](https://github.com/ConduitIO/conduit/blob/main/.goreleaser.yml) +1. A trigger for the release, in a GitHub workflow, [workflows/release.yml](https://github.com/ConduitIO/conduit/blob/main/.github/workflows/release.yml#L3-L6). The trigger is a tag push. + +What we have works well for major, minor and patch releases. What this process clearly doesn’t have is scheduling nightly builds nor a cleanup. + +## Implementing Nightly Builds + +Now we come to the design of the nightly build process: A scheduled GitHub action is pushing a nightly tag: + +```yaml +on: + schedule: + # * is a special character in YAML, so you have to quote this string + # doing builds Tue-Sat, so we have changes from Fri + # available already on Sat + - cron: "0 0 * * 2-6" +``` + +That will trigger the full release. Then, we use a GitHub action to clean up older GitHub releases, and also a GitHub action to clean up older Docker images. All of that together can be found in [workflows/trigger-nightly.yml](https://github.com/ConduitIO/conduit/blob/main/.github/workflows/trigger-nightly.yml). Here’s the big picture of everything together: + +1. To create a release, GoReleaser and Docker actions are used. +1. A release is triggered by a tag push. +1. “Normal” (major, minor, patch) releases are triggered by manually pushing a tag. +1. Nightly builds (releases) are triggered by pushing a “nightly tag” +1. A workflow is responsible for pushing the nightly tag and also for the cleanup. + +## Don’t forget to clean up + +Nightly builds will accumulate over time, but we don’t want to keep all of them. In the end, those are not stable releases and relying on them for a longer period of time is not recommended anyway. + +We need to clean up older GitHub releases as well as Docker images. Here’s the relevant code snippet from [workflows/trigger-nightly.yml](https://github.com/ConduitIO/conduit/blob/main/.github/workflows/trigger-nightly.yml): + +```yaml +- name: "Clean up nightly releases" + uses: dev-drprasad/delete-older-releases@v0.2.0 + with: + keep_latest: 5 + delete_tags: true + delete_tag_pattern: nightly + env: + GITHUB_TOKEN: ${{ secrets.NIGHTLY_BUILD_GH_TOKEN }} +- name: "Delete nightly containers older than a week" + uses: snok/container-retention-policy@v1 + with: + image-names: conduit + cut-off: 1 week ago UTC + account-type: org + org-name: ConduitIO + keep-at-least: 5 + token: ${{ secrets.NIGHTLY_BUILD_GH_TOKEN }} + filter-tags: "*-nightly*" +``` + +We’ve been running this for a few weeks now (and also started using in [another repository](https://github.com/meroxa/cli)) and it’s been working fine. One of the improvements we’ve identified so far is to announce the nightly release by using GoReleaser. diff --git a/www/docs/blog/posts/2023-01-10-azure-devops.md b/www/docs/blog/posts/2023-01-10-azure-devops.md new file mode 100644 index 00000000000..a14f17b7464 --- /dev/null +++ b/www/docs/blog/posts/2023-01-10-azure-devops.md @@ -0,0 +1,315 @@ +--- +date: 2023-01-10 +slug: azure-devops +categories: + - tutorials +authors: + - dirien +--- + +# Releasing multi-platform container images with GoReleaser in Azure DevOps + + + +## Introduction + +In this article, we learn how to use [GoReleaser](https://goreleaser.com/) to build and release a multi-platform container image to [Azure Container Registry](https://azure.microsoft.com/en-us/services/container-registry/) in [Azure DevOps](https://azure.microsoft.com/en-us/services/devops/). + +This is particularly interesting for teams, who are using mainly Azure and Azure DevOps for their projects and want to build and release container images to Azure Container Registry. + +I try to follow the great article on how to create multi-platform container +images using GitHub actions written by Carlos, the core maintainer of +GoReleaser. If you had no chance to read his blog, [here](https://carlosbecker.com/posts/multi-platform-docker-images-goreleaser-gh-actions/) is the link to it. + +Before we start, let’s take a look on the prerequisites. + +## Prerequisites + +- [Azure DevOps](https://azure.microsoft.com/en-us/services/devops/) account. +- [The GoReleaser Azure DevOps Extension](https://marketplace.visualstudio.com/items?itemName=GoReleaser.goreleaser) installed. +- [Azure](https://azure.microsoft.com/en-us/) account. +- [GoReleaser](https://goreleaser.com/install/) installed on your local machine. + +## The sample application + +Before we can start to set up our pipeline and infrastructure components, lets have a look at the sample application we are going to use in this demo. To keep things simple, I created basic Hello World server using mux library from the Gorilla Web Toolkit. + +Add the library to the `go.mod` file: + +```gomod +module dev.azure.com/goreleaser-container-example + +go 1.19 + +require github.com/gorilla/mux v1.8.0 +``` + +After adding the library, we can move over to implement the basic logic of the application. The server should return a Hello World! string, when we curl the root path of the server. + +In mux this is done, with adding a new route to the router and adding a handler function to it. In my case called HelloWorldHandler. + +Then we can start the server and listen on port 8080. + +```go +package main + +import ( + "log" + "net/http" + "os" + "strings" + "github.com/gorilla/mux" +) + +const ( + // Port is the port the server will listen on + Port = "8080" +) + +func HelloWorldHandler(w http.ResponseWriter, r *http.Request) { + w.Write([]byte("Hello World!")) +} + +func main() { + r := mux.NewRouter() + r.HandleFunc("/", HelloWorldHandler) + port := os.Getenv("PORT") + if port == "" { + port = Port + } + for _, env := range os.Environ() { + if strings.HasPrefix(env, "TEST") { + log.Printf("%s", env) + } + } + log.Println("Listening on port " + port) + log.Fatal(http.ListenAndServe(":"+port, r)) +} +``` + +As we want to create a container image, we need to add a Dockerfile. GoReleaser will then build our container image by copying the previously built binary into the container image. Remember: We don't want to rebuild the binary. So no multi-stage Dockerfile. This way we are sure, that the same binary is used for all distribution methods GoReleaser is offering, and we intended to use. + +```Dockerfile +# Dockerfile +FROM cgr.dev/chainguard/static@sha256:bddbb08d380457157e9b12b8d0985c45ac1072a1f901783a4b7c852e494967d8 +COPY goreleaser-container-example \ + /usr/bin/goreleaser-container-example +ENTRYPOINT ["/usr/bin/goreleaser-container-example"] +``` + +![Chainguard logo](https://cdn-images-1.medium.com/max/2000/0*3X76j809VWDnCLxY) + +You may spot that I use a static container image as base image from Chainguard. Chainguard images are designed for minimalism and security in mind. Many of the images provided by Chainguard are distroless, which means they do not contain a package manager or any other programs that are not required for the specific purpose of the image. Chainguard images are also scanned for vulnerabilities and are regularly updated. You can find more information about Chainguard images here: +[**Chainguard Images** +*Chainguard Images are security-first container base images that are secure by default, signed by Sigstore, and include…*www.chainguard.dev](https://www.chainguard.dev/chainguard-images) + +Let’s pause a minute here and test that everything is working as expected. We can test the application by running it locally: + +```bash +GOOS=linux GOARCH=amd64 go build -o goreleaser-container-example . +docker buildx build --platform linux/amd64 -t goreleaser-container-example . +docker run -p 8080:8080 goreleaser-container-example +``` + +After spinning up the container, you should see the following output: + +```bash +2023/01/10 10:49:31 Listening on port 8080 +``` + +And if we run a curl command in another terminal, we should see the following output: + +```bash +curl localhost:8080 +Hello World! +``` + +Perfect! Everything works as we expected it. Now we can start working on the GoReleaser parts. + +## GoReleaser config file + +We need to provide a goreleaser.yaml config file in the root of our project to tell GoReleaser what to do during the release process. In our case to let GoReleaser to build our container image. To create the goreleaser.yaml we can run following command: + +```bash +goreleaser init +``` + +This should generate the config file for us: + +```bash + • Generating .goreleaser.yaml file + • config created; please edit accordingly to your needs file=.goreleaser.yaml +``` + +The good part when using the init command is, that the goreleaser.yaml comes with some default values. We need to change content as we not need everything GoReleaser is doing by default. Here is the content of the goreleaser.yaml for this demo: + +```yaml +# This is an example .goreleaser.yml file with some sensible defaults. +# Make sure to check the documentation at https://goreleaser.com +before: + hooks: + # You may remove this if you don't use go modules. + - go mod tidy +builds: + - env: + - CGO_ENABLED=0 + goos: + - linux + - darwin +``` + +Later we add the part needed to create the multi-platform container images but for now we dry-run the release process with following GoReleaser command: + +```bash +goreleaser release --snapshot --rm-dist +``` + +Next to the logs of GoReleaser release process, you should also have a dist folder with all the binaries in it. + +> _Exclude this folder in your .gitignore file, to prevent accidentally committing the binaries to your repository._ + +## Azure Container Registry + +> _If you already have an Azure Container Registry you can skip the parts of the creation of the Azure Container Registry._ + +There are several ways, you can create a container registry: You can use the Azure Portal, the Azure CLI, the Azure PowerShell or your favorite Infrastructure as Code tool of choice. + +In this demo, I will use the Azure CLI to create the container registry. You can +find more information about the Azure CLI +[here](https://docs.microsoft.com/en-us/cli/azure/?view=azure-cli-latest). + +First log into the Azure account with the Azure CLI: + +```bash +az login +``` + +We then need to create the resource group and then the container registry service with following commands: + +```bash +# create a resource group in WestEurope datacenter +az group create --name goreleaser-rg --location westeurope +# create the Azure Container registry +az acr create --resource-group goreleaser-rg --name mygoreleaserregistry --sku Basic +``` + +When the container registry is up and running, we can add the dockers configuration to our goreleaser.yaml. But we need to name of our container registry beforehand. + +Use following command to retrieve the name: + +```bash +az acr show --resource-group goreleaser-rg --name mygoreleaserregistry --query loginServer --output tsv +``` + +This is the new part we need to add to our `goreleaser.yaml,` to activate the build of the container image and manifest. + +If you want to know more about the manifest files, I wrote an article about it +[here](https://blog.goreleaser.com/goreleaser-and-docker-manifests-9fe167acf21e) + +```yaml +--- +dockers: + - image_templates: + [ + "mygoreleaserregistry.azurecr.io/{{ .ProjectName }}:{{ .Version }}-amd64", + ] + goarch: amd64 + dockerfile: Dockerfile + use: buildx + build_flag_templates: + - --platform=linux/amd64 + - image_templates: + [ + "mygoreleaserregistry.azurecr.io/{{ .ProjectName }}:{{ .Version }}-arm64", + ] + goarch: arm64 + dockerfile: Dockerfile + use: buildx + build_flag_templates: + - --platform=linux/arm64/v8 +docker_manifests: + - name_template: "mygoreleaserregistry.azurecr.io/{{ .ProjectName }}:{{ .Version }}" + image_templates: + - "mygoreleaserregistry.azurecr.io/{{ .ProjectName }}:{{ .Version }}-amd64" + - "mygoreleaserregistry.azurecr.io/{{ .ProjectName }}:{{ .Version }}-arm64" + - name_template: "mygoreleaserregistry.azurecr.io/{{ .ProjectName }}:latest" + image_templates: + - "mygoreleaserregistry.azurecr.io/{{ .ProjectName }}:{{ .Version }}-amd64" + - "mygoreleaserregistry.azurecr.io/{{ .ProjectName }}:{{ .Version }}-arm64" +``` + +## Azure DevOps + +With the infrastructure done and GoReleaser config finished, we can set up Azure DevOps Service. + +![Switch to the Service Connections screen](https://cdn-images-1.medium.com/max/4060/1*4n5ZoSZ6HMNp0kL--zYFPA.png)_Switch to the Service Connections screen_ + +![Click on the New service connection button](https://cdn-images-1.medium.com/max/4060/1*WBYOMjBmxl_LAEhnUcBSDg.png)_Click on the New service connection button_ + +![Select Azure Container Registry and connect your Azure subscription to it](https://cdn-images-1.medium.com/max/4060/1*kGcBwOXxuKwQsnkwo2870A.png)_Select Azure Container Registry and connect your Azure subscription to it_ + +Time for the last part of our demo: Setting up the Azure DevOps pipeline. I will not go too much into detail about the pipeline, as this is not the focus of this demo. But I will show you the important parts of the pipeline. + +First notable part is the multi-platform command task. I simply followed the instructions from this [article](https://learn.microsoft.com/en-us/azure/devops/pipelines/ecosystems/containers/build-image?view=azure-devops#how-do-i-build-linux-container-images-for-architectures-other-than-x64) on how to setup the task to build multiarch images. + +Next section in the pipeline is the GoReleaser task. This task is using the +GoReleaser extension from the Azure DevOps Marketplace. You can find more +information about the extension [here](https://marketplace.visualstudio.com/items?itemName=ms-azuretools.goreleaser-task). + +I just added the args field and set the value to release `--rm-dist` and defined a condition to only run the task on a tag as GoReleaser will not release on a "dirty" git state. + +This is the complete pipeline: + +```yaml +trigger: + branches: + include: + - main + - refs/tags/* +variables: + GO_VERSION: "1.19.4" + DOCKER_BUILDKIT: 1 +pool: + vmImage: ubuntu-latest +jobs: + - job: Release + steps: + - task: GoTool@0 + inputs: + version: "$(GO_VERSION)" + displayName: Install Go + - task: Docker@2 + inputs: + containerRegistry: "goreleaser" + command: "login" + addPipelineData: false + addBaseImageData: false + - task: CmdLine@2 + displayName: "Install multiarch/qemu-user-static" + inputs: + script: | + docker run --rm --privileged multiarch/qemu-user-static --reset -p yes + - task: goreleaser@0 + condition: and(succeeded(), startsWith(variables['Build.SourceBranch'], 'refs/tags/')) + inputs: + version: "latest" + distribution: "goreleaser" + workdir: "$(Build.SourcesDirectory)" + args: "release --rm-dist" +``` + +To run a release, you need to create a tag in Azure to get the release process started. + +![Logs produced during the release process](https://cdn-images-1.medium.com/max/4060/1*rawcazzmdDWXUzeo-YAIlQ.png)_Logs produced during the release process_ + +And you should see in the Repository tab of your Azure Container Registry service in the Azure Portal UI the multi-platform container images. + +![List of all produced multi-platform container images](https://cdn-images-1.medium.com/max/7184/1*Z8mRJwHIv3o9jWlubU_hhQ.png)_List of all produced multi-platform container images_ + +## Conclusion + +In this demo, I showed you how to create a multi-platform container image using GoReleaser and Azure DevOps and store this image in Azure Container Registry for further usage in your Container based Azure services. + +Setting up all the parts was pretty straight forward where GoReleaser is doing the heavy lifting for us. + +Go ahead and give it a try and let me know what you think about it. diff --git a/www/docs/blog/posts/2023-01-30-goreleaser-ko.md b/www/docs/blog/posts/2023-01-30-goreleaser-ko.md new file mode 100644 index 00000000000..9bdeae128e4 --- /dev/null +++ b/www/docs/blog/posts/2023-01-30-goreleaser-ko.md @@ -0,0 +1,191 @@ +--- +date: 2023-01-30 +slug: goreleaser-ko +categories: + - tutorials +authors: + - developerguy +--- + +# Fast and Furious Building OCI compatible Container Images with GoReleaser and ko + +GoReleaser and [ko][] are popular open-source, well-recognized projects, especially in the containerization and open-source ecosystem for Go applications. +Many people use these projects for their Go applications because they are pretty straightforward and CI-friendly tools that make your releasing artifacts (binary and container image) process super elegant, which also helps you focus more on developing the business logic rather than planning to release software type of works. + + + +I’m so glad to announce that we finally [integrated these fantastic projects](/customization/ko)! + +> If you are interested in learning more about the development process of that +> feature, here is the [PR](https://github.com/goreleaser/goreleaser/pull/3653/) you can take a look. + +As a result, starting from [GoReleaser v1.15](https://github.com/goreleaser/goreleaser/milestone/17), you can build container images by setting configuration options for ko in GoReleaser without having ko installed on your environment. + +This post will be a quick walkthrough to guide people about how things work. + +Before diving into that, let’s refresh our minds about these projects with a quick recap. + +GoReleaser is a tool for creating and releasing Go projects. It automates the +process of building, packaging, and publishing Go binaries and container images, +basically the [fanciest way of releasing Go projects](https://medium.com/trendyol-tech/the-fanciest-way-of-releasing-go-binaries-with-goreleaser-dbbd3d44c7fb). +It is a super user-friendly, easy-to-use, go-to CLI tool and also provides +[GitHub Actions](https://github.com/goreleaser/goreleaser-action) to be +CI-friendly. It also includes a bunch of features for mitigating the risks of +the software supply chain attacks, such as [generating +SBOMs](/customization/sbom), [signing the artifacts](/customization/sign), and +many others. To get more detail, here is the [blog post](https://blog.goreleaser.com/goreleaser-and-software-supply-chain-security-e8917b26924b) for you. + +On the other hand, [ko][] is specifically designed for building and publishing +container images for Go projects. But the utmost vital features in ko are that +it doesn’t require you to run any Docker daemon or write well-designed +Dockerfiles to make the build process cache-efficient, fast and secure. The good +news is that ko will consider all these and build OCI-compatible container +images with all the security options enabled by default, [such as using build +arguments while making Go binaries for +reproducibility](https://ko.build/configuration/#overriding-go-build-settings), +[generating SBOMs, and uploading them to the OCI registry](https://ko.build/features/sboms/), +using the +[smallest and CVE-less base +image](https://github.com/ko-build/ko/blob/453bf803e379696a0b9142c772402ba4599cff34/pkg/commands/options/build.go#L35) +from [Chainguard images](https://github.com/chainguard-images/images) and +providing base information using OCI [base image +annotations](https://github.com/opencontainers/image-spec/blob/main/annotations.md) +and also it [makes easier multi-platform +builds](https://ko.build/features/multi-platform/) by using the +cross-compilation in Go. To get more detail, here is the [blog post](https://blog.kubesimplify.com/getting-started-with-ko-a-fast-container-image-builder-for-your-go-applications) for you. + +It is worth mentioning that [ko applied to become a CNCF sandbox project the last year](https://opensource.googleblog.com/2022/10/ko-applies-to-become-a-cncf-sandbox-project.html), +and glad to see that this application [got accepted by the CNCF](https://lists.cncf.io/g/cncf-toc/message/7743), +which means that ko is now officially a CNCF Sandbox project. + +Without further ado, let’s dive into the details of the integration by showing it in a real-world example. + +> You will find all the source code in GitHub repository, +> [here](https://github.com/developer-guy/goreleaser-with-ko). + +Let’s start with creating a proper directory to host the source code: + +```bash +$ mkdir -p goreleaser-with-ko +$ cd goreleaser-with-ko +``` + +Next, initialize the project: + +```bash +go mod init github.com//goreleaser-with-ko +cat < main.go +package main + +import ( + "fmt" + "os" +) +var ( + // Version is the current version of the application. + Version = "main" +) +func main() { + fmt.Fprintf(os.Stdout, "GoReleaser supports ko! Version: %s", Version) +} +EOF +``` + +It is time to create the configuration file for GoReleaser, which is +[.goreleaser.yml](/customization). + +The easiest way of creating that file is run: `goreleaser init`, which requires +GoReleaser CLI to be installed on your environment; please refer to the +installation page [here](/install) to install it. + +```bash +# it will create the .goreleaser.yml configuration file +# with bunch of default configuration options. +$ goreleaser init +• Generating .goreleaser.yaml file +• config created; please edit accordingly to your needs file=.goreleaser.yaml +``` + +Next, set ko configuration options into .goreleaser.yml. Fortunately, we have +good documentation explaining how we can do this [here](/customization/ko). + +```bash +$ cat <> .goreleaser.yml +kos: + - id: goreleaser-with-ko + platforms: + - linux/amd64 + - linux/arm64 + tags: + - latest + - '{{.Tag}}' + bare: true + flags: + - -trimpath + ldflags: + - -s -w + - -extldflags "-static" + - -X main.Version={{.Tag}} +EOF +``` + +Finally, we’ll automate this workflow on the GitHub Actions platform. +To do this, we need to create a proper folder structure, `.github/workflows` and +put the workflow file into it: + +```bash +$ mkdir -p .github/workflows +$ cat < .github/workflows/release.yaml +name: Releasing artifacts with GoReleaser and ko +on: + push: + tags: + - 'v*' +permissions: + contents: write # needed to write releases + packages: write # needed for ghcr access +jobs: + release: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v3 + with: + fetch-depth: 0 # this is important, otherwise it won't checkout the full tree (i.e. no previous tags) + - uses: actions/setup-go@v3 + with: + go-version: 1.19 + cache: true + - uses: goreleaser/goreleaser-action@v4 # run goreleaser + with: + version: latest + args: release --rm-dist + env: + KO_DOCKER_REPO: ghcr.io/${{ github.repository_owner }}/goreleaser-with-ko + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} +EOF +``` + +As you saw from the file above, we didn’t do anything special about ko +installation, but in case you need to install it into your workflow, you can use +the [setup-ko](https://github.com/ko-build/setup-ko) GitHub Action for that. But how? + +Since ko’s core packages that provide such building and publishing capabilities +are exported functions, you can use them in your own Go projects to get more +detail [here](https://ko.build/advanced/go-packages/). +The following projects are great examples of that: + +- [terraform-provider-ko](https://github.com/ko-build/terraform-provider-ko/blob/main/internal/provider/resource_ko_image.go) +- [miniko](https://github.com/imjasonh/miniko) +- And now, [GoReleaser](https://github.com/goreleaser/goreleaser/blob/main/internal/pipe/ko/ko.go) + +And that’s it. All you need to do at that point is give a tag to your project and wait for the GitHub workflow to be completed to release your software. + +```bash +$ git commit -m"initial commit" -s +$ git tag v0.1.0 -m"first release" +$ git push origin v0.1.0 +``` + +One last note: please remember to use this feature and provide feedback to help us improve this process. Thanks for reading; I hope you enjoyed it; see you in the next blog posts. + +[ko]: https://ko.build/