Skip to content

Commit

Permalink
adding post on uptodate
Browse files Browse the repository at this point in the history 
Signed-off-by: vsoch <vsoch@users.noreply.github.com>
  • Loading branch information
@vsoch
vsoch committed Sep 19, 2021
1 parent c1613b5 commit 616e333
Showing 1 changed file with 386 additions and 0 deletions.
386 changes: 386 additions & 0 deletions _posts/2021/2021-09-19-uptodate.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,386 @@
---
title: "Uptodate"
date: 2021-09-19 08:30:00
---

I recently had an itch to scratch - and that itch was writing a library in Go.
We don't use Go much for my work, so I figured out a compelling reason to start a new personal project -
a command line tool written in Go (and matching GitHub action) to help keep things up to
date in a repository. Appropriately, I called it <a href="https://vsoch.github.io/uptodate/docs/#/" target="_blank">uptodate</a>!
It was hugely inspired from the <a href="https://github.com/autamus/binoc" target="_blank">binoc</a> (short for "binoculars")
library that can also perform specific kinds of updates, but I wanted more of a focus on
Docker, and to have total control so I could go wild and crazy with writing Go code
without worrying about forcing it on the owner, <a href="https://github.com/alecbcs" target="_blank">alecbcs</a>, to merge my wild ideas.

<br>

<div class="padding:20px">
<img src="https://vsoch.github.io/uptodate/assets/img/uptodate.png">
</div>

## Uptodate

Uptodate is a command line tool in Go and GitHub action that makes it easy to:

<ol class="custom-counter">
<li> Update FROM statements in Dockerfile to have the latest shas</li>
<li> Update build arguments that are for spack versions, GitHub releases and commits, and container hashes.</li>
<li> Generate a matrix of Docker builds from a single configuration file</li>
<li> Generate a matrix of changed files in a repository.</li>
<li> List Dockerfile in a repository that have been changed.</li>
</ol>

With all of the above, you can imagine a workflow that first updates Dockerfile
FROM statements and build args, and then re-builds and deploys these containers -
the assumption being that the underlying dependency such as a GitHub commit
or spack version has an update. Uptodate also will take a nested structure
that I call a docker "build hierarchy" and add new folders and Dockerfile when
a new tag is detected. A kind of updater in uptodate is naturally called an "updater"
and this means for the docker build and docker hierarchy updaters, we can write
a yaml configuration file with our preferences for versions to be added, and
other metadata. You should check out the <a href="https://vsoch.github.io/uptodate/docs/#/user-guide/user-guide" target="_blank">user guide</a>
for detailed usage, or read about <a href="https://vsoch.github.io/uptodate/docs/#/user-guide/github-action" target="_blank">the GitHub action</a>

## How does it work?

I'll give a brief overview of a few of the commands and then a quick example GitHub workflow,
and I'll recommend that you read the documentation for the latest updates on uptodate, harharhar.
The examples below assumed that you've <a href="https://vsoch.github.io/uptodate/docs/#/user-guide/user-guide?id=install" target="_blank">installed</a> uptodate
and have the binary "uptodate" in your path.

### Dockerfile

If you have one or more Dockerfile in your repository you can run uptodate to update digests.
For example:

```
$ uptodate dockerfile .
```

will find Dockerfile in the present working directory and subfolders and update.
For digests, you might see that:

```dockerfile
FROM ubuntu:20.04
```

is updated to

```dockerfile
FROM ubuntu:18.04@sha256:9bc830af2bef73276515a29aa896eedfa7bdf4bdbc5c1063b4c457a4bbb8cd79
```

Note in the above we still have the digest and the tag, so subsequent updates can
further update the sha by looking up the container based on the tag.
And we can also update build arguments that match a particular format! This one,
specifically:

```dockerfile
ARG uptodate_<build-arg-type>_<build-arg-value>=<default>
```

The above flags the build argument for uptodate to look at using the prefix of the library
name, and then the next string after the underscore is the kind of update, followed by
specific metadata for that updater, and of course the value! A few examples are provided below.

#### Spack Build Arguments

<a href="https://github.com/spack/spack" target="_blank">Spack</a> is a package manager intended for HPC, and it's
huge at the lab where I work. So naturally, it made sense for uptodate to be able to
look up the latest spack versions for some package.
To create an argument that matched to a spack package (and its version) you might see:

```dockerfile
ARG uptodate_spack_ace=6.5.6
```

After the updater runs, if it finds a new version 6.5.12, the line will read:

```dockerfile
ARG uptodate_spack_ace=6.5.12
```

This works by using the static API that is deployed alongside the <a href="https://spack.github.io/packages/" target="_blank">Spack Packages</a>
repository that I designed earlier this year. So the updater will get the latest versions
as known within the last 24 hours.

#### GitHub Release Build Argument

If we want an updated version from a GitHub release (let's say the spack software itself)
we might see this:

```dockerfile
ARG uptodate_github_release_spack__spack=v0.16.1
```

The above will look for new releases from spack on GitHub and update as follows:

```dockerfile
ARG uptodate_github_release_spack__spack=v0.16.2
```

#### GitHub Commit Build Argument

Similarity, if we want more "bleeding edge" changes we can ask for a commit
from a specific branch, following this pattern:

```dockerfile
ARG uptodate_github_commit_<org>__<name>__<branch>=<release-tag>
```

Here is an example of asking for updates for the develop branch.

```dockerfile
ARG uptodate_github_commit_spack__spack__develop=NA
```

which wouldn't care about the first "commit" NA as it would update to:

```dockerfile
ARG uptodate_github_commit_spack__spack__develop=be8e52fbbec8106150680fc628dc72e69e5a20be
```

And then to use it in your Dockerfile, you might pop into an environment variable:

```dockerfile
ENV spack_commit=${uptodate_github_commit_spack__spack__develop}
```

See the <a href="https://vsoch.github.io/uptodate/docs/#/user-guide/user-guide?id=dockerfile" target="_blank">docs</a> for more detailed usage and an example for the Dockerfile updater.

### Docker Build

The second updater that I think is pretty useful is the Docker build updater.
This updated will read a config file, an uptodate.yaml, and then follow instructions
for version regular expressoins and different kinds of builds args to generate a matrix of
builds (intended for GitHub actions). For example, let's say that we start with this configuration file:

```yaml

dockerbuild:
build_args:

# This is an example of a manual build arg, versions are required
llvm_version:

# The key is a shorthand used for naming (required)
key: llvm
versions:
- "4.0.0"
- "5.0.1"
- "6.0.0"

# This is an example of a spack build arg, the name is the package
abyss_version:
key: abyss
name: abyss
type: spack

# This will be parsed by the Dockerfile parser, name is the container name
ubuntu_version:

key: ubuntu
name: ubuntu
type: container
startat: "16.04"
endat: "20.04"
filter:
- "^[0-9]+[.]04$"
skips:
- "17.04"
- "19.04"
```

You'll see the primary section of interest is under "dockerbuild" and under this
we have three build args for a manually defined set of versions, a version from
a spack package, and a container. You could run this in a repository root
to look for these config files (and a Dockerfile that they render with in
the same directory or below it) to generate a build matrix.

```bash
$ uptodate dockerbuild
```

Or to only include changed uptodate.yaml files:

```bash
$ uptodate dockerbuild --changes
```

If you provide a registry URI that the containers build to, we can actually check
these containers to look at current build args (that are saved as labels and then
viewable in the image config by uptodate) to determine if an update is needed.

```bash
$ uptodate dockerbuild --registry ghcr.io/rse-radiuss
```

the container. I think this is one of the neatest features - it was just added
in evenings this last week! Check out an
<a href="https://crane.ggcr.dev/config/ghcr.io/rse-radiuss/ubuntu:20.04" target="_blank">example image config</a> that has these labels!
This registry URI will also be included in the output to make it easy to build
In a GitHub action, it might be used like this:

```yaml
jobs:
generate:
name: Generate Build Matrix
runs-on: ubuntu-latest
outputs:
dockerbuild_matrix: {% raw %}${{ steps.dockerbuild.outputs.dockerbuild_matrix }}{% endraw %}
empty_matrix: {% raw %}${{ steps.dockerbuild.outputs.dockerbuild_matrix_empty }}{% endraw %}

steps:
- uses: actions/checkout@v2
if: github.event_name == 'pull_request'
with:
fetch-depth: 0
ref: {% raw %}${{ github.event.pull_request.head.ref }}{% endraw %}

- uses: actions/checkout@v2
if: github.event_name != 'pull_request'
with:
fetch-depth: 0

- name: Generate Build Matrix
uses: vsoch/uptodate@main
id: dockerbuild
with:
root: .
parser: dockerbuild
flags: "--registry ghcr.io/myreponame"

- name: View and Check Build Matrix Result
env:
result: {% raw %}${{ steps.dockerbuild.outputs.dockerbuild_matrix }}{% endraw %}
run: |
echo ${result}
build:
needs:
- generate
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
result: {% raw %}${{ fromJson(needs.generate.outputs.dockerbuild_matrix) }}{% endraw %}
if: {% raw %}${{ needs.generate.outputs.empty_matrix == 'false' }}{% endraw %}

name: "Build {% raw %}${{ matrix.result.container_name }}"{% endraw %}
steps:
- name: Checkout Repository
uses: actions/checkout@v2

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v1

- name: Build {% raw %}${{ matrix.result.container_name }}{% endraw %}
id: builder
env:
container: {% raw %}${{ matrix.result.container_name }}{% endraw %}
prefix: {% raw %}${{ matrix.result.command_prefix }}{% endraw %}
filename: {% raw %}${{ matrix.result.filename }}{% endraw %}
run: |
basedir=$(dirname $filename)
cd $basedir
${prefix} -t ${container} .
```

Of course you'd want to login to a registry, and then also possibly calculate metrics for
the container, so consider this a very simple example.
The build matrix that is being passed between those steps has entries like this:

```json
[
{
"name": "ubuntu/clang/uptodate.yaml",
"container_name": "ghcr.io/rse-radiuss/clang-ubuntu-20.04:llvm-10.0.0",
"filename": "ubuntu/clang/Dockerfile",
"parser": "dockerbuild",
"buildargs": {
"llvm_version": "10.0.0",
"ubuntu_version": "20.04"
},
"command_prefix": "docker build -f Dockerfile --build-arg llvm_version=10.0.0 --build-arg ubuntu_version=20.04",
"description": "ubuntu/clang llvm_version:10.0.0 ubuntu_version:20.04"
},
...
]
```


### Git Updater

I also like this updater because it easily generates for you a matrix of files
that are changed, according to git. Running locally it looks like this:


```bash
$ ./uptodate git /path/to/repo
_ _ _
_ _ _ __ | |_ ___ __| | __ _| |_ ___
| | | | '_ \| __/ _ \ / _ |/ _ | __/ _ \
| |_| | |_) | || (_) | (_| | (_| | || __/
\__,_| .__/ \__\___/ \__,_|\__,_|\__\___|
|_| git
⭐️ Changed Files ⭐️
.github/workflows/build-matrices.yaml: Modify
```
And would generate a matrix for a GitHub action too:
```json
[
{
"name": "Modify",
"filename": "cli/dockerbuild.go"
},
{
"name": "Modify",
"filename": "parsers/common.go"
},
{
"name": "Insert",
"filename": "parsers/docker/buildargs.go"
},
{
"name": "Modify",
"filename": "parsers/docker/docker.go"
},
{
"name": "Modify",
"filename": "tests/ubuntu/21.04/Dockerfile"
},
{
"name": "Modify",
"filename": "tests/ubuntu/clang/Dockerfile"
}
]
```
And of course you can change the default "main" to another branch:
```bash
$ ./uptodate git /path/to/repo --branch master
```
and that also pipes into a GitHub action. I don't want to redundantly reproduce the docs,
so if you are interested you can read more
at the <a href="https://vsoch.github.io/uptodate/docs/#/user-guide/user-guide" target="_blank">user guide</a>
or <a href="https://vsoch.github.io/uptodate/docs/#/user-guide/github-action" target="_blank">GitHub action pages</a>.
Mind you that the library is heavily under develop, so if you have a request for a new updater or want to report
a a bug, please <a href="https://github.com/vsoch/uptodate/issues" target="_blank">let me know!</a>.


## Overview

I have loved working on this library. I think it's the first library in Go where
I've been proficient enough to not look everything up that I need - the code has just
flowed from my fingers! Mind you I'm still figuring out my own design preferences,
and I'm at the stage where I'll write a new functionality, and then immediately not like
my design, and want to re-write it. But I think that means I'll eventually get better.
But it's always good to have one or more projects you are passionate about, because
I don't personally see a point in being a software engineer if I don't (yes, I know it
makes a salary, but I require more than that).

0 comments on commit 616e333

Please sign in to comment.