Automated updates of base images

[SgtCoDFish]

If you were sent here from a reminder in #cert-manager-dev do the following:

• Run make update-kind-images
• Run make update-base-images

If anything was changed, create a PR for the changes to be merged

---

In #3740 an out-of-date base image resulted in a failed vulnerability scan. There likely wasn't any actual security issue, but the risk of ca-certificates and tzdata getting out of date in cert-manager containers that we distribute is non-trivial.

We should investigate how we can prevent this from happening again.

Discussions in the biweekly meeting on 19/05 included:

• Switching to using a tag rather than a digest for base images (e.g. by changing this)
• Embedding certs / tzdata into cert-manager binaries and using scratch over distroless as a base (this doesn't help on its own, but might accompany a different solution)
• Adding automation to detect an out-of-date base image

/kind feature

[munnerz]

Switching to using a tag rather than a digest for base images:

My concern with this approach is that we then don't really know when the base image has changed. If there were some security issue that needed backporting, we'd then not (easily) be able to identify whether our base images in an older release series do contain the patch or not (without inspecting images directly). By using a digest, the action of upgrading the base image is explicit and version control will clearly indicate whether a release used a patched base image.

Adding automation to detect an out-of-date base image

I think this option makes a lot of sense - I wonder if initially, having automated CVE scanning (and alerts) would also suffice? Are we trying to detect security vulnerabilities, or out of date images? (or both?)

Would subscribing the mailing list to say, a release update stream for distroless suffice? (so we are notified when distroless releases a new image?)

[SgtCoDFish]

By using a digest, the action of upgrading the base image is explicit and version control will clearly indicate whether a release used a patched base image.

I agree with this; in a vacuum I'd prefer to have the digest version recorded in source control. Switching to a tag would improve the situation immediately and easily, but I guess the likelihood is that using the tag would inevitably become long term in lieu of work towards automated checks.

a release update stream for distroless

I've had a quick look around and I can't find anything obvious provided by the distroless project for this, or else this would be a good stop-gap solution.

I think ultimately scanning is going to be a good thing to have both for finding out of date images and security issues. Either using ArtifactHub (#4032) or something like https://github.com/quay/clair

Now that I think about it, there's another aspect to this: out of date base images might be updated currently, but I suspect base image updates won't have been backported even to currently supported versions of cert-manager, e.g. #3741 wasn't backported to v1.1.

[munnerz]

We could write a bot that runs say, nightly in a ProwJob, which checks for the latest sha for distroless and then uses bazelisk (the bazel tool for manipulating BUILD files) to patch the file and create a PR to bump image digests for all supported versions perhaps? This has the added benefit of automatically including a release note in the PR so it is clear to end users that a particular patch release also contains an update of the base image. Using a tag will make it very difficult to know whether a given release is safe for use/up to date is my concern 😕

…

On Thu, 20 May 2021 at 13:55, Ashley Davis ***@***.***> wrote: By using a digest, the action of upgrading the base image is explicit and version control will clearly indicate whether a release used a patched base image. I agree with this; in a vacuum I'd prefer to have the digest version recorded in source control. Switching to a tag would improve the situation immediately and easily, but I guess the likelihood is that using the tag would inevitably become long term in lieu of work towards automated checks. a release update stream for distroless I've had a quick look around and I can't find anything obvious provided by the distroless project for this, or else this would be a good stop-gap solution. I think ultimately scanning is going to be a good thing to have both for finding out of date images and security issues. Either using ArtifactHub ( #4032 <#4032>) or something like https://github.com/quay/clair Now that I think about it, there's another aspect to this: out of date base images might be updated currently, but I suspect base image updates won't have been backported even to currently supported versions of cert-manager, e.g. #3741 <#3741> wasn't backported to v1.1. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#4033 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABRWP3PCVSKGW35QXHZQ4TTOUBEDANCNFSM45FEPUQQ> .

[SgtCoDFish]

Using a tag will make it very difficult to know whether a given release is safe for use/up to date is my concern

Yeah, I think I'll cross that off the list in the original post.

We could write a bot that runs say, nightly in...

This sounds like the approach to take long term 👍

In the short term, as a low tech approach, maybe it's worth me adding, say, a slack reminder to #cert-manager-dev every fortnight or so which just says:

Check base image versions, and respond to this comment with a 👍 when you've checked

with a link to this issue. That'll be an improvement we could make today, with almost zero engineering effort.

[SgtCoDFish]

I've set up the reminder as a low-tech stopgap.

[jetstack-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to jetstack.
/lifecycle stale

[SgtCoDFish]

/remove-lifecycle stale

[jetstack-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to jetstack.
/lifecycle stale

[SgtCoDFish]

/remove-lifecycle stale

This will be improved by #4554 but won't be completed by that. In any case, this should remain open.

[jetstack-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to jetstack.
/lifecycle stale

[SgtCoDFish]

/remove-lifecycle stale

Still a TODO!

[SgtCoDFish]

Bumping again... I feel like this is getting closer and it's definitely still just as important!

[jetstack-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to jetstack.
/lifecycle stale

[SgtCoDFish]

/remove-lifecycle stale

[erikgb]

Could it be an option to use Dependabot or Renovate to assist us here? From my experience, Renovate is more flexible - as it allows for markers in custom files, not supported natively by Renovate.

[jetstack-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to jetstack.
/lifecycle stale

[jetstack-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.
Send feedback to jetstack.
/lifecycle rotten
/remove-lifecycle stale

[jetstack-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
Send feedback to jetstack.
/close

[jetstack-bot]

@jetstack-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
Send feedback to jetstack.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[SgtCoDFish]

/remove-lifecycle rotten
/reopen

[jetstack-bot]

@SgtCoDFish: Reopened this issue.

In response to this:

/remove-lifecycle rotten
/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[cert-manager-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
/lifecycle stale

[SgtCoDFish]

/remove-lifecycle stale