Add `--twice` (or `--ntimes=N`) flag to test reproducibility

[AkihiroSuda]

Description

The --twice (or --ntimes=N) flag will run a build twice, and bail if the digests do not match.
The second run will automatically run with --no-cache to disable layer caches (but keep cached mounts).

The SOURCE_DATE_EPOCH will be set to the time of the command execution if not explicitly set.

[crazy-max]

The --twice (or --ntimes=N) flag will run a build twice, and bail if the digests do not match.

Maybe a --validate flag would be better?

[AkihiroSuda]

The --twice (or --ntimes=N) flag will run a build twice, and bail if the digests do not match.

Maybe a --validate flag would be better?

--validate sounds confusing. --validate-reproducibility would be fine, but too long.

[crazy-max]

--validate sounds confusing. --validate-reproducibility would be fine, but too long.

Let's say in the future we want another kind of validation. Maybe then --validate could be a slice and would take as value what we want to validate. For this case would be --validate=repro and for potential other validations: --validate=repro --validate=foo .... If empty would run all of them? cc @oyabun

[thaJeztah]

Such a flag feels more like a debugging tool; is that something a users would run normally?

[AkihiroSuda]

Such a flag feels more like a debugging tool; is that something a users would run normally?

Eventually yes, at least on CIs.
Can be just an option for the GHA action.

[thaJeztah]

Yes, wondering indeed if having an option in GHA action would be more suitable.

Mostly concerned about overloading the (already extensive list of flags/options), especially for cases where there may already be alternatives (e.g. someone could have other tooling to verify such things).

If we do think it's good to include this in the CLI, we could (should?) start looking at Cobra's features to group options. Or at least, I THINK I saw they added options to group things to allow flags/options to be shown in separate sections. That way we could keep the "more commonly used" options separate from "more esoteric" options.

(But still in favour of considering do we really need this in the cli itself first 🤗)

[lohitkolluri]

Opened PR #3872 proposing an extensible --validate flag with an initial --validate=repro implementation. It runs the build twice (second run forces --no-cache) and compares the resulting image digest. If SOURCE_DATE_EPOCH is not explicitly set, it is pinned for both runs to make the comparison meaningful.