Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: make image generation deterministic by default #51

Merged
merged 6 commits into from Mar 4, 2022

Conversation

kruskall
Copy link
Contributor

@kruskall kruskall commented Mar 3, 2022

This PR makes image generation deterministic by default:

  • timestamps are zeroed out for reproducibility (see Zero out timestamps for reproducibility #48)
  • apk is writing a tar archive to lib/apk/db/scripts.tar, to support reproducible builds the timestamps inside the archive needs to be deterministic/zeroed.

Note: according to the spec of SOURCE_DATE_EPOCH (https://reproducible-builds.org/specs/source-date-epoch/):

Build processes MUST use this variable for embedded timestamps in place of the "current" date and time.

Since time.Now has been replaced with the zero value (January 1, year 1, 00:00:00.000000000 UTC) there's no need for SOURCE_DATE_EPOCH.

Note: reproducible image generation has been tested with https://diffoscope.org/

Closes #48

cc @imjasonh

pkg/build/apk.go Outdated Show resolved Hide resolved
@kaniini
Copy link
Contributor

kaniini commented Mar 3, 2022

Since time.Now has been replaced with the zero value (January 1, year 1, 00:00:00.000000000 UTC) there's no need for SOURCE_DATE_EPOCH.

Some users do not like seeing timestamps of 1970 in things like ls output. It would be nice to support SOURCE_DATE_EPOCH, and also have an option to set SOURCE_DATE_EPOCH to time.Now().

@kruskall
Copy link
Contributor Author

kruskall commented Mar 3, 2022

Some users do not like seeing timestamps of 1970 in things like ls output. It would be nice to support SOURCE_DATE_EPOCH, and also have an option to set SOURCE_DATE_EPOCH to time.Now().

I'm ok with supporting SOURCE_DATE_EPOCH but an environment variable is a bit opaque in term of UI, I think there should be a easier/simpler way for the user to make the image reproducible. Maybe an additional flag ? WDYT ?

@kaniini
Copy link
Contributor

kaniini commented Mar 4, 2022

I'm ok with supporting SOURCE_DATE_EPOCH but an environment variable is a bit opaque in term of UI, I think there should be a easier/simpler way for the user to make the image reproducible. Maybe an additional flag ? WDYT ?

Yes, a global option --build-date would be great, with --build-date=now being equivalent to time.Now().

pkg/build/build.go Outdated Show resolved Hide resolved
pkg/cli/publish.go Outdated Show resolved Hide resolved
pkg/build/image_builder.go Show resolved Hide resolved
pkg/cli/build-minirootfs.go Outdated Show resolved Hide resolved
pkg/cli/build-minirootfs.go Outdated Show resolved Hide resolved
@kruskall
Copy link
Contributor Author

kruskall commented Mar 4, 2022

rebased on the main branch and fixed the merge conflicts 🎉

@kaniini
Copy link
Contributor

kaniini commented Mar 4, 2022

@imjasonh @puerco what do you think?

Copy link
Member

@imjasonh imjasonh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@kaniini kaniini merged commit 22d03b1 into chainguard-dev:main Mar 4, 2022
@imjasonh
Copy link
Member

imjasonh commented Mar 4, 2022

Ugh, I'm a dummy, it's supposed to be seconds since epoch, not RFC3339: https://github.com/google/ko/blob/3fc720f912ac5192cff518a31fa6621e4a0dba51/pkg/commands/config.go#L129

Sorry for the confusion. I won't be able to send a PR until maybe tonight, but I'll happily review anything.

@imjasonh
Copy link
Member

imjasonh commented Mar 4, 2022

Extra dumb! That's exactly what you have. Ignore me, it's the Fridays.

@kruskall kruskall deleted the feat/reproducibility branch March 4, 2022 21:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Zero out timestamps for reproducibility
4 participants