New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initial support for reproducible builds #3255

Merged
merged 9 commits into from Dec 30, 2018

Conversation

Projects
None yet
4 participants
@rn
Copy link
Member

rn commented Dec 29, 2018

This PR adds initial support for reproducible linuxkit builds (wip #2813).

  • Files created during the build have their timestamp fixed
  • Slices/Arrays are sorted before being marshalled to JSON

Currently, the tar, tar-kernel-initrd, docker, and kernel+initrd output formats support reproducible builds and tests for tar and kernel+initrd are added with this PR.

ladybugs

rn added some commits Dec 22, 2018

build: Fix the ModTime for files created during build
When creating files for the "intermediate" tar ball,
fix the ModTime. This reduces the difference between
LinuxKit images build from identical inputs.

Signed-off-by: Rolf Neugebauer <rn@rneugeba.io>
build: Improve debug output
Signed-off-by: Rolf Neugebauer <rn@rneugeba.io>
build: Fix timestamp of files created by 'docker export'
Currently 'docker export' is used to convert a linuxkit entry
in the YAML file to a tar file of the root filesystem. This
process creates a number of files and directories which have
the timestamp of when the 'docker export' is run. Fix 'em up.

Signed-off-by: Rolf Neugebauer <rn@rneugeba.io>
build: Make list of capabilities deterministic
Sort list of capabilities that go into config.json.

Signed-off-by: Rolf Neugebauer <rn@rneugeba.io>
build: Make list of mounts deterministic
Sort the list of mount points by destination. This makes the list
deterministic for reproducible builds and also ensures that, e.g.,
the mount for /dev happens before the mount for /dev/pts.

Signed-off-by: Rolf Neugebauer <rn@rneugeba.io>
tests: Add tests for reproducible builds
Test the 'tar' and 'kernel+initrd' formats for now.

Signed-off-by: Rolf Neugebauer <rn@rneugeba.io>
docs: Add details about reproducible builds
Signed-off-by: Rolf Neugebauer <rn@rneugeba.io>
docs: Link reproducible build doc from top-level README
Signed-off-by: Rolf Neugebauer <rn@rneugeba.io>
docs: Dev reports are monthly not weekly
Signed-off-by: Rolf Neugebauer <rn@rneugeba.io>
@deitch

This comment has been minimized.

Copy link
Collaborator

deitch commented Dec 29, 2018

This is rather nice!

Is there concern that the modtime now is just the timestamp that lkt open sourced? Might we want to have a cmdline option to set the date-time as an input, with a default for that time, and maybe an option that says, "now"?

E.g.

$ lkt build --modtime now ...
$ lkt build --modtime 20181229T175500Z ...

Or -timestamp or some other flag name, but that idea? And if not specified, just uses the date/time lkt was open-sourced?

@rn

This comment has been minimized.

Copy link
Member

rn commented Dec 29, 2018

Yeah, was considering making the timestamp configurable. But rather than via the commandline, we should probably support this proposal and make it configurable via the environment variable SOURCE_DATE_EPOCH. Just want to get the basics in place and then iterate over refinements...

@deitch

This comment has been minimized.

Copy link
Collaborator

deitch commented Dec 29, 2018

If that’s where it’s going as a standard, I guess. I like command line options as they are explicit. But not going to be a stickler about it. :-)

@justincormack

This comment has been minimized.

Copy link
Collaborator

justincormack commented Dec 30, 2018

LGTM as a first pass. I think setting all times to the git commit time of the linuxkit build itself might also be a reasonable option in future, but a static time is fine for now.

@rn

This comment has been minimized.

Copy link
Member

rn commented Dec 30, 2018

I like the idea of using the git commit time as the default. That way it is different for different linuxkit versions. Might look at doing this as an improvement later along with support for SOURCE_DATE_EPOCH.

@rn rn merged commit 2b826be into linuxkit:master Dec 30, 2018

3 checks passed

ci/circleci Your tests passed on CircleCI!
Details
ci/datakit/linuxkit-ci All tests passed
Details
dco-signed All commits are signed

@rn rn deleted the rn:repeat branch Dec 30, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment