cmd/image/export: Allow generating deterministic tarballs #2904

mauriciovasquezbernal · 2024-05-27T20:25:09Z

The build command now supports the SOURCE_DATE_EPOCH env variable[0] to allow reproducible builds. Also, the export command is changed to always sort the index to guarentee the exported tarball is deterministic.

Testing

Before

# IG_SOURCE_PATH is needed to build in-tree gadgets that contain a wasm module
$ export IG_SOURCE_PATH=<absolute path to IG source location>

$ cd gadgets/trace_open # any other gadget works as well

# multiple invocations of build produce an image with a different digest each time
$ sudo -E ig image build . -t trace_open:last
INFO[0000] Experimental features enabled
Successfully built ghcr.io/inspektor-gadget/gadget/trace_open:last@sha256:a00e322b2ad607cfb7051dbe241f976b96a189fe7618fc5f4a48e65ffa2fa518

$ sudo -E ig image build . -t trace_open:last
INFO[0000] Experimental features enabled
Successfully built ghcr.io/inspektor-gadget/gadget/trace_open:last@sha256:6c530558cb88a5bf3e6be89866477e819111d5af2e077d192c86e42fe05f7946

# exporting those images will for sure produce a tar file with changing digest a well.

After

# IG_SOURCE_PATH is needed to build in-tree gadgets that contain a wasm module
$ export IG_SOURCE_PATH=<absolute path to IG source location>

# the digest is constant when setting SOURCE_DATE_EPOCH

$ SOURCE_DATE_EPOCH=0 sudo -E ig image build . -t trace_open:latest
INFO[0000] Experimental features enabled
Pulling builder image ghcr.io/inspektor-gadget/ebpf-builder:latest
latest: Pulling from inspektor-gadget/ebpf-builder
Digest: sha256:aa544c4316d583ad7ac4a500b97b67cfbb22e7f3125f8a9a002a4eaedf7f45ef
Status: Image is up to date for ghcr.io/inspektor-gadget/ebpf-builder:latest
Successfully built ghcr.io/inspektor-gadget/gadget/trace_open:latest@sha256:faec585ef38bf306497bb3989753fe9657c05a0fc14d0fabd590ef5ec4aab0b3

$ SOURCE_DATE_EPOCH=0 sudo -E ig image build . -t trace_open:latest
INFO[0000] Experimental features enabled
Pulling builder image ghcr.io/inspektor-gadget/ebpf-builder:latest
latest: Pulling from inspektor-gadget/ebpf-builder
Digest: sha256:aa544c4316d583ad7ac4a500b97b67cfbb22e7f3125f8a9a002a4eaedf7f45ef
Status: Image is up to date for ghcr.io/inspektor-gadget/ebpf-builder:latest
Successfully built ghcr.io/inspektor-gadget/gadget/trace_open:latest@sha256:faec585ef38bf306497bb3989753fe9657c05a0fc14d0fabd590ef5ec4aab0b3

# the exported images are constant as well
$ SOURCE_DATE_EPOCH=0 sudo -E ig image build . -t trace_open:latest
INFO[0000] Experimental features enabled
Pulling builder image ghcr.io/inspektor-gadget/ebpf-builder:latest
latest: Pulling from inspektor-gadget/ebpf-builder
Digest: sha256:aa544c4316d583ad7ac4a500b97b67cfbb22e7f3125f8a9a002a4eaedf7f45ef
Status: Image is up to date for ghcr.io/inspektor-gadget/ebpf-builder:latest
Successfully built ghcr.io/inspektor-gadget/gadget/trace_open:latest@sha256:faec585ef38bf306497bb3989753fe9657c05a0fc14d0fabd590ef5ec4aab0b3
$ sudo -E ig image export trace_open:latest /tmp/image1.tar
INFO[0000] Experimental features enabled
Successfully exported images to /tmp/image1.tar

$ SOURCE_DATE_EPOCH=0 sudo -E ig image build . -t trace_open:latest
INFO[0000] Experimental features enabled
Pulling builder image ghcr.io/inspektor-gadget/ebpf-builder:latest
latest: Pulling from inspektor-gadget/ebpf-builder
Digest: sha256:aa544c4316d583ad7ac4a500b97b67cfbb22e7f3125f8a9a002a4eaedf7f45ef
Status: Image is up to date for ghcr.io/inspektor-gadget/ebpf-builder:latest
Successfully built ghcr.io/inspektor-gadget/gadget/trace_open:latest@sha256:faec585ef38bf306497bb3989753fe9657c05a0fc14d0fabd590ef5ec4aab0b3
$ sudo -E ig image export trace_open:latest /tmp/image2.tar
INFO[0000] Experimental features enabled
Successfully exported images to /tmp/image2.tar

$ sha256sum /tmp/image1.tar /tmp/image2.tar
c3d83221dbda9770558a82cd85c6d1e93891c9dc0e111f5fdd9cd1926459bc54  /tmp/image1.tar
c3d83221dbda9770558a82cd85c6d1e93891c9dc0e111f5fdd9cd1926459bc54  /tmp/image2.tar
mauriciov@mvb-desktop:~/.../trace_open$

This will be useful for #2853 and for tests artifacts generated in #2818

alban · 2024-05-27T20:52:25Z

Instead of disabling CreatedAt, could you pick the last mtime of the source files (gadget.yaml, program.bpf.c, etc.)?

cc @vbatts for https://github.com/vbatts/tar-split

mauriciovasquezbernal · 2024-05-27T21:22:04Z

Instead of disabling CreatedAt, could you pick the last mtime of the source files (gadget.yaml, program.bpf.c, etc.)?

Do you mean in all cases? Or only when --disable-created-at is passed?

(only issue I see in the future is understanding which file to use, there are going to be gadgets with program.bpf.c and program.go, etc. or any of them)

vbatts · 2024-05-28T18:30:04Z

vbatts · 2024-05-28T19:00:40Z

First off: neat. Reproducibility helps make things more predictable and usually has ripple benefits.

Question: just how deterministic to you want or need these archives to be?

The approach is likely "good enough" for some simple use cases, but any slightly more complicated use cases like changes within the versions of golang archive/tar, differences between golang vs libarchive vs gnu vs bsd implementations, etc. then this is not far enough for reproducibility.
Further sometimes even subtle issues like the xattr metadata is a key-value dictionary that is not order dependent, so sometimes with multiple xattrs like selinux or IMA hash or cap permissions, the order changes everytime, and the hashes of the tar will never be consistent.

So, it all depends on your use-cases

mauriciovasquezbernal · 2024-05-28T21:03:27Z

I'd like the export command to generate a deterministic tarball, so we only care about the implementation in golang and not other libraries. How does your library work? Is it just a matter of using github.com/vbatts/tar-split/archive/tar instead of archive/tar? or do I need to do something else?

vbatts · 2024-05-31T09:44:44Z

the tar-split library is a stream reader that you can put inline before you would use archive/tar, and it builds something like a reply-log so you can reassemble the exact tarball from what has been extracted.
It doesn't sound like it matches your use-case.
Perhaps what you've done is simple and good enough.

mauriciovasquezbernal · 2024-05-31T19:00:09Z

Instead of disabling CreatedAt, could you pick the last mtime of the source files (gadget.yaml, program.bpf.c, etc.)?

Updated the logic to use this when a --deterministic flag is passed.

vbatts · 2024-06-03T12:29:03Z

pkg/oci/oci.go

@@ -331,6 +380,10 @@ func tarFolderToFile(src, filePath string) error {
 			return err
 		}

+		header.ModTime = headerTime


Also, I'll say I don't recall the particulars about filepath.Walk() logic. With just a couple of files it may not matter, but could also be a source of non-determinism. That may want a little investigation.

Documentation says:

The files are walked in lexical order, which makes the output deterministic but requires Walk to read an entire directory into memory before proceeding to walk that directory.

So I think we're fine or are you thinking about something else?

It really depends on what we are walking on.
If we walk only on the files of interest, they should not change between several build and should produce the same OCI images.
But, if we are walking all the files, then adding or removing one can have impact on the order of walking and act as a side effect.

In our specific case, I guess this is OK because the directory is temporary one and is used as OCI store, so it should only contains files of interest.

eiffel-fl

Hi!

I am not sure not understand the usage of EBPFSource.

Best regards.

cmd/common/image/build.go

pkg/oci/oci.go

alban

LGTM

Also, shouldn't it be the default?

I'm not sure. I think it's fine to have this disable by default so the created annotation is the time when the image was built.

I am also tempted to make it the default, but I don't have a strong opinion.

mauriciovasquezbernal · 2024-06-24T14:55:00Z

I'll mark this on hold for a bit, perhaps having the --deterministic flag on the build command is a bad idea. I'll check how it's done in docker.

vbatts · 2024-06-25T14:13:06Z

I would not add a flag, and if this is not breaking anything than I would roll forward with it.

mauriciovasquezbernal · 2024-06-25T21:52:05Z

I didn't find a way to remove the flag. Having this by default would imply that the org.opencontainers.image.created annotation is set to the the latest mod time of the source files, which I don't think is correct. So I'd rather stick to have a specific flag for it.

Btw, I checked and docker doesn't support reproducible builds (yet?).

eiffel-fl · 2024-06-26T08:21:59Z

Btw, I checked and docker doesn't support reproducible builds (yet?).

buildkit/buildx, which we use to bake the arm64 image, permits reproducible images:
https://github.com/moby/buildkit/blob/master/docs/build-repro.md
https://docs.docker.com/build/ci/github-actions/reproducible-builds/

vbatts · 2024-06-26T13:15:38Z

On 25/06/24 14:52 -0700, Mauricio Vásquez wrote: Btw, I checked and docker doesn't support reproducible builds (yet?).

no. Only reproducible once the image is pushed to a registry. The reproducibility is only when pulled, that it can again be pushed and not mutate.

mauriciovasquezbernal · 2024-06-26T15:04:04Z

buildkit/buildx, which we use to bake the arm64 image, permits reproducible images:
https://github.com/moby/buildkit/blob/master/docs/build-repro.md
https://docs.docker.com/build/ci/github-actions/reproducible-builds/

Thanks for the pointer. Now I understood the piece I was missing: https://reproducible-builds.org/docs/source-date-epoch/.

I updated the logic to use this variable, and now the --deterministic flag is gone.

@alban @eiffel-fl could you please check it again given that I did some changes?

(I also updated a missing piece of logic to iterate in order through a map, it was also causing the images to be not deterministic)

integration/components/image/image_test.go

docs/core-concepts/images.md

The build command now supports the SOURCE_DATE_EPOCH env variable[0] to allow reproducible builds. Also, the export command is changed to always sort the index to guarentee the exported tarball is deterministic. [0]: https://reproducible-builds.org/docs/source-date-epoch/ Signed-off-by: Mauricio Vásquez <mauriciov@microsoft.com>

Signed-off-by: Mauricio Vásquez <mauriciov@microsoft.com>

eiffel-fl

Hi!

I have one small comment with regard to time parsing.
Otherwise it looks good to me and I definitely think this is a great addition.

Best regards.

eiffel-fl · 2024-06-26T15:44:58Z

cmd/common/image/build.go

+	}
+
+	if sourceDateEpoch, ok := os.LookupEnv("SOURCE_DATE_EPOCH"); ok {
+		sde, err := strconv.ParseInt(sourceDateEpoch, 10, 64)


Should we rather use a time related function to parse this value?
Or since this is always seconds this is OK to use "simple" string conversion.

I think it's fine to use the string conversion as the goal is to convert from string to number. I don't think the time package provides any function that will take an unix time as a string.

Btw, this parsing code was taken from https://reproducible-builds.org/docs/source-date-epoch/.

pkg/oci/build.go

eiffel-fl · 2024-06-26T15:50:14Z

pkg/oci/oci.go

@@ -331,6 +380,10 @@ func tarFolderToFile(src, filePath string) error {
 			return err
 		}

+		header.ModTime = headerTime


It really depends on what we are walking on.
If we walk only on the files of interest, they should not change between several build and should produce the same OCI images.
But, if we are walking all the files, then adding or removing one can have impact on the order of walking and act as a side effect.

In our specific case, I guess this is OK because the directory is temporary one and is used as OCI store, so it should only contains files of interest.

mauriciovasquezbernal · 2024-06-26T16:58:13Z

Thanks for the review and all comments folks, I'll merge it now!

mauriciovasquezbernal requested review from flyth and alban as code owners May 27, 2024 20:25

mauriciovasquezbernal changed the title ~~cm/image/export: Allow generating deterministic tarballs~~ cmd/image/export: Allow generating deterministic tarballs May 27, 2024

mauriciovasquezbernal force-pushed the mauricio/stable-export branch from 6270935 to 476b7f5 Compare May 29, 2024 01:52

mauriciovasquezbernal requested review from blanquicet and mqasimsarfraz as code owners May 29, 2024 01:52

mauriciovasquezbernal changed the base branch from main to mauricio/test-import-export May 29, 2024 01:52

mauriciovasquezbernal force-pushed the mauricio/test-import-export branch from 6049284 to 60351dc Compare May 30, 2024 21:10

mauriciovasquezbernal force-pushed the mauricio/test-import-export branch 3 times, most recently from 7f7452d to 9a1a7d0 Compare May 31, 2024 17:02

Base automatically changed from mauricio/test-import-export to main May 31, 2024 17:55

mauriciovasquezbernal force-pushed the mauricio/stable-export branch from 476b7f5 to 9ca3c41 Compare May 31, 2024 18:59

vbatts reviewed Jun 3, 2024

View reviewed changes

mauriciovasquezbernal force-pushed the mauricio/stable-export branch from 9ca3c41 to f87cbfd Compare June 11, 2024 19:16

eiffel-fl reviewed Jun 12, 2024

View reviewed changes

cmd/common/image/build.go Outdated Show resolved Hide resolved

cmd/common/image/build.go Outdated Show resolved Hide resolved

mauriciovasquezbernal force-pushed the mauricio/stable-export branch 2 times, most recently from a91af8d to e87f983 Compare June 13, 2024 12:35

alban reviewed Jun 24, 2024

View reviewed changes

pkg/oci/oci.go Outdated Show resolved Hide resolved

alban approved these changes Jun 24, 2024

View reviewed changes

mauriciovasquezbernal force-pushed the mauricio/stable-export branch from e87f983 to b5cad43 Compare June 24, 2024 14:43

mauriciovasquezbernal added the state/hold label Jun 24, 2024

mauriciovasquezbernal force-pushed the mauricio/stable-export branch from b5cad43 to f942b0f Compare June 25, 2024 21:50

mauriciovasquezbernal force-pushed the mauricio/stable-export branch from f942b0f to 43a49da Compare June 26, 2024 14:53

mauriciovasquezbernal requested a review from alban June 26, 2024 15:04

mauriciovasquezbernal removed the state/hold label Jun 26, 2024

alban reviewed Jun 26, 2024

View reviewed changes

integration/components/image/image_test.go Outdated Show resolved Hide resolved

docs/core-concepts/images.md Outdated Show resolved Hide resolved

mauriciovasquezbernal added 2 commits June 26, 2024 10:41

test/images: Test that export command is deterministic

6095949

Signed-off-by: Mauricio Vásquez <mauriciov@microsoft.com>

mauriciovasquezbernal force-pushed the mauricio/stable-export branch from 43a49da to 6095949 Compare June 26, 2024 15:41

eiffel-fl approved these changes Jun 26, 2024

View reviewed changes

mauriciovasquezbernal merged commit f687e2a into main Jun 26, 2024
64 of 65 checks passed

mauriciovasquezbernal deleted the mauricio/stable-export branch June 26, 2024 16:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cmd/image/export: Allow generating deterministic tarballs #2904

cmd/image/export: Allow generating deterministic tarballs #2904

mauriciovasquezbernal commented May 27, 2024 •

edited

Loading

alban commented May 27, 2024

mauriciovasquezbernal commented May 27, 2024

vbatts commented May 28, 2024

vbatts commented May 28, 2024

mauriciovasquezbernal commented May 28, 2024

vbatts commented May 31, 2024

mauriciovasquezbernal commented May 31, 2024 •

edited

Loading

vbatts Jun 3, 2024

mauriciovasquezbernal Jun 11, 2024

eiffel-fl Jun 26, 2024

eiffel-fl left a comment

alban left a comment

mauriciovasquezbernal commented Jun 24, 2024

vbatts commented Jun 25, 2024

mauriciovasquezbernal commented Jun 25, 2024

eiffel-fl commented Jun 26, 2024

vbatts commented Jun 26, 2024 via email

mauriciovasquezbernal commented Jun 26, 2024

eiffel-fl left a comment

eiffel-fl Jun 26, 2024

mauriciovasquezbernal Jun 26, 2024

eiffel-fl Jun 26, 2024

mauriciovasquezbernal commented Jun 26, 2024

cmd/image/export: Allow generating deterministic tarballs #2904

cmd/image/export: Allow generating deterministic tarballs #2904

Conversation

mauriciovasquezbernal commented May 27, 2024 • edited Loading

Testing

Before

After

alban commented May 27, 2024

mauriciovasquezbernal commented May 27, 2024

vbatts commented May 28, 2024

vbatts commented May 28, 2024

mauriciovasquezbernal commented May 28, 2024

vbatts commented May 31, 2024

mauriciovasquezbernal commented May 31, 2024 • edited Loading

vbatts Jun 3, 2024

Choose a reason for hiding this comment

mauriciovasquezbernal Jun 11, 2024

Choose a reason for hiding this comment

eiffel-fl Jun 26, 2024

Choose a reason for hiding this comment

eiffel-fl left a comment

Choose a reason for hiding this comment

alban left a comment

Choose a reason for hiding this comment

mauriciovasquezbernal commented Jun 24, 2024

vbatts commented Jun 25, 2024

mauriciovasquezbernal commented Jun 25, 2024

eiffel-fl commented Jun 26, 2024

vbatts commented Jun 26, 2024 via email

mauriciovasquezbernal commented Jun 26, 2024

eiffel-fl left a comment

Choose a reason for hiding this comment

eiffel-fl Jun 26, 2024

Choose a reason for hiding this comment

mauriciovasquezbernal Jun 26, 2024

Choose a reason for hiding this comment

eiffel-fl Jun 26, 2024

Choose a reason for hiding this comment

mauriciovasquezbernal commented Jun 26, 2024

mauriciovasquezbernal commented May 27, 2024 •

edited

Loading

mauriciovasquezbernal commented May 31, 2024 •

edited

Loading