SOURCE_DATE_EPOCH: differ support #3180
Comments
AkihiroSuda
added a commit
to AkihiroSuda/buildkit_poc
that referenced
this issue
Jan 31, 2023
… layers Propagate the `build-arg:SOURCE_DATE_EPOCH` opt value to the differ via the ctx, to limit the upper bound of the file timestamps and set the whiteout timestamps. With this commit, the following workarounds mentioned in `docs/build-repro.md` are no longer needed for reproducible builds: > ```dockerfile > # Limit the timestamp upper bound to SOURCE_DATE_EPOCH. > # Workaround for moby#3180 > ARG SOURCE_DATE_EPOCH > RUN find $( ls / | grep -E -v "^(dev|mnt|proc|sys)$" ) -newermt "@${SOURCE_DATE_EPOCH}" -writable -xdev | xargs touch --date="@${SOURCE_DATE_EPOCH}" --no-dereference > ``` > ```dockerfile > # Squash the entire stage for resetting the whiteout timestamps. > # Workaround for moby#3168 > FROM scratch > COPY --from=0 / / > ``` Limitations: * containerd 1.7 is needed for the containerd worker mode. Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
AkihiroSuda
added a commit
to AkihiroSuda/buildkit_poc
that referenced
this issue
Mar 7, 2023
… layers Propagate the `build-arg:SOURCE_DATE_EPOCH` opt value to the differ via the ctx, to limit the upper bound of the file timestamps and set the whiteout timestamps. With this commit, the following workarounds mentioned in `docs/build-repro.md` are no longer needed for reproducible builds: > ```dockerfile > # Limit the timestamp upper bound to SOURCE_DATE_EPOCH. > # Workaround for moby#3180 > ARG SOURCE_DATE_EPOCH > RUN find $( ls / | grep -E -v "^(dev|mnt|proc|sys)$" ) -newermt "@${SOURCE_DATE_EPOCH}" -writable -xdev | xargs touch --date="@${SOURCE_DATE_EPOCH}" --no-dereference > ``` > ```dockerfile > # Squash the entire stage for resetting the whiteout timestamps. > # Workaround for moby#3168 > FROM scratch > COPY --from=0 / / > ``` Limitations: * containerd 1.7 is needed for the containerd worker mode. Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
AkihiroSuda
added a commit
to AkihiroSuda/buildkit_poc
that referenced
this issue
Mar 7, 2023
… layers Propagate the `build-arg:SOURCE_DATE_EPOCH` opt value to the differ via the ctx, to limit the upper bound of the file timestamps and set the whiteout timestamps. With this commit, the following workarounds mentioned in `docs/build-repro.md` are no longer needed for reproducible builds: > ```dockerfile > # Limit the timestamp upper bound to SOURCE_DATE_EPOCH. > # Workaround for moby#3180 > ARG SOURCE_DATE_EPOCH > RUN find $( ls / | grep -E -v "^(dev|mnt|proc|sys)$" ) -newermt "@${SOURCE_DATE_EPOCH}" -writable -xdev | xargs touch --date="@${SOURCE_DATE_EPOCH}" --no-dereference > ``` > ```dockerfile > # Squash the entire stage for resetting the whiteout timestamps. > # Workaround for moby#3168 > FROM scratch > COPY --from=0 / / > ``` Limitations: * containerd 1.7 is needed for the containerd worker mode. Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
AkihiroSuda
added a commit
to AkihiroSuda/buildkit_poc
that referenced
this issue
Mar 7, 2023
… layers Propagate the `build-arg:SOURCE_DATE_EPOCH` opt value to the differ via the ctx, to limit the upper bound of the file timestamps and set the whiteout timestamps. With this commit, the following workarounds mentioned in `docs/build-repro.md` are no longer needed for reproducible builds: > ```dockerfile > # Limit the timestamp upper bound to SOURCE_DATE_EPOCH. > # Workaround for moby#3180 > ARG SOURCE_DATE_EPOCH > RUN find $( ls / | grep -E -v "^(dev|mnt|proc|sys)$" ) -newermt "@${SOURCE_DATE_EPOCH}" -writable -xdev | xargs touch --date="@${SOURCE_DATE_EPOCH}" --no-dereference > ``` > ```dockerfile > # Squash the entire stage for resetting the whiteout timestamps. > # Workaround for moby#3168 > FROM scratch > COPY --from=0 / / > ``` Limitations: * containerd 1.7 is needed for the containerd worker mode. Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
AkihiroSuda
added a commit
to AkihiroSuda/buildkit_poc
that referenced
this issue
Mar 11, 2023
… layers Propagate the `build-arg:SOURCE_DATE_EPOCH` opt value to the differ via the ctx, to limit the upper bound of the file timestamps and set the whiteout timestamps. With this commit, the following workarounds mentioned in `docs/build-repro.md` are no longer needed for reproducible builds: > ```dockerfile > # Limit the timestamp upper bound to SOURCE_DATE_EPOCH. > # Workaround for moby#3180 > ARG SOURCE_DATE_EPOCH > RUN find $( ls / | grep -E -v "^(dev|mnt|proc|sys)$" ) -newermt "@${SOURCE_DATE_EPOCH}" -writable -xdev | xargs touch --date="@${SOURCE_DATE_EPOCH}" --no-dereference > ``` > ```dockerfile > # Squash the entire stage for resetting the whiteout timestamps. > # Workaround for moby#3168 > FROM scratch > COPY --from=0 / / > ``` Limitations: * containerd 1.7 is needed for the containerd worker mode. Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
AkihiroSuda
added a commit
to AkihiroSuda/buildkit_poc
that referenced
this issue
Mar 14, 2023
… layers Propagate the `build-arg:SOURCE_DATE_EPOCH` opt value to the differ via the ctx, to limit the upper bound of the file timestamps and set the whiteout timestamps. With this commit, the following workarounds mentioned in `docs/build-repro.md` are no longer needed for reproducible builds: > ```dockerfile > # Limit the timestamp upper bound to SOURCE_DATE_EPOCH. > # Workaround for moby#3180 > ARG SOURCE_DATE_EPOCH > RUN find $( ls / | grep -E -v "^(dev|mnt|proc|sys)$" ) -newermt "@${SOURCE_DATE_EPOCH}" -writable -xdev | xargs touch --date="@${SOURCE_DATE_EPOCH}" --no-dereference > ``` > ```dockerfile > # Squash the entire stage for resetting the whiteout timestamps. > # Workaround for moby#3168 > FROM scratch > COPY --from=0 / / > ``` Limitations: * containerd 1.7 is needed for the containerd worker mode. Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
AkihiroSuda
added a commit
to AkihiroSuda/buildkit_poc
that referenced
this issue
Mar 22, 2023
… layers Propagate the `build-arg:SOURCE_DATE_EPOCH` opt value to the differ via the ctx, to limit the upper bound of the file timestamps and set the whiteout timestamps. With this commit, the following workarounds mentioned in `docs/build-repro.md` are no longer needed for reproducible builds: > ```dockerfile > # Limit the timestamp upper bound to SOURCE_DATE_EPOCH. > # Workaround for moby#3180 > ARG SOURCE_DATE_EPOCH > RUN find $( ls / | grep -E -v "^(dev|mnt|proc|sys)$" ) -newermt "@${SOURCE_DATE_EPOCH}" -writable -xdev | xargs touch --date="@${SOURCE_DATE_EPOCH}" --no-dereference > ``` > ```dockerfile > # Squash the entire stage for resetting the whiteout timestamps. > # Workaround for moby#3168 > FROM scratch > COPY --from=0 / / > ``` Limitations: * containerd 1.7 is needed for the containerd worker mode. Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
AkihiroSuda
added a commit
to AkihiroSuda/buildkit_poc
that referenced
this issue
Mar 30, 2023
… layers Propagate the `build-arg:SOURCE_DATE_EPOCH` opt value to the differ, to limit the upper bound of the file timestamps and set the whiteout timestamps. With this commit, the following workarounds mentioned in `docs/build-repro.md` are no longer needed for reproducible builds: > ```dockerfile > # Limit the timestamp upper bound to SOURCE_DATE_EPOCH. > # Workaround for moby#3180 > ARG SOURCE_DATE_EPOCH > RUN find $( ls / | grep -E -v "^(dev|mnt|proc|sys)$" ) -newermt "@${SOURCE_DATE_EPOCH}" -writable -xdev | xargs touch --date="@${SOURCE_DATE_EPOCH}" --no-dereference > ``` > ```dockerfile > # Squash the entire stage for resetting the whiteout timestamps. > # Workaround for moby#3168 > FROM scratch > COPY --from=0 / / > ``` Limitations: * containerd 1.7 is needed for the containerd worker mode. Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
AkihiroSuda
added a commit
to AkihiroSuda/buildkit_poc
that referenced
this issue
Mar 30, 2023
… layers Propagate the `build-arg:SOURCE_DATE_EPOCH` opt value to the differ, to limit the upper bound of the file timestamps and set the whiteout timestamps. With this commit, the following workarounds mentioned in `docs/build-repro.md` are no longer needed for reproducible builds: > ```dockerfile > # Limit the timestamp upper bound to SOURCE_DATE_EPOCH. > # Workaround for moby#3180 > ARG SOURCE_DATE_EPOCH > RUN find $( ls / | grep -E -v "^(dev|mnt|proc|sys)$" ) -newermt "@${SOURCE_DATE_EPOCH}" -writable -xdev | xargs touch --date="@${SOURCE_DATE_EPOCH}" --no-dereference > ``` > ```dockerfile > # Squash the entire stage for resetting the whiteout timestamps. > # Workaround for moby#3168 > FROM scratch > COPY --from=0 / / > ``` Limitations: * containerd 1.7 is needed for the containerd worker mode. Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
AkihiroSuda
added a commit
to AkihiroSuda/buildkit_poc
that referenced
this issue
Mar 30, 2023
… layers Propagate the `build-arg:SOURCE_DATE_EPOCH` opt value to the differ, to limit the upper bound of the file timestamps and set the whiteout timestamps. With this commit, the following workarounds mentioned in `docs/build-repro.md` are no longer needed for reproducible builds: > ```dockerfile > # Limit the timestamp upper bound to SOURCE_DATE_EPOCH. > # Workaround for moby#3180 > ARG SOURCE_DATE_EPOCH > RUN find $( ls / | grep -E -v "^(dev|mnt|proc|sys)$" ) -newermt "@${SOURCE_DATE_EPOCH}" -writable -xdev | xargs touch --date="@${SOURCE_DATE_EPOCH}" --no-dereference > ``` > ```dockerfile > # Squash the entire stage for resetting the whiteout timestamps. > # Workaround for moby#3168 > FROM scratch > COPY --from=0 / / > ``` Limitations: * containerd 1.7 is needed for the containerd worker mode. Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
AkihiroSuda
added a commit
to AkihiroSuda/buildkit_poc
that referenced
this issue
Apr 3, 2023
… layers Propagate the `build-arg:SOURCE_DATE_EPOCH` opt value to the differ, to limit the upper bound of the file timestamps and set the whiteout timestamps. With this commit, the following workarounds mentioned in `docs/build-repro.md` are no longer needed for reproducible builds: > ```dockerfile > # Limit the timestamp upper bound to SOURCE_DATE_EPOCH. > # Workaround for moby#3180 > ARG SOURCE_DATE_EPOCH > RUN find $( ls / | grep -E -v "^(dev|mnt|proc|sys)$" ) -newermt "@${SOURCE_DATE_EPOCH}" -writable -xdev | xargs touch --date="@${SOURCE_DATE_EPOCH}" --no-dereference > ``` > ```dockerfile > # Squash the entire stage for resetting the whiteout timestamps. > # Workaround for moby#3168 > FROM scratch > COPY --from=0 / / > ``` Limitations: * containerd 1.7 is needed for the containerd worker mode. Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
AkihiroSuda
added a commit
to AkihiroSuda/buildkit_poc
that referenced
this issue
May 24, 2023
… layers Propagate the `build-arg:SOURCE_DATE_EPOCH` opt value to the differ, to limit the upper bound of the file timestamps and set the whiteout timestamps. With this commit, the following workarounds mentioned in `docs/build-repro.md` are no longer needed for reproducible builds: > ```dockerfile > # Limit the timestamp upper bound to SOURCE_DATE_EPOCH. > # Workaround for moby#3180 > ARG SOURCE_DATE_EPOCH > RUN find $( ls / | grep -E -v "^(dev|mnt|proc|sys)$" ) -newermt "@${SOURCE_DATE_EPOCH}" -writable -xdev | xargs touch --date="@${SOURCE_DATE_EPOCH}" --no-dereference > ``` > ```dockerfile > # Squash the entire stage for resetting the whiteout timestamps. > # Workaround for moby#3168 > FROM scratch > COPY --from=0 / / > ``` Limitations: * containerd 1.7 is needed for the containerd worker mode. Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
AkihiroSuda
added a commit
to AkihiroSuda/buildkit_poc
that referenced
this issue
Jun 6, 2023
… layers Propagate the `build-arg:SOURCE_DATE_EPOCH` opt value to the differ, to limit the upper bound of the file timestamps and set the whiteout timestamps. With this commit, the following workarounds mentioned in `docs/build-repro.md` are no longer needed for reproducible builds: > ```dockerfile > # Limit the timestamp upper bound to SOURCE_DATE_EPOCH. > # Workaround for moby#3180 > ARG SOURCE_DATE_EPOCH > RUN find $( ls / | grep -E -v "^(dev|mnt|proc|sys)$" ) -newermt "@${SOURCE_DATE_EPOCH}" -writable -xdev | xargs touch --date="@${SOURCE_DATE_EPOCH}" --no-dereference > ``` > ```dockerfile > # Squash the entire stage for resetting the whiteout timestamps. > # Workaround for moby#3168 > FROM scratch > COPY --from=0 / / > ``` Limitations: * containerd 1.7 is needed for the containerd worker mode. Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
AkihiroSuda
added a commit
to AkihiroSuda/buildkit_poc
that referenced
this issue
Jun 8, 2023
… layers Propagate the `build-arg:SOURCE_DATE_EPOCH` opt value to the differ, to limit the upper bound of the file timestamps and set the whiteout timestamps. With this commit, the following workarounds mentioned in `docs/build-repro.md` are no longer needed for reproducible builds: > ```dockerfile > # Limit the timestamp upper bound to SOURCE_DATE_EPOCH. > # Workaround for moby#3180 > ARG SOURCE_DATE_EPOCH > RUN find $( ls / | grep -E -v "^(dev|mnt|proc|sys)$" ) -newermt "@${SOURCE_DATE_EPOCH}" -writable -xdev | xargs touch --date="@${SOURCE_DATE_EPOCH}" --no-dereference > ``` > ```dockerfile > # Squash the entire stage for resetting the whiteout timestamps. > # Workaround for moby#3168 > FROM scratch > COPY --from=0 / / > ``` Limitations: * containerd 1.7 is needed for the containerd worker mode. Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
When differ is creating blobs it should be possible to clamp the timestamps added to the tarballs to the
SOURCE_DATE_EPOCHvalue. Probably this should be opt-in. Currently, the build itself needs to make sure it uses constant timestamps for the files it creates.This does mean changes to the cache storage management. We need to take into account that a blob might exist for a cache record but that doesn't mean that this blob had the timestamps clamped based on current epoch configuration. The most similar concept we have atm is how the blobs with different compressions are managed.
We also need to make sure that remote cache is not matched from blob with invalid timestamps. Otoh if possible then the local cache should still match even if the same build runs with different epoch values (unless
ARG SOURCE_DATE_EPOCHused of course).@AkihiroSuda @ktock
The text was updated successfully, but these errors were encountered: