Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Make whiteout timestamps reproducible #3168

Closed
AkihiroSuda opened this issue Oct 13, 2022 · 2 comments · Fixed by #3981
Closed

Make whiteout timestamps reproducible #3168

AkihiroSuda opened this issue Oct 13, 2022 · 2 comments · Fixed by #3981
Labels
area/reproducible-builds

Comments

@AkihiroSuda
Copy link
Member 

   > Is this going to apply as well for whiteout files?

Needs:

Originally posted by @AkihiroSuda in #2918 (comment)
@tonistiigi tonistiigi added this to the v0.11.0 milestone Nov 7, 2022
@tonistiigi tonistiigi removed this from the v0.11.0 milestone Nov 21, 2022
@AkihiroSuda AkihiroSuda mentioned this issue Dec 15, 2022
Merged
AkihiroSuda added a commit to AkihiroSuda/buildkit_poc that referenced this issue Jan 31, 2023
@AkihiroSuda
Apply SOURCE_DATE_EPOCH to the files and the whiteouts inside OCI tar…
ac6f97b 
… layers

Propagate the `build-arg:SOURCE_DATE_EPOCH` opt value to the differ via the ctx,
to limit the upper bound of the file timestamps and set the whiteout timestamps.

With this commit, the following workarounds mentioned in `docs/build-repro.md`
are no longer needed for reproducible builds:

> ```dockerfile
> # Limit the timestamp upper bound to SOURCE_DATE_EPOCH.
> # Workaround for moby#3180
> ARG SOURCE_DATE_EPOCH
> RUN find $( ls / | grep -E -v "^(dev|mnt|proc|sys)$" ) -newermt "@${SOURCE_DATE_EPOCH}" -writable -xdev | xargs touch --date="@${SOURCE_DATE_EPOCH}" --no-dereference
> ```

> ```dockerfile
> # Squash the entire stage for resetting the whiteout timestamps.
> # Workaround for moby#3168
> FROM scratch
> COPY --from=0 / /
> ```

Limitations:
* containerd 1.7 is needed for the containerd worker mode.

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
AkihiroSuda added a commit to AkihiroSuda/buildkit_poc that referenced this issue Mar 7, 2023
@AkihiroSuda
Apply SOURCE_DATE_EPOCH to the files and the whiteouts inside OCI tar…
523e758 
… layers

Propagate the `build-arg:SOURCE_DATE_EPOCH` opt value to the differ via the ctx,
to limit the upper bound of the file timestamps and set the whiteout timestamps.

With this commit, the following workarounds mentioned in `docs/build-repro.md`
are no longer needed for reproducible builds:

> ```dockerfile
> # Limit the timestamp upper bound to SOURCE_DATE_EPOCH.
> # Workaround for moby#3180
> ARG SOURCE_DATE_EPOCH
> RUN find $( ls / | grep -E -v "^(dev|mnt|proc|sys)$" ) -newermt "@${SOURCE_DATE_EPOCH}" -writable -xdev | xargs touch --date="@${SOURCE_DATE_EPOCH}" --no-dereference
> ```

> ```dockerfile
> # Squash the entire stage for resetting the whiteout timestamps.
> # Workaround for moby#3168
> FROM scratch
> COPY --from=0 / /
> ```

Limitations:
* containerd 1.7 is needed for the containerd worker mode.

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
AkihiroSuda added a commit to AkihiroSuda/buildkit_poc that referenced this issue Mar 7, 2023
@AkihiroSuda
Apply SOURCE_DATE_EPOCH to the files and the whiteouts inside OCI tar…
6155189 
… layers

Propagate the `build-arg:SOURCE_DATE_EPOCH` opt value to the differ via the ctx,
to limit the upper bound of the file timestamps and set the whiteout timestamps.

With this commit, the following workarounds mentioned in `docs/build-repro.md`
are no longer needed for reproducible builds:

> ```dockerfile
> # Limit the timestamp upper bound to SOURCE_DATE_EPOCH.
> # Workaround for moby#3180
> ARG SOURCE_DATE_EPOCH
> RUN find $( ls / | grep -E -v "^(dev|mnt|proc|sys)$" ) -newermt "@${SOURCE_DATE_EPOCH}" -writable -xdev | xargs touch --date="@${SOURCE_DATE_EPOCH}" --no-dereference
> ```

> ```dockerfile
> # Squash the entire stage for resetting the whiteout timestamps.
> # Workaround for moby#3168
> FROM scratch
> COPY --from=0 / /
> ```

Limitations:
* containerd 1.7 is needed for the containerd worker mode.

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
AkihiroSuda added a commit to AkihiroSuda/buildkit_poc that referenced this issue Mar 7, 2023
@AkihiroSuda
Apply SOURCE_DATE_EPOCH to the files and the whiteouts inside OCI tar…
45acbcd 
… layers

Propagate the `build-arg:SOURCE_DATE_EPOCH` opt value to the differ via the ctx,
to limit the upper bound of the file timestamps and set the whiteout timestamps.

With this commit, the following workarounds mentioned in `docs/build-repro.md`
are no longer needed for reproducible builds:

> ```dockerfile
> # Limit the timestamp upper bound to SOURCE_DATE_EPOCH.
> # Workaround for moby#3180
> ARG SOURCE_DATE_EPOCH
> RUN find $( ls / | grep -E -v "^(dev|mnt|proc|sys)$" ) -newermt "@${SOURCE_DATE_EPOCH}" -writable -xdev | xargs touch --date="@${SOURCE_DATE_EPOCH}" --no-dereference
> ```

> ```dockerfile
> # Squash the entire stage for resetting the whiteout timestamps.
> # Workaround for moby#3168
> FROM scratch
> COPY --from=0 / /
> ```

Limitations:
* containerd 1.7 is needed for the containerd worker mode.

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
AkihiroSuda added a commit to AkihiroSuda/buildkit_poc that referenced this issue Mar 11, 2023
@AkihiroSuda
Apply SOURCE_DATE_EPOCH to the files and the whiteouts inside OCI tar…
75276f4 
… layers

Propagate the `build-arg:SOURCE_DATE_EPOCH` opt value to the differ via the ctx,
to limit the upper bound of the file timestamps and set the whiteout timestamps.

With this commit, the following workarounds mentioned in `docs/build-repro.md`
are no longer needed for reproducible builds:

> ```dockerfile
> # Limit the timestamp upper bound to SOURCE_DATE_EPOCH.
> # Workaround for moby#3180
> ARG SOURCE_DATE_EPOCH
> RUN find $( ls / | grep -E -v "^(dev|mnt|proc|sys)$" ) -newermt "@${SOURCE_DATE_EPOCH}" -writable -xdev | xargs touch --date="@${SOURCE_DATE_EPOCH}" --no-dereference
> ```

> ```dockerfile
> # Squash the entire stage for resetting the whiteout timestamps.
> # Workaround for moby#3168
> FROM scratch
> COPY --from=0 / /
> ```

Limitations:
* containerd 1.7 is needed for the containerd worker mode.

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
AkihiroSuda added a commit to AkihiroSuda/buildkit_poc that referenced this issue Mar 14, 2023
@AkihiroSuda
Apply SOURCE_DATE_EPOCH to the files and the whiteouts inside OCI tar…
7121a39 
… layers

Propagate the `build-arg:SOURCE_DATE_EPOCH` opt value to the differ via the ctx,
to limit the upper bound of the file timestamps and set the whiteout timestamps.

With this commit, the following workarounds mentioned in `docs/build-repro.md`
are no longer needed for reproducible builds:

> ```dockerfile
> # Limit the timestamp upper bound to SOURCE_DATE_EPOCH.
> # Workaround for moby#3180
> ARG SOURCE_DATE_EPOCH
> RUN find $( ls / | grep -E -v "^(dev|mnt|proc|sys)$" ) -newermt "@${SOURCE_DATE_EPOCH}" -writable -xdev | xargs touch --date="@${SOURCE_DATE_EPOCH}" --no-dereference
> ```

> ```dockerfile
> # Squash the entire stage for resetting the whiteout timestamps.
> # Workaround for moby#3168
> FROM scratch
> COPY --from=0 / /
> ```

Limitations:
* containerd 1.7 is needed for the containerd worker mode.

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
AkihiroSuda added a commit to AkihiroSuda/buildkit_poc that referenced this issue Mar 22, 2023
@AkihiroSuda
Apply SOURCE_DATE_EPOCH to the files and the whiteouts inside OCI tar…
a274214 
… layers

Propagate the `build-arg:SOURCE_DATE_EPOCH` opt value to the differ via the ctx,
to limit the upper bound of the file timestamps and set the whiteout timestamps.

With this commit, the following workarounds mentioned in `docs/build-repro.md`
are no longer needed for reproducible builds:

> ```dockerfile
> # Limit the timestamp upper bound to SOURCE_DATE_EPOCH.
> # Workaround for moby#3180
> ARG SOURCE_DATE_EPOCH
> RUN find $( ls / | grep -E -v "^(dev|mnt|proc|sys)$" ) -newermt "@${SOURCE_DATE_EPOCH}" -writable -xdev | xargs touch --date="@${SOURCE_DATE_EPOCH}" --no-dereference
> ```

> ```dockerfile
> # Squash the entire stage for resetting the whiteout timestamps.
> # Workaround for moby#3168
> FROM scratch
> COPY --from=0 / /
> ```

Limitations:
* containerd 1.7 is needed for the containerd worker mode.

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
AkihiroSuda added a commit to AkihiroSuda/buildkit_poc that referenced this issue Mar 30, 2023
@AkihiroSuda
Apply SOURCE_DATE_EPOCH to the files and the whiteouts inside OCI tar…
a9b438e 
… layers

Propagate the `build-arg:SOURCE_DATE_EPOCH` opt value to the differ,
to limit the upper bound of the file timestamps and set the whiteout timestamps.

With this commit, the following workarounds mentioned in `docs/build-repro.md`
are no longer needed for reproducible builds:

> ```dockerfile
> # Limit the timestamp upper bound to SOURCE_DATE_EPOCH.
> # Workaround for moby#3180
> ARG SOURCE_DATE_EPOCH
> RUN find $( ls / | grep -E -v "^(dev|mnt|proc|sys)$" ) -newermt "@${SOURCE_DATE_EPOCH}" -writable -xdev | xargs touch --date="@${SOURCE_DATE_EPOCH}" --no-dereference
> ```

> ```dockerfile
> # Squash the entire stage for resetting the whiteout timestamps.
> # Workaround for moby#3168
> FROM scratch
> COPY --from=0 / /
> ```

Limitations:
* containerd 1.7 is needed for the containerd worker mode.

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
AkihiroSuda added a commit to AkihiroSuda/buildkit_poc that referenced this issue Mar 30, 2023
@AkihiroSuda
Apply SOURCE_DATE_EPOCH to the files and the whiteouts inside OCI tar…
300f800 
… layers

Propagate the `build-arg:SOURCE_DATE_EPOCH` opt value to the differ,
to limit the upper bound of the file timestamps and set the whiteout timestamps.

With this commit, the following workarounds mentioned in `docs/build-repro.md`
are no longer needed for reproducible builds:

> ```dockerfile
> # Limit the timestamp upper bound to SOURCE_DATE_EPOCH.
> # Workaround for moby#3180
> ARG SOURCE_DATE_EPOCH
> RUN find $( ls / | grep -E -v "^(dev|mnt|proc|sys)$" ) -newermt "@${SOURCE_DATE_EPOCH}" -writable -xdev | xargs touch --date="@${SOURCE_DATE_EPOCH}" --no-dereference
> ```

> ```dockerfile
> # Squash the entire stage for resetting the whiteout timestamps.
> # Workaround for moby#3168
> FROM scratch
> COPY --from=0 / /
> ```

Limitations:
* containerd 1.7 is needed for the containerd worker mode.

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
AkihiroSuda added a commit to AkihiroSuda/buildkit_poc that referenced this issue Mar 30, 2023
@AkihiroSuda
Apply SOURCE_DATE_EPOCH to the files and the whiteouts inside OCI tar…
415a7ae 
… layers

Propagate the `build-arg:SOURCE_DATE_EPOCH` opt value to the differ,
to limit the upper bound of the file timestamps and set the whiteout timestamps.

With this commit, the following workarounds mentioned in `docs/build-repro.md`
are no longer needed for reproducible builds:

> ```dockerfile
> # Limit the timestamp upper bound to SOURCE_DATE_EPOCH.
> # Workaround for moby#3180
> ARG SOURCE_DATE_EPOCH
> RUN find $( ls / | grep -E -v "^(dev|mnt|proc|sys)$" ) -newermt "@${SOURCE_DATE_EPOCH}" -writable -xdev | xargs touch --date="@${SOURCE_DATE_EPOCH}" --no-dereference
> ```

> ```dockerfile
> # Squash the entire stage for resetting the whiteout timestamps.
> # Workaround for moby#3168
> FROM scratch
> COPY --from=0 / /
> ```

Limitations:
* containerd 1.7 is needed for the containerd worker mode.

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
AkihiroSuda added a commit to AkihiroSuda/buildkit_poc that referenced this issue Apr 3, 2023
@AkihiroSuda
Apply SOURCE_DATE_EPOCH to the files and the whiteouts inside OCI tar…
a7bc2f0 
… layers

Propagate the `build-arg:SOURCE_DATE_EPOCH` opt value to the differ,
to limit the upper bound of the file timestamps and set the whiteout timestamps.

With this commit, the following workarounds mentioned in `docs/build-repro.md`
are no longer needed for reproducible builds:

> ```dockerfile
> # Limit the timestamp upper bound to SOURCE_DATE_EPOCH.
> # Workaround for moby#3180
> ARG SOURCE_DATE_EPOCH
> RUN find $( ls / | grep -E -v "^(dev|mnt|proc|sys)$" ) -newermt "@${SOURCE_DATE_EPOCH}" -writable -xdev | xargs touch --date="@${SOURCE_DATE_EPOCH}" --no-dereference
> ```

> ```dockerfile
> # Squash the entire stage for resetting the whiteout timestamps.
> # Workaround for moby#3168
> FROM scratch
> COPY --from=0 / /
> ```

Limitations:
* containerd 1.7 is needed for the containerd worker mode.

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
AkihiroSuda added a commit to AkihiroSuda/buildkit_poc that referenced this issue May 24, 2023
@AkihiroSuda
Apply SOURCE_DATE_EPOCH to the files and the whiteouts inside OCI tar…
ced39e5 
… layers

Propagate the `build-arg:SOURCE_DATE_EPOCH` opt value to the differ,
to limit the upper bound of the file timestamps and set the whiteout timestamps.

With this commit, the following workarounds mentioned in `docs/build-repro.md`
are no longer needed for reproducible builds:

> ```dockerfile
> # Limit the timestamp upper bound to SOURCE_DATE_EPOCH.
> # Workaround for moby#3180
> ARG SOURCE_DATE_EPOCH
> RUN find $( ls / | grep -E -v "^(dev|mnt|proc|sys)$" ) -newermt "@${SOURCE_DATE_EPOCH}" -writable -xdev | xargs touch --date="@${SOURCE_DATE_EPOCH}" --no-dereference
> ```

> ```dockerfile
> # Squash the entire stage for resetting the whiteout timestamps.
> # Workaround for moby#3168
> FROM scratch
> COPY --from=0 / /
> ```

Limitations:
* containerd 1.7 is needed for the containerd worker mode.

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
AkihiroSuda added a commit to AkihiroSuda/buildkit_poc that referenced this issue Jun 6, 2023
@AkihiroSuda
Apply SOURCE_DATE_EPOCH to the files and the whiteouts inside OCI tar…
b04f204 
… layers

Propagate the `build-arg:SOURCE_DATE_EPOCH` opt value to the differ,
to limit the upper bound of the file timestamps and set the whiteout timestamps.

With this commit, the following workarounds mentioned in `docs/build-repro.md`
are no longer needed for reproducible builds:

> ```dockerfile
> # Limit the timestamp upper bound to SOURCE_DATE_EPOCH.
> # Workaround for moby#3180
> ARG SOURCE_DATE_EPOCH
> RUN find $( ls / | grep -E -v "^(dev|mnt|proc|sys)$" ) -newermt "@${SOURCE_DATE_EPOCH}" -writable -xdev | xargs touch --date="@${SOURCE_DATE_EPOCH}" --no-dereference
> ```

> ```dockerfile
> # Squash the entire stage for resetting the whiteout timestamps.
> # Workaround for moby#3168
> FROM scratch
> COPY --from=0 / /
> ```

Limitations:
* containerd 1.7 is needed for the containerd worker mode.

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
AkihiroSuda added a commit to AkihiroSuda/buildkit_poc that referenced this issue Jun 8, 2023
@AkihiroSuda
Apply SOURCE_DATE_EPOCH to the files and the whiteouts inside OCI tar…
497fd3a 
… layers

Propagate the `build-arg:SOURCE_DATE_EPOCH` opt value to the differ,
to limit the upper bound of the file timestamps and set the whiteout timestamps.

With this commit, the following workarounds mentioned in `docs/build-repro.md`
are no longer needed for reproducible builds:

> ```dockerfile
> # Limit the timestamp upper bound to SOURCE_DATE_EPOCH.
> # Workaround for moby#3180
> ARG SOURCE_DATE_EPOCH
> RUN find $( ls / | grep -E -v "^(dev|mnt|proc|sys)$" ) -newermt "@${SOURCE_DATE_EPOCH}" -writable -xdev | xargs touch --date="@${SOURCE_DATE_EPOCH}" --no-dereference
> ```

> ```dockerfile
> # Squash the entire stage for resetting the whiteout timestamps.
> # Workaround for moby#3168
> FROM scratch
> COPY --from=0 / /
> ```

Limitations:
* containerd 1.7 is needed for the containerd worker mode.

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
AkihiroSuda added a commit to AkihiroSuda/buildkit_poc that referenced this issue Jun 30, 2023
@AkihiroSuda
overlay: set whiteout timestamps to 1970-01-01 (not to SOURCE_DATE_EP…
60c078d 
…OCH)

Even with this commit, the `COPY --from=0 / /` workaround mentioned in `docs/build-repro.md`
is *still* needed for most Dockerfiles. (See PR 3980 for the reason)

Closes moby#3168 (for overlayfs and stargz)

Relevant: containerd/containerd PR 8764

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
@AkihiroSuda AkihiroSuda mentioned this issue Jun 30, 2023
Merged
@AkihiroSuda AkihiroSuda changed the title SOURCE_DATE_EPOCH: support whiteouts Make whiteout timestamps reproducible Jun 30, 2023
@AkihiroSuda
Copy link
Member Author

@AkihiroSuda AkihiroSuda reopened this Jun 30, 2023
@AkihiroSuda
Copy link
Member Author

Implemented in:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/reproducible-builds
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

overlay: set whiteout timestamps to 1970-01-01 (not to SOURCE_DATE_EPOCH) AkihiroSuda/buildkit_poc
2 participants
@tonistiigi @AkihiroSuda