Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Some performance optimizations to dockerTools.build{,Layered}Image #87154

Merged
merged 2 commits into from
May 19, 2020
Merged

Some performance optimizations to dockerTools.build{,Layered}Image #87154

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
69f6294
dockerTools.buildLayeredImage: Avoid appending to tarballs when build…
utdemir May 6, 2020
f12346d
dockerTools: Calculate tarsum's on the fly
utdemir May 6, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 6 additions & 15 deletions pkgs/build-support/docker/default.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -392,14 +392,10 @@ rec {
(cd layer; eval "$extraCommands")
fi

# Tar up the layer and throw it into 'layer.tar'.
# Tar up the layer and throw it into 'layer.tar', while calculating its checksum.
echo "Packing layer..."
mkdir $out
tar --transform='s|^\./||' -C layer --sort=name --mtime="@$SOURCE_DATE_EPOCH" --owner=${toString uid} --group=${toString gid} -cf $out/layer.tar .

# Compute a checksum of the tarball.
echo "Computing layer checksum..."
tarhash=$(tarsum < $out/layer.tar)
tarhash=$(tar --transform='s|^\./||' -C layer --sort=name --mtime="@$SOURCE_DATE_EPOCH" --owner=${toString uid} --group=${toString gid} -cf - . | tee $out/layer.tar | tarsum)

# Add a 'checksum' field to the JSON, with the value set to the
# checksum of the tarball.
Expand Down Expand Up @@ -449,11 +445,7 @@ rec {
# Tar up the layer and throw it into 'layer.tar'.
echo "Packing layer..."
mkdir $out
tar -C layer --hard-dereference --sort=name --mtime="@$SOURCE_DATE_EPOCH" --owner=${toString uid} --group=${toString gid} -cf $out/layer.tar .

# Compute a checksum of the tarball.
echo "Computing layer checksum..."
tarhash=$(tarsum < $out/layer.tar)
tarhash=$(tar -C layer --hard-dereference --sort=name --mtime="@$SOURCE_DATE_EPOCH" --owner=${toString uid} --group=${toString gid} -cf - . | tee $out/layer.tar | tarsum)

# Add a 'checksum' field to the JSON, with the value set to the
# checksum of the tarball.
Expand Down Expand Up @@ -537,11 +529,10 @@ rec {

echo "Packing layer..."
mkdir -p $out
tar -C layer --hard-dereference --sort=name --mtime="@$SOURCE_DATE_EPOCH" -cf $out/layer.tar .
tarhash=$(tar -C layer --hard-dereference --sort=name --mtime="@$SOURCE_DATE_EPOCH" -cf - . |
tee $out/layer.tar |
${tarsum}/bin/tarsum)

# Compute the tar checksum and add it to the output json.
echo "Computing checksum..."
tarhash=$(${tarsum}/bin/tarsum < $out/layer.tar)
cat ${baseJson} | jshon -s "$tarhash" -i checksum > $out/json
# Indicate to docker that we're using schema version 1.0.
echo -n "1.0" > $out/VERSION
Expand Down
46 changes: 21 additions & 25 deletions pkgs/build-support/docker/store-path-to-layer.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,39 +11,35 @@ echo "Creating layer #$layerNumber for $@"
mkdir -p "$layerPath"

# Make sure /nix and /nix/store appear first in the archive.
#
# We create the directories here and use them because
# when there are other things being added to the
# nix store, tar could fail, saying,
# "tar: /nix/store: file changed as we read it"
mkdir -p nix/store
tar -cf "$layerPath/layer.tar" \
--mtime="@$SOURCE_DATE_EPOCH" \
--owner=0 --group=0 \
--transform='s,nix,/nix,' \
nix

# We change into the /nix/store in order to avoid a similar
# "file changed as we read it" error as above. Namely,
# if we use the absolute path of /nix/store/123-pkg
# and something new is added to the nix store while tar
# is running, it will detect a change to /nix/store and
# fail. Instead, if we cd into the nix store and copy
# the relative nix store path, tar will ignore changes
# to /nix/store. In order to create the correct structure
# in the tar file, we transform the relative nix store
# path to the absolute store path.
for storePath in "$@"; do
n=$(basename "$storePath")
tar -C /nix/store -rpf "$layerPath/layer.tar" \

# Then we change into the /nix/store in order to
# avoid a similar "file changed as we read it" error
# as above. Namely, if we use the absolute path of
# /nix/store/123-pkg and something new is added to the nix
# store while tar is running, it will detect a change to
# /nix/store and fail. Instead, if we cd into the nix store
# and copy the relative nix store path, tar will ignore
# changes to /nix/store. In order to create the correct
# structure in the tar file, we transform the relative nix
# store path to the absolute store path.
tarhash=$(
basename -a "$@" |
tar -cp nix \
-C /nix/store --verbatim-files-from --files-from - \
--hard-dereference --sort=name \
--mtime="@$SOURCE_DATE_EPOCH" \
--owner=0 --group=0 \
--transform="s,$n,/nix/store/$n," \
$n
done

# Compute a checksum of the tarball.
tarhash=$(tarsum < $layerPath/layer.tar)
--transform 's,^nix(/|$),/nix/,' \
--transform 's,^[^/],/nix/store/\0,rS' |
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hm, it seems my sed powers are now powerful enough ;)

  • what is the purpose of ^[^/]? Why ^ is not sufficient?
  • I also don't understand why rS flags are added? What is the default behavior, especially regarding the r flag?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it seems my sed powers are now powerful enough ;)

I think that's only because they're tar's variant of sed :).

what is the purpose of ^[^/]? Why ^ is not sufficient?

The transforms get applied one after another, and since the first one converts ^nix to ^/nix, I had to make sure that the second transform won't match to that. Otherwise that nix/ directory at the root would become /nix/store/nix.

I also don't understand why rS flags are added? What is the default behavior, especially regarding the r flag?

The previous code was only replacing the exact store path(s,$n,/nix/store/$n), but since we now archive multiple paths at the same time, we need to prepend /nix/store to everything. But turns out those transformations are applied to symlink targets too, and even when they are relative (eg. ...libpng.so.1 -> libpng.so would become ...libpng.so.1 -> /nix/store/libpng.so). That S flag disables transforming the symlinks. However, just setting S overrides all of the default flags(rsh is the default), so it doesn't apply the transformations to the regular files either. Only setting both r and S makes it so that the transformation is applied to regular files, but not to the symlinks.

Here's the doc I used: https://www.gnu.org/software/tar/manual/html_section/tar_51.html

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wow... ok. Thanks for explanations!

tee "$layerPath/layer.tar" |
tarsum
)

# Add a 'checksum' field to the JSON, with the value set to the
# checksum of the tarball.
Expand Down