dist: set -eu, fix shellcheck, make reproducible and smaller tarballs

[vszakats]

• set bash -eu and fix fallouts.
• fix shellcheck warnings.
• set and use SOURCE_DATE_EPOCH for reproducibility.
  Authored-by: Daniel J. H.
  Ref: dist: add SOURCE_DATE_EPOCH env var to maketgz script #13280
• set TZ=UTC and LC_ALL=C for reproducibility.
• make file timestamps in tarball/zip reproducible.
• make directory timestamps in zip reproducible.
• make timestamps of tarballs/zip reproducible.
• make file order in tarball/zip reproducible.
• omit extra file metadata from zip for reproducibility.
• use maximum zip compression.
• use POSIX ustar tarball format to avoid supply chain vulnerability:
  https://seclists.org/oss-sec/2021/q4/0
• make uid/gid in tarball reproducible.
• omit owner user/group names from tarball for reproducibility and privacy.
• omit current timestamp from .gz header for reproducibility.
• display SHA-256 hashes of produced tarballs/zip.
• fix whitespace.

.tar.gz also became smaller in the process: 4,462,311 -> 4,148,249 bytes (8.7.1)

Requires GNU tar, GNU date, sha256sum.

Ref: #13250
Closes #13299

[bagder]

👍

[vszakats]

Haven't found yet where the .tar file is created — depending on how its done
(and if customization is possible), there are some options we could add
to improve its reproducibility too. I've been using those in curl-for-win for a while.

[bagder]

The tarball is made by the make -sj dist command. The dist target is created by automake.

[vszakats]

The tarball is made by the make -sj dist command. The dist target is created by automake.

That was my fear :) I'll have to look...

After looking:

tardir=curl-- && ${TAR-tar} chof - "$tardir" | [...]

• the tar format isn't set, so it's not necessarily ustar. On my macOS, it is, but the official tarball isn't (it says 'non-POSIX', probably GNU).
  It should be ustar (or maybe pax) to avoid supply chain vulnerability: https://seclists.org/oss-sec/2021/q4/0.
• file order is non-reproducible.
• local uid/gid are spilling into the tarball. Repro issue.
• local user/group names may be spilling into the tarball. (On my macOS. In the official tarball this is fortunately not the case.) Privacy and repro issue.

As for how to tell autotools to do the right thing, I have not explored yet. I assume it's impossible or at least painful.

edit:

• it is managed by [...]/automake/1.16.5/share/aclocal-1.16/tar.m4 on my machine.
• one option is to untar and repack. Costs extra build work and logic to maintain for us, plus may make portability trickier.

Ref: https://mgorny.pl/articles/portability-of-tar-features.html

[bagder]

I think we can settle with shipping the content reproducible as start, and not the tarball files themselves.

[bagder]

one option is to untar and repack. Costs extra build work and logic to maintain for us

If we want the tarball images themselves reproducible I figure that's a rather easy thing to add.

[vszakats]

FWIW curl-for-win uses this:

TZ=UTC tar --create \
  --format=ustar \
  --owner=0 --group=0 --numeric-owner \
  --files-from "${_FLS}"

where ${_FLS} lines are reproducibly ordered.

Also needs to install and remap gtar to tar on macOS for example. (Possibly bsdtar can be setup to create the desired output, though this was simpler/safer.)

I'd vote to make the tar images reproducible, if we're tackling this part anyway. (in a separate PR though)

By easy, which method would you go?

[bagder]

By easy, which method would you go?

I figure untaring and retaring with the proper options should work fine. Then we can let make dist work as it does and get reproducibility by our own means.

[bagder]

Maybe like this?

diff --git a/maketgz b/maketgz
index 602f1071b..089d1bfb8 100755
--- a/maketgz
+++ b/maketgz
@@ -174,10 +174,28 @@ res=$?
 if test "$res" != 0; then
     echo "make dist failed"
     exit 2
 fi
 
+retar() {
+  tempdir=$1
+  rm -rf $tempdir
+  mkdir $tempdir
+  cd $tempdir
+  gzip -dc ../$targz | tar -xf -
+  find curl-* | sort > files
+  tar --create --format=ustar --owner=0 --group=0 --numeric-owner --files-from files -zf out.tar.gz
+  rm files
+  mv out.tar.gz ../
+  cd ..
+  rm -rf $tempdir
+}
+
+retar ".tarbuild"
+echo "replace $targz with out.tar.gz"
+mv out.tar.gz "$targz"
+
 ############################################################################
 #
 # Now make a bz2 archive from the tar.gz original
 #
 

[vszakats]

Oh well, make dist fails all over the place inside autotools on macOS. Let's stick to Linux and GNU tools.

[vszakats]

Source tarballs and zip should be fully reproducible now. (Assuming SOURCE_DATE_EPOCH is set to e.g. the tagged commit's timestamp, e.g. with git log -1 '--format=%ct' curl-8_7_1)

Couldn't test maketgz as a whole. Let me know of any issues.

Testing on macOS needs this snippet (plus manually nugding autotools with some command, then copying a missing autotools file into the build directory):

case "$(uname)" in
  Darwin*)
    date() { gdate "$@"; }
    tar() { gtar "$@"; }
    ;;
esac