change `tar` command used in `get_source_tarball_from_git` to get reproducible tarballs #4248

Rovanion · 2023-04-20T09:19:30Z

Hi,
This pull request contains two commits performing two changes to easybuild that were needed to build a piece of Python software that was not located on the Python Package Index but only existed in a git repository.

The first change allows sources in exts_list to specify a git_config. The second change changes the command used to produce tarballs from git archives so that they are made in a reproducible manner. Otherwise each rerun of eb would result in a checksum mismatch.

These two patches have been tested on CentOS 7 and no other operating system. The command used to produce the tarball from git is complicated by the old versions of software that is found on CentOS/RHEL 7. But in the comments instructions on how to drop support for such outdated operating systems is also included.

If there are any questions, feel free to ask.
Cheerio!

easybuild/framework/easyblock.py

easybuild/tools/filetools.py

Rovanion · 2023-05-11T07:13:41Z

Hi, Hopefully I can adress your concerns.

1.a. This may not be enough to get 100% reproducible tarballs across a variety of systems.

This patch could be insufficient, that is always a possibility, but I have done my best to be compatible with old versions of GNU Tar. Given that the patch is based on documentation from reproducible-builds.org, is it not probable that it will improve the situation compared to the code in place today? The code can always be improved upon if a user of easybuild comes with a report of the code not producing reproducible tarballs. 1.b. I'm fairly sure that a different version of gzip being used may

already be sufficient, since that may do better compression, etc.

I do not think this is the case. Sorting the files, zeroing out time and user metadata - all of this will be included by default in all versions of gzip that I am aware of, unless we tell gzip otherwise. The efficiency of the compression algorithm itself is irrelevant for the reproducability of the tarball. 1.c. Another factor could be locale, which may affect the order of paths

produced with the find command.

Which is why the results of find ar piped through sort with LC_ALL set to C.

1.d. If the tarball creation not 100% reproducible across a wide variety of systems, it's basically useless, since people who get a slightly different tarball will be running into incorrect checksums all the time, which would not be good.

I am not sure I understand. It is already possible to create tarballs from git archives using easybuild today, right? This patch, at a minimum, makes the creation of tarballs from git reproducible on some systems whereas before they were never (?) reproducible. I say never since git does not track file creation or modification time, so they would always be set to the time at which the repository was cloned. [1] But I could be totally wrong here since I'm relatively new to easybuild.

2. The side effect of using --mtime to reset time stamps and --owner=0 is that information gets lost. Especially the timestamp info may be interesting, so it should be retained.

This is the very core of the patch. These variables, data that differ between downloads of the same git repository, must be zeroed out or set to known values so that each tarball of the same git repository and commit/tag/hash is the same. You cannot create a reproduceable tarball of a git repository without erasing the mtime of the files on the local filesystem in some manner.

3. This likely breaks the creating of tarball for easyconfigs using git_config on macOS, which is a regression. Although you currently won't get very far with installing software on macOS with EasyBuild, features like downloading sources with eb --fetch should work. Do you want me to run some test protocol on a mac? I could probably get

hold of such a machine. [1]: https://stackoverflow.com/a/62039285/501017

…

Message ID: <easybuilders/easybuild-framework/pull/4248/review/1420648038@ github.com>

Rovanion · 2023-05-11T08:17:18Z

Hej Åke!

Changing mtime is an absolute do-not-do-that.

Could you clearify why that is? If I understand correctly Git does not record the mtime of the original source files. This means that the mtime of the local files on disks at the point of archive creation only reflects when the git repository was cloned, no more no less.

It also has the problem of not generating a tar file that has the same content as what tar zcf does. A small test I did shows that the file/sort/tar pipe above actually adds each file twice in the tar file. Example: mkdir q echo hej > q/t echo hopp > q/w find q | sort | tar --create --file q.tar --files-from - tar cf q2.tar q tar tf q2.tar q/ q/t q/w tar tf q.tar q/ q/t q/w q/t q/w this is caused by directory coming first, thereby making tar copy the whole dir into the tar file, then the list of files, making tar add the files once more.

Thank you for finding the bug and the clear demonstration! I had missed attaching the flag --no-recursion to the tar-invocation. I have pushed a modified version of the patch that rectifies this shortcomming. Thank you also for prodding me to have a look at contents of the archive my version of easybuild produced. Turns out .git wasn't properly pruned either. But the contents are removed in the latest version of the patch.

…

Message ID: <easybuilders/easybuild-framework/pull/4248/review/1421806504@ github.com>

boegel · 2023-06-07T14:08:46Z

We've discussed this PR a bit in a recent EasyBuild conf call, and the general consensus was that we can indeed proceed with this, but do so in the scope of the EasyBuild 5.0 effort (since it's a significant change).

@Rovanion Are you up for making the tests happy here that complain about style issues?
Once that's done, we could look into moving this over to target the 5.0.x branch...

Rovanion · 2023-06-07T15:14:26Z

Yeah I can do that. I'll see if I can get it in before vacation.

Rovanion · 2023-06-13T09:20:33Z

All linter complaints should be fixed now.

test/framework/filetools.py

Rovanion · 2023-06-19T15:24:06Z

The only workflow errors that seem to occur are running Python 2.7 and 3.5 which seem to have been (partly?) dropped from the list of supported Pythons to run Easybuild on?

Rovanion · 2023-06-21T08:57:32Z

The check "Tests for Apptainer container support / build (3.7, 1.0.0) (pull_request)" have failed due to a fail to write a file, looks like the machine used for testing failed in this case.

Rovanion · 2023-07-04T15:27:14Z

Is there anything else that needs to be done before this can be merged?

boegel · 2023-07-05T14:30:04Z

@Rovanion We need to take another detailed look at this, but what definitely needs to happen is re-targeting this to the 5.0.x branch, since we're actively working towards EasyBuild 5.0 which includes some breaking changes (see https://github.com/orgs/easybuilders/projects/18 +

jfgrimm · 2023-12-20T15:55:51Z

when not excluding .git, then:

.git/logs/HEAD, .git/logs/refs/heads/main, .git/logs/refs/remotes/origin/HEAD contain unix timestamps from when the repo was cloned
.git/index differs
.git/objects/pack/pack-*.{idx,pack} have different names, and contents

the unix timestamps are easy enough to deal with, but the metadata encoded in the index and pack files is more complicated

boegel · 2024-01-03T15:04:52Z

@Rovanion Any idea if the issues that @jfgrimm raised can be fixed?

Rovanion · 2024-01-08T11:06:56Z

Hi,
I've thought some about this but not reached any conclusions I'm satisfied with.

On the BSD/Mac side: Does EasyBuild have a bootstrapping process/story in place so that it may use a self-built version of GNU Tar?

If not: I read up on flags for FreeBSDs Tar one day during the Holidays, was not able to find any definitive answer but hints that it should be possible to coax it into submission.

On the subject of including .git: It could perhaps be possible to rewrite the files in .git, but then its contents wouldn't be the same as when the binary artifact was built so is it even valid by then? My main question is though: For which usecase is does this feature exist?

lexming · 2024-02-14T22:32:15Z

I brought this PR to the last EB5 conf call and we might have a solution for it:

use a common set of features between GNU tar and BSD tar to guarantee compatibility
discard --mtime due to point 1 and instead manually set the timestamps of each file with touch -t 197001010100 prior to tar
use reproducible tarballs for sources from git repos that are pulled without .git (that is the default behaviour)
sources from git repos explicitly setting keep_git_dir=True will be kept with .git and won't be checksumed, because we can't as discussed here

@Rovanion is this a workable solution for you?

lexming · 2024-02-14T22:48:00Z

A little proof of concept with touch:

$ git clone https://github.com/easybuilders/easybuild
Cloning into 'easybuild'...
remote: Enumerating objects: 11089, done.
remote: Counting objects: 100% (909/909), done.
remote: Compressing objects: 100% (457/457), done.
remote: Total 11089 (delta 482), reused 851 (delta 442), pack-reused 10180
Receiving objects: 100% (11089/11089), 595.66 MiB | 10.09 MiB/s, done.
Resolving deltas: 100% (7324/7324), done.
# Delete .git folder
$ rm -rf easybuild/.git
# Checksum of downloaded files
$ tar --create --file easybuild.tar easybuild/
$ sha256sum easybuild.tar
98457ba141e829641878550bcf82681c87380acd106d695235385043dc7327ad  easybuild.tar
# Checksum of downloaded files with timestamp reset
$ find easybuild/ -exec touch -t 197001010100 {} \;
$ tar --create --owner=0 --group=0 --numeric-owner --format=gnu --null --file easybuild.0.tar easybuild/
$ sha256sum easybuild.0.tar
e1c54e194764bcd89165da527c25b4f5d82fcbd132be9d5f1f75ee40d9eb6ddc  easybuild.0.tar
# Wait some time and repeat the clone + touch + tar
$ sha256sum easybuild.1.tar
e1c54e194764bcd89165da527c25b4f5d82fcbd132be9d5f1f75ee40d9eb6ddc  easybuild.1.tar

Rovanion · 2024-02-28T13:58:15Z

Yes, that would work. I'm not available to work on EasyBuild at the moment, I have been directed to prioritize other projects. Feel free to implement this until I am able to work on EasyBuild again.

…et timestamps with touch

…th reproducible tar commands

lexming

Thanks for the feedback. I made a PR to this one with the discussed changes. Please check Rovanion#1

The PR looks messy because I'm also syncing to current state of 5.0.x branch.
The meaningful commit is Rovanion@8d9e14e

use reproducible archive for git repos without .git. dir
set timestamps with touch
avoid deprecated GZIP environment variable by piping into gzip

…plementations

Update for PR 4248

lexming · 2024-03-01T00:34:51Z

easybuild/tools/filetools.py

+            # print names of all files and folders excluding .git directory
+            'find', repo_name, '-name ".git"', '-prune', '-o', '-print0',
+            # reset access and modification timestamps
+            '-exec', 'touch', '-t 197001010100', '{}', '\;', '|',


@Rovanion I missed using a raw string for this bit:

Suggested change

'-exec', 'touch', '-t 197001010100', '{}', '\;', '|',

'-exec', 'touch', '-t 197001010100', '{}', r'\;', '|',

for \; in get_source_tarball_from_git.

boegel · 2024-03-02T13:06:17Z

@Rovanion Problems with Apptainer tests was fixed in #4472, which was also applied to the 5.0.x branch (so you'll need to sync up your PR branch with the current 5.0.x branch)

lexming

@Rovanion I opened another PR to this one that just syncs with recent changes in the 5.0.x branch. Unfortunately, at the time of the previous PR, unit tests were broken. Can you please merge Rovanion#2?

boegel · 2024-03-11T10:12:33Z

@Rovanion ping?

Sync PR#4248 with 5.0.x

Rovanion · 2024-03-18T16:03:22Z

I think all tests have passed now. All that remains are the three reviews.

req. changes made

lexming

@Rovanion Thanks a lot for keeping up with the updates!

LGTM

Rovanion · 2024-03-20T21:37:18Z

Cheers! Den mån 18 mars 2024 kl 22:57 skrev Alex Domingo ***@***.***>:

…

Merged #4248 <#4248> into 5.0.x. — Reply to this email directly, view it on GitHub <#4248 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAE2PRYY6VUWJ22ESTQZVRLYY5PK5AVCNFSM6AAAAAAXFGZW66VHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMJSGE2TSOJYGM4TINY> . You are receiving this because you were mentioned.Message ID: ***@***.*** .com>

Rovanion force-pushed the reproducible-git-tarballs branch from 354b94b to 1b4e18d Compare April 20, 2023 09:24

Micket added the enhancement label May 4, 2023

Micket previously requested changes May 4, 2023

View reviewed changes

easybuild/framework/easyblock.py Outdated Show resolved Hide resolved

easybuild/tools/filetools.py Outdated Show resolved Hide resolved

Rovanion force-pushed the reproducible-git-tarballs branch from 1b4e18d to 049b0cb Compare May 5, 2023 07:42

boegel added this to the 4.x milestone May 10, 2023

boegel changed the title ~~Reproducible git tarballs~~ change tar command used in get_source_tarball_from_git to get reproducible tarballs May 10, 2023

Rovanion force-pushed the reproducible-git-tarballs branch 2 times, most recently from 82cb6f0 to d257534 Compare May 11, 2023 08:15

Rovanion force-pushed the reproducible-git-tarballs branch from d257534 to d40c8d4 Compare May 11, 2023 08:21

Rovanion changed the title ~~change tar command used in get_source_tarball_from_git to get reproducible tarballs~~ Change tar command used in get_source_tarball_from_git to get reproducible tarballs May 11, 2023

boegel modified the milestones: 4.x, 5.0 Jun 7, 2023

boegel added the EasyBuild-5.0 EasyBuild 5.0 label Jun 7, 2023

Rovanion force-pushed the reproducible-git-tarballs branch 2 times, most recently from 768d66a to f621000 Compare June 12, 2023 11:03

boegel previously requested changes Jun 13, 2023

View reviewed changes

test/framework/filetools.py Outdated Show resolved Hide resolved

Rovanion force-pushed the reproducible-git-tarballs branch 3 times, most recently from 9daa03c to bf4f3f6 Compare June 19, 2023 15:15

boegel added the change label Jun 20, 2023

Rovanion changed the base branch from develop to 5.0.x July 5, 2023 15:49

boegel changed the title ~~Change tar command used in get_source_tarball_from_git to get reproducible tarballs~~ change tar command used in get_source_tarball_from_git to get reproducible tarballs Jan 17, 2024

lexming mentioned this pull request Feb 14, 2024

{data}[gfbf/2023a] polars v0.19.19 easybuilders/easybuild-easyconfigs#19770

Merged

4 tasks

lexming added 3 commits February 29, 2024 01:32

Merge branch '5.0.x' into PR4248

88de22e

make reproducible archives only of git repos without .git dir and res…

8d9e14e

…et timestamps with touch

update FileToolsTest.test_github_get_source_tarball_from_git tests wi…

46e61c8

…th reproducible tar commands

lexming mentioned this pull request Feb 29, 2024

Update for PR 4248 Rovanion/easybuild-framework#1

Merged

lexming previously requested changes Feb 29, 2024

View reviewed changes

lexming and others added 2 commits February 29, 2024 09:56

remove TODO about sort option in tar as it is not supported across im…

8c76561

…plementations

Merge pull request #1 from lexming/PR4248

e733cf4

Update for PR 4248

lexming reviewed Mar 1, 2024

View reviewed changes

filetools: Fix missing raw string

4363f9f

for \; in get_source_tarball_from_git.

Merge branch '5.0.x' into sync-pr-4248

0637f1f

lexming reviewed Mar 6, 2024

View reviewed changes

Merge pull request #2 from lexming/sync-pr-4248

e1f95b1

Sync PR#4248 with 5.0.x

lexming approved these changes Mar 18, 2024

View reviewed changes

lexming merged commit db2ab3a into easybuilders:5.0.x Mar 18, 2024
32 checks passed

akesandgren mentioned this pull request Apr 3, 2024

{data}[gfbf/2023a] polars v0.20.2 easybuilders/easybuild-easyconfigs#20275

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

change `tar` command used in `get_source_tarball_from_git` to get reproducible tarballs #4248

change `tar` command used in `get_source_tarball_from_git` to get reproducible tarballs #4248

Rovanion commented Apr 20, 2023

Rovanion commented May 11, 2023 via email

Rovanion commented May 11, 2023 via email

boegel commented Jun 7, 2023

Rovanion commented Jun 7, 2023

Rovanion commented Jun 13, 2023

Rovanion commented Jun 19, 2023

Rovanion commented Jun 21, 2023

Rovanion commented Jul 4, 2023

boegel commented Jul 5, 2023

jfgrimm commented Dec 20, 2023

boegel commented Jan 3, 2024

Rovanion commented Jan 8, 2024

lexming commented Feb 14, 2024 •

edited

lexming commented Feb 14, 2024

Rovanion commented Feb 28, 2024

lexming left a comment •

edited

lexming Mar 1, 2024

boegel commented Mar 2, 2024

lexming left a comment •

edited

boegel commented Mar 11, 2024

Rovanion commented Mar 18, 2024

lexming left a comment

Rovanion commented Mar 20, 2024 via email

	'-exec', 'touch', '-t 197001010100', '{}', '\;', '\|',
	'-exec', 'touch', '-t 197001010100', '{}', r'\;', '\|',

change tar command used in get_source_tarball_from_git to get reproducible tarballs #4248

change tar command used in get_source_tarball_from_git to get reproducible tarballs #4248

Conversation

Rovanion commented Apr 20, 2023

Rovanion commented May 11, 2023 via email

Rovanion commented May 11, 2023 via email

boegel commented Jun 7, 2023

Rovanion commented Jun 7, 2023

Rovanion commented Jun 13, 2023

Rovanion commented Jun 19, 2023

Rovanion commented Jun 21, 2023

Rovanion commented Jul 4, 2023

boegel commented Jul 5, 2023

jfgrimm commented Dec 20, 2023

boegel commented Jan 3, 2024

Rovanion commented Jan 8, 2024

lexming commented Feb 14, 2024 • edited

lexming commented Feb 14, 2024

Rovanion commented Feb 28, 2024

lexming left a comment • edited

Choose a reason for hiding this comment

lexming Mar 1, 2024

Choose a reason for hiding this comment

boegel commented Mar 2, 2024

lexming left a comment • edited

Choose a reason for hiding this comment

boegel commented Mar 11, 2024

Rovanion commented Mar 18, 2024

lexming left a comment

Choose a reason for hiding this comment

Rovanion commented Mar 20, 2024 via email

change `tar` command used in `get_source_tarball_from_git` to get reproducible tarballs #4248

change `tar` command used in `get_source_tarball_from_git` to get reproducible tarballs #4248

lexming commented Feb 14, 2024 •

edited

lexming left a comment •

edited

lexming left a comment •

edited