Set SOURCE_DATE_EPOCH #274

harens · 2022-06-29T15:07:13Z

Closes: https://trac.macports.org/ticket/59672

This PR allows base to define the SOURCE_DATE_EPOCH environment variable to assist with reproducible builds. The value can be manually set in a portfile via the source_date_epoch variable.

If the variable isn't assigned, its value is set to the greatest Unix timestamp of the files present in the source code (credit to Ryan for implementation details).

harens · 2022-06-29T17:41:18Z

Just to show how powerful SOURCE_DATE_EPOCH is, I've pushed a commit that standardises the file modification timestamps produced by MacPorts archives.

I tested this with a port I maintain, ioping. I ran sudo port -ds archive ioping twice, and then compared the archives using diffoscope.

Before this PR: File timestamps are different, so the archives are as well.

$ gsha256sum ioping-1.3_0.darwin_21.arm64\ 2.tbz2
6135d8dff9c87a3074fbf1dbaeea13db4ab0ef7774bcd5072e1c5196246d7396  ioping-1.3_0.darwin_21.arm64 2.tbz2
$ gsha256sum ioping-1.3_0.darwin_21.arm64.tbz2
6958d0826cc9c0d2775f3d9f9c20372f2c2fa5369caa44f80b7fbc4746504743  ioping-1.3_0.darwin_21.arm64.tbz2

Afterwards: The archives are completely identical.

$ gsha256sum ioping-1.3_0.darwin_21.arm64\ 2.tbz2
2406a76abc668c91ef9ec066ab9735dbee55ceba615ea6ec15aff6537d6d71ce  ioping-1.3_0.darwin_21.arm64 2.tbz2
$ gsha256sum ioping-1.3_0.darwin_21.arm64.tbz2
2406a76abc668c91ef9ec066ab9735dbee55ceba615ea6ec15aff6537d6d71ce  ioping-1.3_0.darwin_21.arm64.tbz2

Below is the diffoscope output showing how the timestamps in the original archives are different:

@@ -1,14 +1,14 @@
-drwxr-xr-x   0 root         (0) wheel        (0)        0 2023-02-25 12:13:31.000000 ./
--rw-r--r--   0 root         (0) wheel        (0)      287 2023-02-25 12:13:31.000000 ./+STATE
+drwxr-xr-x   0 root         (0) wheel        (0)        0 2023-02-25 12:13:03.000000 ./
+-rw-r--r--   0 root         (0) wheel        (0)      287 2023-02-25 12:13:03.000000 ./+STATE
 -rw-r--r--   0 root         (0) wheel        (0)     1039 2022-09-04 11:18:25.000000 ./+PORTFILE
--rw-r--r--   0 root         (0) wheel        (0)      405 2023-02-25 12:13:31.000000 ./+CONTENTS
--rw-r--r--   0 root         (0) wheel        (0)      152 2023-02-25 12:13:31.000000 ./+DESC
--rw-r--r--   0 root         (0) wheel        (0)       39 2023-02-25 12:13:31.000000 ./+COMMENT
-drwxr-xr-x   0 root         (0) wheel        (0)        0 2023-02-25 12:13:31.000000 ./opt/
-drwxr-xr-x   0 root         (0) wheel        (0)        0 2023-02-25 12:13:31.000000 ./opt/local/
-drwxr-xr-x   0 root         (0) wheel        (0)        0 2023-02-25 12:13:31.000000 ./opt/local/bin/
-drwxr-xr-x   0 root         (0) wheel        (0)        0 2023-02-25 12:13:31.000000 ./opt/local/share/
-drwxr-xr-x   0 root         (0) wheel        (0)        0 2023-02-25 12:13:31.000000 ./opt/local/share/man/
-drwxr-xr-x   0 root         (0) wheel        (0)        0 2023-02-25 12:13:31.000000 ./opt/local/share/man/man1/
--r--r--r--   0 root         (0) wheel        (0)     3603 2023-02-25 12:13:31.000000 ./opt/local/share/man/man1/ioping.1.gz
--rwxr-xr-x   0 root         (0) wheel        (0)    73539 2023-02-25 12:13:31.000000 ./opt/local/bin/ioping
+-rw-r--r--   0 root         (0) wheel        (0)      405 2023-02-25 12:13:03.000000 ./+CONTENTS
+-rw-r--r--   0 root         (0) wheel        (0)      152 2023-02-25 12:13:03.000000 ./+DESC
+-rw-r--r--   0 root         (0) wheel        (0)       39 2023-02-25 12:13:03.000000 ./+COMMENT
+drwxr-xr-x   0 root         (0) wheel        (0)        0 2023-02-25 12:13:03.000000 ./opt/
+drwxr-xr-x   0 root         (0) wheel        (0)        0 2023-02-25 12:13:03.000000 ./opt/local/
+drwxr-xr-x   0 root         (0) wheel        (0)        0 2023-02-25 12:13:03.000000 ./opt/local/bin/
+drwxr-xr-x   0 root         (0) wheel        (0)        0 2023-02-25 12:13:03.000000 ./opt/local/share/
+drwxr-xr-x   0 root         (0) wheel        (0)        0 2023-02-25 12:13:03.000000 ./opt/local/share/man/
+drwxr-xr-x   0 root         (0) wheel        (0)        0 2023-02-25 12:13:03.000000 ./opt/local/share/man/man1/
+-r--r--r--   0 root         (0) wheel        (0)     3603 2023-02-25 12:13:03.000000 ./opt/local/share/man/man1/ioping.1.gz
+-rwxr-xr-x   0 root         (0) wheel        (0)    73539 2023-02-25 12:13:03.000000 ./opt/local/bin/ioping

neverpanic

Thanks for working on this, this is and important step and a good thing to have. I have a few concerns, though.

src/port1.0/portextract.tcl

neverpanic · 2022-06-30T18:24:03Z

src/port1.0/portinstall.tcl

+    # Set standardised file modification times
+    fs-traverse file $destpath {
+        # Keep the original times for files that have not been modified during the build
+        if {[file mtime $file] > $source_date_epoch} {
+            file mtime $file $source_date_epoch
+        } 
+    }


I'm not sure that will work for all ports as we expect it to. As far as I recall, Python scripts check that the timestamps of their associated .pyc files is strictly larger than the timestamp of the script itself.

For python scripts generated during the build (e.g., because of some autotools replacement magic), this would mean that we might end up shipping a pre-compiled .pyc file that ends up being ignored by all users.

Maybe we should provide an option to turn this behavior off for ports where it would have adverse effects, or add a method to override the time for certain files?

Sorry it's taken me this long to get to this @neverpanic. Appreciate the feedback.

As far as I recall, Python scripts check that the timestamps of their associated .pyc files is strictly larger than the timestamp of the script itself.

Just looking it up, it seems like it should be fine for all supported python versions (see PEP 552). Setting SOURCE_DATE_EPOCH triggers using hash-based pyc files (python/cpython@ccbe581).

However, I completely agree that there might be other scenarios where this has adverse effects.

Maybe we should provide an option to turn this behavior off for ports where it would have adverse effects, or add a method to override the time for certain files?

I've added a reproducible_timestamps option. It's default is set to true, but this can be changed if that's the general consensus.

harens · 2023-02-08T19:32:51Z

~~Seem to be a lot of "failed to extract" errors:~~

error: failed to extract dependencies-b: /private/tmp/macports-tests/opt/local/var/macports/build/_users_runner_work_macports-base_macports-base_tests_test_dependencies-b/dependencies-b/work/dependencies-b-1: no such file or directory
...
Error: Failed to extract envvariables: /private/tmp/macports-tests/opt/local/var/macports/build/_Users_runner_work_macports-base_macports-base_tests_test_envvariables/envvariables/work/envvariables-1: no such file or directory
...
etc.

~~Any help debugging appreciated.~~

EDIT: Fix incoming.

Closes: https://trac.macports.org/ticket/59672

ryandesign · 2023-05-26T05:52:34Z

src/port1.0/portmain.tcl

@@ -113,6 +114,7 @@ default destroot {${destpath}}
 default filesdir files
 default revision 0
 default epoch 0
+default reproducible_timestamps 1


Although using 1 probably works, MacPorts convention is to use yes and no for Booleans.

Suggested change

default reproducible_timestamps 1

default reproducible_timestamps yes

ryandesign · 2023-05-26T05:53:36Z

src/port1.0/portinstall.tcl

@@ -308,6 +308,19 @@ proc portinstall::create_archive {location archive.type} {
    puts $fd "@cxx_stdlib_overridden ${portinstall::cxx_stdlib_overridden}"
    close $fd

+    if {[get_statefile_value "SOURCE_DATE_EPOCH" $target_state_fd source_date_epoch] != 0 && $reproducible_timestamps == 1} {


This will need a corresponding change.

ryandesign · 2023-05-26T05:56:47Z

src/port1.0/portinstall.tcl

+        fs-traverse file $destpath {
+            if {[file type $file] == "link"} {
+                # file mtime doesn't work on symbolic links
+                exec touch -ht $source_date_epoch $file


I would rather see the MacPorts touch procedure be used than using the external touch command. The MacPorts touch procedure doesn't support -h yet but perhaps that could be added.

ryandesign · 2023-05-26T06:17:52Z

Took me a minute of reading the code, but I see now that you implemented:

setting SOURCE_DATE_EPOCH during all phases, and
changing the timestamp of all files in the archive to that value

In the ticket I was only proposing doing (1) and I don't know what the consequences are of doing (2).

harens · 2023-06-27T14:59:53Z

Took me a minute of reading the code, but I see now that you implemented:
1. setting `SOURCE_DATE_EPOCH` during all phases, and

2. changing the timestamp of all files in the archive to that value
In the ticket I was only proposing doing (1) and I don't know what the consequences are of doing (2).

Thank you for the review Ryan, you make a very valid point. I think the easiest thing for now is to simply add SOURCE_DATE_EPOCH to base as was discussed in the original ticket. Standardising the timestamps can be discussed at a later date.

neverpanic reviewed Jun 30, 2022

View reviewed changes

harens force-pushed the source_date_epoch branch from 0d7ec86 to 0e6ae65 Compare February 8, 2023 18:56

Set SOURCE_DATE_EPOCH

3d5a913

Closes: https://trac.macports.org/ticket/59672

harens force-pushed the source_date_epoch branch from 0e6ae65 to fce6e4b Compare February 8, 2023 19:58

harens requested review from jmroot, ryandesign and neverpanic February 25, 2023 12:32

ryandesign reviewed May 26, 2023

View reviewed changes

harens force-pushed the source_date_epoch branch from fce6e4b to 3d5a913 Compare June 27, 2023 14:49

jmroot closed this in 4dfabb9 May 12, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Set SOURCE_DATE_EPOCH #274

Set SOURCE_DATE_EPOCH #274

harens commented Jun 29, 2022

harens commented Jun 29, 2022 •

edited

Loading

neverpanic left a comment

neverpanic Jun 30, 2022

harens Feb 8, 2023

harens commented Feb 8, 2023 •

edited

Loading

ryandesign May 26, 2023

ryandesign May 26, 2023

ryandesign May 26, 2023

ryandesign commented May 26, 2023

harens commented Jun 27, 2023

	default reproducible_timestamps 1
	default reproducible_timestamps yes

Set SOURCE_DATE_EPOCH #274

Set SOURCE_DATE_EPOCH #274

Conversation

harens commented Jun 29, 2022

harens commented Jun 29, 2022 • edited Loading

neverpanic left a comment

Choose a reason for hiding this comment

neverpanic Jun 30, 2022

Choose a reason for hiding this comment

harens Feb 8, 2023

Choose a reason for hiding this comment

harens commented Feb 8, 2023 • edited Loading

ryandesign May 26, 2023

Choose a reason for hiding this comment

ryandesign May 26, 2023

Choose a reason for hiding this comment

ryandesign May 26, 2023

Choose a reason for hiding this comment

ryandesign commented May 26, 2023

harens commented Jun 27, 2023

harens commented Jun 29, 2022 •

edited

Loading

harens commented Feb 8, 2023 •

edited

Loading