bpo-29708: support SOURCE_DATE_EPOCH env var in py_compile

to allow for reproducible builds of python packages

See https://reproducible-builds.org/ for why this is good
and https://reproducible-builds.org/specs/source-date-epoch/
for the definition of this variable.

Background:
In some distributions like openSUSE, binary rpms contain precompiled .pyc files.

And packages like amqp or twisted dynamically generate .py files at build time
so those have the current time and that timestamp gets embedded
into the .pyc file header.
When we then adapt file timestamps in rpms to be constant,
the timestamp in the .pyc header will no more match
the .py timestamp in the filesystem.
The software will still work, but it will not use the .pyc file as it should.

Doc/library/py_compile.rst

@@ -53,6 +53,10 @@ byte-code cache files in the directory containing the source code.
    :func:`compile` function.  The default of ``-1`` selects the optimization
    level of the current interpreter.
 
+   If the SOURCE_DATE_EPOCH environment variable is set, the .py file mtime
+   and timestamp entry in .pyc file header, will be limited to this value.
+   See https://reproducible-builds.org/specs/source-date-epoch/ for more info.
+
    .. versionchanged:: 3.2
       Changed default value of *cfile* to be :PEP:`3147`-compliant.  Previous
       default was *file* + ``'c'`` (``'o'`` if optimization was enabled).

Lib/py_compile.py

@@ -137,6 +137,10 @@ def compile(file, cfile=None, dfile=None, doraise=False, optimize=-1):
     except FileExistsError:
         pass
     source_stats = loader.path_stats(file)
+    sde = os.environ.get('SOURCE_DATE_EPOCH')
+    if sde and source_stats['mtime'] > int(sde):
+        source_stats['mtime'] = int(sde)
+        os.utime(file, (source_stats['mtime'], source_stats['mtime']))
     bytecode = importlib._bootstrap_external._code_to_bytecode(
             code, source_stats['mtime'], source_stats['size'])
     mode = importlib._bootstrap_external._calc_mode(file)

Lib/test/test_py_compile.py

@@ -98,6 +98,20 @@ def test_bad_coding(self):
         self.assertFalse(os.path.exists(
             importlib.util.cache_from_source(bad_coding)))
 
+    def test_source_date_epoch(self):
+        testtime = 123456789
+        with support.EnvironmentVarGuard() as env:
+            env["SOURCE_DATE_EPOCH"] = str(testtime)
+            py_compile.compile(self.source_path, self.pyc_path)
+        self.assertTrue(os.path.exists(self.pyc_path))
+        self.assertFalse(os.path.exists(self.cache_path))
+        statinfo = os.stat(self.source_path)
+        self.assertEqual(statinfo.st_mtime, testtime)
+        with open(self.pyc_path, "rb") as f:
+            f.read(4) # skip the magic .pyc number
+            timebytes = f.read(4) # read timestamp from pyc header
+        self.assertEqual(testtime, int.from_bytes(timebytes, 'little'))
+
     @unittest.skipIf(sys.flags.optimize > 0, 'test does not work with -O')
     def test_double_dot_no_clobber(self):
         # http://bugs.python.org/issue22966

Misc/NEWS.d/next/Build/2017-09-07-04-20-49.bpo-29708.YCaHEx.rst

@@ -0,0 +1,2 @@
+If the SOURCE_DATE_EPOCH environment variable is set,
+py_compile will use it to override the timestamps it puts into .pyc files.