Preserving timestamps on copy

[akhmerov]

We're using timestamps to determine whether the .c files generated with cython are up to date with the source .pyx files. However pip seems to modify timestamps (I suppose when copying the files from cache dir to build dir). Would it be possible to preserve the timestamps on copy?

[srinivasreddy]

@xavfernandez I would like to take this up. Could you point me where this happens?

[xavfernandez]

I'm not really sure what the use case of @akhmerov is.
But if it is for pip install ./some_directory then the copy happens here: https://github.com/pypa/pip/blob/30f4af0/pip/download.py#L671

[akhmerov]

We observed this behavior when using pypi install. I believe the timestamps get modified when files are copied from cache dir to build dir. The package suffering from the issue was kwant==1.1.0.

[xavfernandez]

@akhmerov Correct me if I'm wrong.

For example, the lapack.pyx is used to produce the lapack.c file and since in the .tar.gz file the timestamp of lapack.pyx is smaller than the one of lapack.c, it means the lapack.c is up-to-date and does not need to be regenerated.

On PyPI, kwant is only available as a .tar.gz which would be extracted directly in the build directory.
And our untar_file function (https://github.com/pypa/pip/blob/98ee8761/pip/utils/__init__.py#L505-L577) does not update the modification time.

[grothesque]

To reproduce the problem, execute something like

virtualenv foo
. foo/bin/activate
pip install --no-clean kwant==1.1.0

On my (and also @akhmerov's) machine, the build fails with a message like this:

************************************ Error ************************************
Some Cython source files are newer than files that should have
 been derived from them, but Cython is not installed.

The reason is that all the files have the same timestamp:

(foo)cwg@neron:/tmp/foo% ls -lt /tmp/pip-build-kYopVI/kwant/kwant
total 1376
-rw-rw-r-- 1 cwg cwg 1200152 Dec 18 18:06 _system.c
-rw-rw-r-- 1 cwg cwg   12192 Dec 18 18:06 _system.pyx
drwxrwxr-x 3 cwg cwg     360 Dec 18 18:06 linalg
-rw-rw-r-- 1 cwg cwg    3519 Dec 18 18:06 digest.py
drwxrwxr-x 3 cwg cwg     140 Dec 18 18:06 physics
-rw-rw-r-- 1 cwg cwg   67709 Dec 18 18:06 plotter.py
-rw-rw-r-- 1 cwg cwg    9836 Dec 18 18:06 system.py
-rw-rw-r-- 1 cwg cwg    3076 Dec 18 18:06 _common.py
-rw-rw-r-- 1 cwg cwg      60 Dec 18 18:06 _kwant_version.py
-rw-rw-r-- 1 cwg cwg   52592 Dec 18 18:06 builder.py
drwxrwxr-x 4 cwg cwg     380 Dec 18 18:06 graph
drwxrwxr-x 2 cwg cwg     180 Dec 18 18:06 tests
-rw-rw-r-- 1 cwg cwg    2019 Dec 18 18:06 __init__.py
-rw-rw-r-- 1 cwg cwg   27122 Dec 18 18:06 lattice.py
-rw-rw-r-- 1 cwg cwg    9454 Dec 18 18:06 rmt.py
drwxrwxr-x 3 cwg cwg     160 Dec 18 18:06 solvers

If you download and untar https://pypi.python.org/packages/source/k/kwant/kwant-1.1.0.tar.gz manually, the timestamps are of course preserved.

Note that we now work around this problem by disabling cythonization by default. But I think it would be still nice if the problem was fixed in pip.