Please make the generated requires.txt files reproducible

Originally reported by: lambyuk (Bitbucket: lambyuk, GitHub: Unknown)

---

Whilst working on the "reproducible builds" effort [0], we noticed that
python-setuptools generates non-deterministic output when creating
requires.txt files:

│   │   │   │  [all]
│   │   │   │ -fake-factory==0.5.2
│   │   │   │ -pytz
│   │   │   │  numpy>=1.9.0
│   │   │   │  pytz
│   │   │   │ +pytz
│   │   │   │  django>=1.7
│   │   │   │ +fake-factory==0.5.2


Patch attached.

 [0] https://wiki.debian.org/ReproducibleBuilds

Filed in Debian as https://bugs.debian.org/804249

---

• Bitbucket: https://bitbucket.org/pypa/setuptools/issue/458

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

I agree with the sentiment here, but I'm unsure about the patch. I know order of resolution is important, so sorting lexicographically could violate other expectations (or goals). I'd prefer if setuptools could capture the user's order and retain that. I haven't had a chance to investigate this in more detail, so I don't know what implications that retention would have. Would you be willing to explore that possibility?

Original comment by lambyuk (Bitbucket: lambyuk, GitHub: Unknown):

---

Three points:

First, I doubt it makes any difference whatsoever.

Second, sorting lexographically and possibly introducing unexpected expectations would be at least no worse than the status quo which is—as the bug title implies—a non-determinstic output. Nobody can really be relying on the current behaviour as it's inherently unreliable!

Lastly, nobody should be relying on the current behaviour without a reams of comments. Just far too bizarre and/or the software is broken in some other, horrible, way.

[anthraxx]

@jaraco any updates on this? debian is already applying a custom patch to this issue. currently this is blocking reproducible python builds for our distro. I hope we can get some movement here after 2 years

[jaraco]

Apologies for the delay. It’s the unfortunate situation that passing time affords little benefit for open Setuptools tickets (with the exception of regressions). Pull Requests get the lion’s share of attention.

But I’ll see if I can knock this one out.

[jaraco]

I'm trying to understand the issue. By using the patch as a starting point, I'm looking at how dist.install_requires is generated, and it seems to be taking the sequence of requirements directly from the package declaration, and if that declaration was deterministically ordered, so also would be the requires.txt file.

I'll see if I can replicate the finding as reported, but my suspicion is it happens only if the package being built uses a non-deterministically ordered sequence (set or dict) for their install_requires declaration.

[jaraco]

Although i pushed that commit with the 'fix' (application of the patch from the bitbucket ticket), I've stripped that commit and I'm pushing the tests separately. Using c461d51, I've determined that the issue exists only when the requirements were declared by the user in a non-deterministically-ordered sequence:

$ tox --hashseed 1983024029 -- -v -k 'install_requires and deterministic and not in_setup_cfg'
python develop-inst-nodeps: /Users/jaraco/Dropbox/code/main/setuptools
python installed: apipkg==1.4,contextlib2==0.5.5,execnet==1.5.0,flake8==3.5.0,mccabe==0.6.1,mock==2.0.0,path.py==10.5,pbr==3.1.1,py==1.5.2,pycodestyle==2.3.1,pyflakes==1.6.0,pytest==3.2.5,pytest-fixture-config==1.2.11,pytest-flake8==0.9.1,pytest-shutil==1.2.11,pytest-virtualenv==1.2.11,six==1.11.0,virtualenv==15.1.0
python runtests: PYTHONHASHSEED='1983024029'
python runtests: commands[0] | py.test -v -k install_requires and deterministic and not in_setup_cfg
====================================== test session starts =======================================
platform darwin -- Python 3.6.3, pytest-3.2.5, py-1.5.2, pluggy-0.4.0 -- /Users/jaraco/Dropbox/code/main/setuptools/.tox/python/bin/python3
cachedir: .cache
rootdir: /Users/jaraco/Dropbox/code/main/setuptools, inifile: pytest.ini
plugins: virtualenv-1.2.11, shutil-1.2.11, flake8-0.9.1
collected 390 items / 2 skipped                                                                   

setuptools/tests/test_egg_info.py::TestEggInfo::test_requires[install_requires_deterministic] PASSED
setuptools/tests/test_egg_info.py::TestEggInfo::test_requires[install_requires_set_deterministic] FAILED
==================================== short test summary info =====================================
SKIP [2] /Users/jaraco/Dropbox/code/main/setuptools/setuptools/tests/test_msvc.py:18: could not import 'distutils.msvc9compiler'

============================================ FAILURES ============================================
_________________ TestEggInfo.test_requires[install_requires_set_deterministic] __________________

self = <setuptools.tests.test_egg_info.TestEggInfo object at 0x10d9369b0>
tmpdir_cwd = local('/Users/jaraco/Dropbox/code/main/setuptools')
env = '/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/setuptools-test.ro7wax9k'
requires = 'install_requires={"fake-factory==0.5.2", "pytz"}', use_setup_cfg = False
expected_requires = 'fake-factory==0.5.2\npytz\n', install_cmd_kwargs = {}

    @RequiresTestHelper.parametrize(
        # Format of a test:
        #
        # id
        # install_cmd_kwargs [optional]
        #
        # requires block (when used in setup.py)
        #
        # requires block (when used in setup.cfg)
        #
        # expected contents of requires.txt
    
        '''
            install_requires_deterministic
    
            install_requires=["fake-factory==0.5.2", "pytz"]
    
            [options]
            install_requires =
                fake-factory==0.5.2
                pytz
    
            fake-factory==0.5.2
            pytz
            ''',
    
        '''
            install_requires_set_deterministic
    
            install_requires={{"fake-factory==0.5.2", "pytz"}}
    
            [options]
            install_requires =
                fake-factory==0.5.2
                pytz
    
            fake-factory==0.5.2
            pytz
            ''',
    
        '''
            install_requires_with_marker
    
            install_requires=["barbazquux;{mismatch_marker}"],
    
            [options]
            install_requires =
                barbazquux; {mismatch_marker}
    
            [:{mismatch_marker_alternate}]
            barbazquux
            ''',
    
        '''
            install_requires_with_extra
            {'cmd': ['egg_info']}
    
            install_requires=["barbazquux [test]"],
    
            [options]
            install_requires =
                barbazquux [test]
    
            barbazquux[test]
            ''',
    
        '''
            install_requires_with_extra_and_marker
    
            install_requires=["barbazquux [test]; {mismatch_marker}"],
    
            [options]
            install_requires =
                barbazquux [test]; {mismatch_marker}
    
            [:{mismatch_marker_alternate}]
            barbazquux[test]
            ''',
    
        '''
            setup_requires_with_markers
    
            setup_requires=["barbazquux;{mismatch_marker}"],
    
            [options]
            setup_requires =
                barbazquux; {mismatch_marker}
    
            ''',
    
        '''
            tests_require_with_markers
            {'cmd': ['test'], 'output': "Ran 0 tests in"}
    
            tests_require=["barbazquux;{mismatch_marker}"],
    
            [options]
            tests_require =
                barbazquux; {mismatch_marker}
    
            ''',
    
        '''
            extras_require_with_extra
            {'cmd': ['egg_info']}
    
            extras_require={{"extra": ["barbazquux [test]"]}},
    
            [options.extras_require]
            extra = barbazquux [test]
    
            [extra]
            barbazquux[test]
            ''',
    
        '''
            extras_require_with_extra_and_marker_in_req
    
            extras_require={{"extra": ["barbazquux [test]; {mismatch_marker}"]}},
    
            [options.extras_require]
            extra =
                barbazquux [test]; {mismatch_marker}
    
            [extra]
    
            [extra:{mismatch_marker_alternate}]
            barbazquux[test]
            ''',
    
        # FIXME: ConfigParser does not allow : in key names!
        '''
            extras_require_with_marker
    
            extras_require={{":{mismatch_marker}": ["barbazquux"]}},
    
            @xfail
            [options.extras_require]
            :{mismatch_marker} = barbazquux
    
            [:{mismatch_marker}]
            barbazquux
            ''',
    
        '''
            extras_require_with_marker_in_req
    
            extras_require={{"extra": ["barbazquux; {mismatch_marker}"]}},
    
            [options.extras_require]
            extra =
                barbazquux; {mismatch_marker}
    
            [extra]
    
            [extra:{mismatch_marker_alternate}]
            barbazquux
            ''',
    
        '''
            extras_require_with_empty_section
    
            extras_require={{"empty": []}},
    
            [options.extras_require]
            empty =
    
            [empty]
            ''',
        # Format arguments.
        invalid_marker=invalid_marker,
        mismatch_marker=mismatch_marker,
        mismatch_marker_alternate=mismatch_marker_alternate,
    )
    def test_requires(
            self, tmpdir_cwd, env, requires, use_setup_cfg,
            expected_requires, install_cmd_kwargs):
        self._setup_script_with_requires(requires, use_setup_cfg)
        self._run_install_command(tmpdir_cwd, env, **install_cmd_kwargs)
        egg_info_dir = os.path.join('.', 'foo.egg-info')
        requires_txt = os.path.join(egg_info_dir, 'requires.txt')
        if os.path.exists(requires_txt):
            with open(requires_txt) as fp:
                install_requires = fp.read()
        else:
            install_requires = ''
>       assert install_requires.lstrip() == expected_requires
E       AssertionError: assert 'pytz\nfake-factory==0.5.2\n' == 'fake-factory==0.5.2\npytz\n'
E         + fake-factory==0.5.2
E           pytz
E         - fake-factory==0.5.2

/Users/jaraco/Dropbox/code/main/setuptools/setuptools/tests/test_egg_info.py:407: AssertionError
====================================== 388 tests deselected ======================================
================= 1 failed, 1 passed, 2 skipped, 388 deselected in 15.34 seconds =================
ERROR: InvocationError: '/Users/jaraco/Dropbox/code/main/setuptools/.tox/python/bin/py.test -v -k install_requires and deterministic and not in_setup_cfg'
____________________________________________ summary _____________________________________________
ERROR:   python: commands failed

Therefore, the fix should be to either disallow such declarations or to only re-order the declaration when it's not provided in a deterministic sequence.

[jaraco]

According to the docs, install_requires takes a string or list of strings, so the packages that are supplying sets are invalid.