Restructure requirements files

* Use suffixed instead of prefixed sub-requirements files to group
  them alphabetically in the file tree.
* Layer requirements files akin to the in-toto project
  (see in-toto/in-toto#294). The hierarchy is:

  - *requirements.in*
    tuf runtime requirements, including optional requirements
    (pynacl and cyrptography)

  - *requirements.txt*
    pinned tuf runtime requirements, including optional and
    transitive (1 level deep) requirements and their hashes.

    The file is generated semi-automatically using pip-compile
    and a bash script (see document header), based off of
    requirements.in, combining requirements from all supported
    Python versions.

    This file should be auto-updated, by e.g. dependabot, and be used
    for ci/cd tests, to catch issues with new dependencies.

  - *requirements-test.txt*
    additional test runtime requirements

  - *requirements-tox.txt*
    combines requirements.txt, requirements-test.txt and additional
    test tools (for linting and coverage), i.e. everything that is
    needed in each tox environment to run the tests.

  - *requirements-dev.txt*
    lists tox for local development and testing, and also
    requirements-tox.txt and tuf in editable mode to run
    the test suite or individual tests directly.

* Removes an obsolete version constraint on coverage

Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>

docs/CONTRIBUTORS.rst

@@ -102,7 +102,7 @@ To work on the TUF project, it's best to perform a development install.
 
 ::
 
-    $ pip install -r dev-requirements.txt
+    $ pip install -r requirements-dev.txt
 
 
 Testing
@@ -132,9 +132,9 @@ a *venv*), and then install ``tuf`` in editable mode too (in the same *venv*).
 ::
 
     $ cd path/to/securesystemslib
-    $ pip install -r dev-requirements.txt
+    $ pip install -r requirements-dev.txt
     $ cd path/to/tuf
-    $ pip install -r dev-requirements.txt
+    $ pip install -r requirements-dev.txt
 
 
 With `tox <https://testrun.org/tox/>`_ the test suite can be executed in a

requirements-dev.txt

@@ -0,0 +1,8 @@
+# Install tox for local testing, but also everything that tox would install
+# in a test environment, so that we can run the test suite or individual tests
+# directly in the development environment as well.
+tox
+-r requirements-tox.txt
+
+# Install tuf in editable mode
+-e .

requirements-test.txt

@@ -0,0 +1,2 @@
+# Runtime requirements for test suite
+mock; python_version < "3.3"

requirements-tox.txt

@@ -0,0 +1,10 @@
+# tuf runtime dependencies (pinned and with hashes)
+-r requirements.txt
+
+# tuf test suite runtime dependencies
+-r requirements-test.txt
+
+# test tools for linting and test coverage measurement
+coverage
+pylint
+bandit

requirements.txt

@@ -1,26 +1,43 @@
+# These pinned requirements are the combined results of `pip-compile` over
+# "requirements.in" in each supported Python version.
 #
-# This file is (mostly) autogenerated by pip-compile.
-# To update, run:
+# - Below script may be used (copy-paste to bash removing leading '#') to
+#   re-generate this file, e.g. if `requirements.in` is updated.
+# - Manually add version directives like ' ; python_version >= "3.0"' based on
+#   which dependencies are required for Python2 vs Python3 (or other version
+#   constraints).
+# - Version updates should be handled by a GitHub-integrated dependency monitor
+#   (e.g. Dependabot) that regularly scans PyPI for updates, patches this file,
+#   and submits a PR, which triggers CI/CD builds and should catch breaking
+#   updates.
 #
-#    pip-compile --generate-hashes --output-file requirements.txt requirements.in
+# # Gather pip-compile results for each supported Python version
+# for v in 2.7 3.5 3.6 3.7 3.8; do
+#   mkvirtualenv tuf-env-${v} -p python${v};
+#   pip install pip-tools;
+#   pip-compile requirements.in -n 2>&1 | grep -v "^#"  >> requirements.combined;
 #
-# Run pip-compile in both Python2 and Python3, and combine the dependencies,
-# adding the following as necessary:
-#   - Add version directives like ' ; python_version >= "3.0"' based on which
-#     dependencies are required for Python2 vs Python3 (or other version
-#     constraints).
-#   - Add ' # pyup: ignore' for any dependency that must remain outdated
-#     because later versions drop support for (e.g.) Python2.
+#   # Keep one venv as we need it below to add hashes
+#   if [ $v != 3.8 ]; then
+#     deactivate;
+#     rmvirtualenv tuf-env-${v};
+#   fi
+# done;
 #
-# Be sure to leave these comments at the top of the new file.
+# # Create pinned requirements file retaining this doc header
+# cat requirements.txt | grep "^#"  > requirements.tmp
+# mv requirements.tmp requirements.txt
+# cat requirements.combined | grep -v "^Dry-run," | sort -u >> requirements.txt
+# rm requirements.combined
 #
-asn1crypto==1.2.0 \
-    --hash=sha256:7bb1cc02a5620b3d72da4ba070bda2f44f0e61b44dee910a302eddff802b6fb5 \
-    --hash=sha256:87620880a477123e01177a1f73d0f327210b43a3cdbd714efcd2fa49a8d7b384    # via cryptography
-certifi==2019.9.11 \
-    --hash=sha256:e4f3620cfea4f83eedc95b24abd9cd56f3c4b146dd0177e83a21b4eb49e21e50 \
-    --hash=sha256:fd7c7c74727ddcf00e9acd26bba8da604ffec95bf1c2144e67aff7a8b50e6cef \
-    # via requests
+# # Add hashes now (we can't do it before, because of `sort -u`)
+# pip-compile --generate-hashes requirements.txt
+# deactivate
+# rmvirtualenv tuf-env-3.8
+#
+certifi==2019.11.28 \
+    --hash=sha256:017c25db2a153ce562900032d5bc68e9f191e44e9a0f762f373977de9df1fbb3 \
+    --hash=sha256:25b64c7da4cd7479594d035c08c2d809eb4aab3a26e5a990ea98cc450c320f1f
 cffi==1.13.2 \
     --hash=sha256:0b49274afc941c626b605fb59b59c3485c17dc776dc3cc7cc14aca74cc19cc42 \
     --hash=sha256:0e3ea92942cb1168e38c05c1d56b0527ce31f1a370f6117f1d490b8dcd6b3a04 \
@@ -33,6 +50,7 @@ cffi==1.13.2 \
     --hash=sha256:32a262e2b90ffcfdd97c7a5e24a6012a43c61f1f5a57789ad80af1d26c6acd97 \
     --hash=sha256:3c9fff570f13480b201e9ab69453108f6d98244a7f495e91b6c654a47486ba43 \
     --hash=sha256:415bdc7ca8c1c634a6d7163d43fb0ea885a07e9618a64bda407e04b04333b7db \
+    --hash=sha256:42194f54c11abc8583417a7cf4eaff544ce0de8187abaf5d29029c91b1725ad3 \
     --hash=sha256:4424e42199e86b21fc4db83bd76909a6fc2a2aefb352cb5414833c030f6ed71b \
     --hash=sha256:4a43c91840bda5f55249413037b7a9b79c90b1184ed504883b72c4df70778579 \
     --hash=sha256:599a1e8ff057ac530c9ad1778293c665cb81a791421f46922d80a86473c13346 \
@@ -53,14 +71,13 @@ cffi==1.13.2 \
     --hash=sha256:d75c461e20e29afc0aee7172a0950157c704ff0dd51613506bd7d82b718e7410 \
     --hash=sha256:dcd65317dd15bc0451f3e01c80da2216a31916bdcffd6221ca1202d96584aa25 \
     --hash=sha256:e570d3ab32e2c2861c4ebe6ffcad6a8abf9347432a37608fe1fbd157b3f0036b \
-    --hash=sha256:fd43a88e045cf992ed09fa724b5315b790525f2676883a6ea64e3263bae6549d    # via cryptography, pynacl
+    --hash=sha256:fd43a88e045cf992ed09fa724b5315b790525f2676883a6ea64e3263bae6549d
 chardet==3.0.4 \
     --hash=sha256:84ab92ed1c4d4f16916e05906b6b75a6c0fb5db821cc65e70cbd64a3e2a5eaae \
-    --hash=sha256:fc323ffcaeaed0e0a02bf4d117757b98aed530d9ed4531e3e15460124c106691 \
-    # via requests
-colorama==0.4.1 \
-    --hash=sha256:05eed71e2e327246ad6b38c540c4a3117230b19679b875190486ddd2d721422d \
-    --hash=sha256:f8ac84de7840f5b9c4e3347b3c1eaa50f7e49c2b07596221daec5edaabbd7c48
+    --hash=sha256:fc323ffcaeaed0e0a02bf4d117757b98aed530d9ed4531e3e15460124c106691
+colorama==0.4.3 \
+    --hash=sha256:7d73d2a99753107a36ac6b455ee49046802e59d9d076ef8e47b61499fa29afff \
+    --hash=sha256:e96da0d330793e2cb9485e9ddfd918d456036c7149416295932478192f4436a1
 cryptography==2.8 \
     --hash=sha256:02079a6addc7b5140ba0825f542c0869ff4df9a69c360e339ecead5baefa843c \
     --hash=sha256:1df22371fbf2004c6f64e927668734070a8953362cd8370ddd336774d6743595 \
@@ -89,23 +106,19 @@ enum34==1.1.6 \
     --hash=sha256:6bd0f6ad48ec2aa117d3d141940d484deccda84d4fcd884f5c3d93c23ecd8c79 \
     --hash=sha256:8ad8c4783bf61ded74527bffb48ed9b54166685e4230386a9ed9b1279e2df5b1 \
     ; python_version < "3.0"
-    # via cryptography
 idna==2.8 \
     --hash=sha256:c357b3f628cf53ae2c4c05627ecc484553142ca23264e593d327bcde5e9c3407 \
-    --hash=sha256:ea8b7f6188e6fa117537c3df7da9fc686d485087abf6ac197f9c46432f7e4a3c \
-    # via requests
-ipaddress==1.0.22 \
-    --hash=sha256:64b28eec5e78e7510698f6d4da08800a5c575caa4a286c93d651c5d3ff7b6794 \
-    --hash=sha256:b146c751ea45cad6188dd6cf2d9b757f6f4f8d6ffb96a023e6f2e26eea02a72c \
-    ; python_version < "3.0" # pyup: ignore
-    # via cryptography
+    --hash=sha256:ea8b7f6188e6fa117537c3df7da9fc686d485087abf6ac197f9c46432f7e4a3c
+ipaddress==1.0.23 \
+    --hash=sha256:6e0f4a39e66cb5bb9a137b00276a2eff74f93b71dcbdad6f10ff7df9d3557fcc \
+    --hash=sha256:b7f8e0369580bb4a24d5ba1d7cc29660a4a6987763faf1d8a8046830e020e7e2 \
+    ; python_version < "3.0"
 iso8601==0.1.12 \
     --hash=sha256:210e0134677cc0d02f6028087fee1df1e1d76d372ee1db0bf30bf66c5c1c89a3 \
     --hash=sha256:49c4b20e1f38aa5cf109ddcd39647ac419f928512c869dc01d5c7098eddede82 \
     --hash=sha256:bbbae5fb4a7abfe71d4688fd64bff70b91bbd74ef6a99d964bab18f7fdf286dd
 pycparser==2.19 \
-    --hash=sha256:a988718abfad80b6b157acce7bf130a30876d27603738ac39f140993246b25b3 \
-    # via cffi
+    --hash=sha256:a988718abfad80b6b157acce7bf130a30876d27603738ac39f140993246b25b3
 pynacl==1.3.0 \
     --hash=sha256:05c26f93964373fc0abe332676cb6735f0ecad27711035b9472751faa8521255 \
     --hash=sha256:0c6100edd16fefd1557da078c7a31e7b7d7a52ce39fdca2bec29d4f7b6e7600c \
@@ -116,6 +129,7 @@ pynacl==1.3.0 \
     --hash=sha256:30f36a9c70450c7878053fa1344aca0145fd47d845270b43a7ee9192a051bf39 \
     --hash=sha256:37aa336a317209f1bb099ad177fef0da45be36a2aa664507c5d72015f956c310 \
     --hash=sha256:4943decfc5b905748f0756fdd99d4f9498d7064815c4cf3643820c9028b711d1 \
+    --hash=sha256:53126cd91356342dcae7e209f840212a58dcf1177ad52c1d938d428eebc9fee5 \
     --hash=sha256:57ef38a65056e7800859e5ba9e6091053cd06e1038983016effaffe0efcd594a \
     --hash=sha256:5bd61e9b44c543016ce1f6aef48606280e45f892a928ca7068fba30021e9b786 \
     --hash=sha256:6482d3017a0c0327a49dddc8bd1074cc730d45db2ccb09c3bac1f8f32d1eb61b \
@@ -124,26 +138,25 @@ pynacl==1.3.0 \
     --hash=sha256:a39f54ccbcd2757d1d63b0ec00a00980c0b382c62865b61a505163943624ab20 \
     --hash=sha256:aabb0c5232910a20eec8563503c153a8e78bbf5459490c49ab31f6adf3f3a415 \
     --hash=sha256:bd4ecb473a96ad0f90c20acba4f0bf0df91a4e03a1f4dd6a4bdc9ca75aa3a715 \
+    --hash=sha256:bf459128feb543cfca16a95f8da31e2e65e4c5257d2f3dfa8c0c1031139c9c92 \
     --hash=sha256:e2da3c13307eac601f3de04887624939aca8ee3c9488a0bb0eca4fb9401fc6b1 \
     --hash=sha256:f67814c38162f4deb31f68d590771a29d5ae3b1bd64b75cf232308e5c74777e0
 python-dateutil==2.8.1 \
     --hash=sha256:73ebfe9dbf22e832286dafa60473e4cd239f8592f699aa5adaf10050e6e1823c \
-    --hash=sha256:75bb3f31ea686f1197762692a9ee6a7550b59fc6ca3a1f4b5d7e32fb98e2da2a    # via securesystemslib
+    --hash=sha256:75bb3f31ea686f1197762692a9ee6a7550b59fc6ca3a1f4b5d7e32fb98e2da2a
 requests==2.22.0 \
     --hash=sha256:11e007a8a2aa0323f5a921e9e6a2d7e4e67d9877e85773fba9ba6419025cbeb4 \
     --hash=sha256:9cf5292fcd0f598c671cfc1e0d7d1a7f13bb8085e9a590f48c010551dc6c4b31
-securesystemslib==0.12.2 \
-    --hash=sha256:39acbb3db6c3caa94d95a3369ffcc9d5563a04540c89874cc2f158706dbad6c1 \
-    --hash=sha256:f25541fc7226c3e9b830bb285598c6bbdc00d02eea1935575abffd03a45becbf
-six==1.13.0 \
-    --hash=sha256:1f1b7d42e254082a9db6279deae68afb421ceba6158efa6131de7b3003ee93fd \
-    --hash=sha256:30f610279e8b2578cab6db20741130331735c781b56053c59c4076da27f06b66
+securesystemslib==0.14.0 \
+    --hash=sha256:414a722547876294764813f7a3579bba273db6969de81bda2f46f60519e14e3e \
+    --hash=sha256:6cbd5ad0b2ae160a2de0800950757d6beea33a8aad15b41d6cff788b0a2ba926
+six==1.14.0 \
+    --hash=sha256:236bdbdce46e6e6a3d61a337c0f8b763ca1e8717c03b369e87a7ec7ce1319c0a \
+    --hash=sha256:8f3cd2e254d8f793e7f3d6d9df77b92252b52637291d0f0da013c76ea2724b6c
 subprocess32==3.5.4 \
     --hash=sha256:88e37c1aac5388df41cc8a8456bb49ebffd321a3ad4d70358e3518176de3a56b \
     --hash=sha256:eb2937c80497978d181efa1b839ec2d9622cf9600a039a79d0e108d1f9aec79d \
     ; python_version < "3.0"
-    # via securesystemslib
-urllib3==1.25.6 \
-    --hash=sha256:3de946ffbed6e6746608990594d08faac602528ac7015ac28d33cee6a45b7398 \
-    --hash=sha256:9a107b99a5393caf59c7aa3c1249c16e6879447533d0887f4336dde834c7be86 \
-    # via requests
+urllib3==1.25.8 \
+    --hash=sha256:2f3db8b19923a873b3e5256dc9c2dedfa883e33d87c690d9c7913e1f40673cdc \
+    --hash=sha256:87716c2d2a7121198ebcb7ce7cccf6ce5e9ba539041cfbaeecfb641dc0bf6acc

setup.cfg

@@ -3,6 +3,6 @@ universal = 1
 
 [check-manifest]
 ignore =
-  dev-requirements.txt
+  requirements-dev.txt
   .travis.yml
   .coveragerc

tox.ini

@@ -20,7 +20,7 @@ commands =
     coverage report -m --fail-under 97
 
 deps =
-    -r{toxinidir}/ci-requirements.txt
+    -r{toxinidir}/requirements-tox.txt
     # Install TUF in editable mode, instead of tox default virtual environment
     # installation (see `skipsdist`), to get relative paths in coverage reports
     --editable {toxinidir}
@@ -33,7 +33,7 @@ install_command = pip install --pre {opts} {packages}
 [testenv:with-sslib-master]
 deps =
     --editable git+http://github.com/secure-systems-lab/securesystemslib.git@master#egg=securesystemslib[crypto,pynacl]
-    -r{toxinidir}/ci-requirements.txt
+    -r{toxinidir}/requirements-tox.txt
     --editable {toxinidir}
 
 commands =