Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Revise dependency handling #294

Merged
merged 3 commits into from
Jul 15, 2019
Merged

Revise dependency handling #294

merged 3 commits into from
Jul 15, 2019

Conversation

lukpueh
Copy link
Member

@lukpueh lukpueh commented Jul 9, 2019 •
edited
Loading

Fixes issue #:
Related to #270

Description of the changes being introduced by the pull request:
To address the request in #270 I recently enabled dependabot, only to realize that dependency monitors have nothing to do if there are no pinned dependencies.

So I reviewed our dependency management and state of the art strategies and tooling (see collection of links below). The TL;DR is,

  • pin if you want reproducible (not bit-by-bit) deployments of your python application , to avoid installation of broken/incompatible/insecure old or new dependencies, but
  • don't pin, when you aim at re-usability of your library, framework, tool, etc., as pinning also increases the likelihood of dependency conflicts, adds a lot of release overhead (ideally a new release per changed dependency or sub-dependency). As pointed out by @SantiagoTorres this is especially cumbersome if there are downstream packages that rely on different package managers exist as well.

Since in-toto falls into the second category, we leave the dependencies in setup.py and requirements.txt un-pinned. However, in order to regularly and explicitly test against new dependency versions, this PR adds a pinned requirements file that will be updated automatically by
dependabot and used by Travis builds, which should help us catch incompatible new versions.

This PR further restructures the different requirements-*.txt files for different purposes. Please read the commit messages for details.

Corresponding documentation will be updated with #292.

Please verify and check that the pull request fulfills the following
requirements:

  • The code follows the Code Style Guidelines
  • Tests have been added for the bug fix or new feature
  • Docs have been added for the bug fix or new feature

@lukpueh
Fix dependency mismatch in requirements and setup
b64317d 
Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
@lukpueh
Add pinned requirements file
d4ecf84 
Add requirements file with pinned versions based on the combined
results from the `pip-compile` tool run over `requirements.txt` in
every supported Python version.

This requirements file is intended to be used in automated testing
together with automated dependency monitoring and updating to catch
incompatible updates.

We deliberately don't pin versions in the main `requirements.txt`
or in `setup.py`, to allow end-users more flexible installs.

Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
@lukpueh
Restructure requirements files
06a2898 
Removes inconsistent/outdated requirements-ci.txt and instead adds
three new sub requirements files and uses them as appropriate in
ci/cd files:

- requirements-test.txt: Lists test suite runtime requirements
  (corresponds to setup.py's `tests_require` field).
- requirements-tox.txt: Lists everything that needs to be installed
  in each tox test environment, i.e. in-toto runtime requirements
  (uses periodically and automatically updated
  requirements-pinned.txt to catch incompatible updates), in-toto
  test suite runtime requirements (uses requirements-test.txt), and
  test tools for linting and coverage.
- requirements-dev.txt: Lists all requirements to run tests
  directly in the development environment (uses
  requirements-tox.txt), and tox to run tox locally, and an
  editable install of in-toto.

Tox (see tox.ini) now only needs install requirements-tox.txt and
travis (see .travis.yml), which runs tox, only needs to install
tox.

appveyor (see appveyor.yml), which does not support tox, installs
pinned in-toto runtime requirements and test run time requirements,
as well as in-toto directly.

Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
@lukpueh
Copy link
Member Author

lukpueh commented Jul 10, 2019

@SantiagoTorres, could you take a look at this PR? I'm not dead set on the different requirement-*.txt files, but to me the division made sense. d4ecf84 talks about the pinned one and 06a2898 has a brief description for each of the others. Let me know what you think.

@lukpueh lukpueh mentioned this pull request Jul 10, 2019
Closed
SantiagoTorres
SantiagoTorres approved these changes Jul 15, 2019
View reviewed changes
Copy link
Member

@SantiagoTorres SantiagoTorres left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Very commendable way to make dep monitoring work + ensuring library compat 👍

@SantiagoTorres SantiagoTorres merged commit 9c374a9 into in-toto:develop Jul 15, 2019
@lukpueh lukpueh mentioned this pull request Jul 17, 2019
Open
@lukpueh lukpueh mentioned this pull request Sep 19, 2019
Closed
@lukpueh lukpueh mentioned this pull request Oct 15, 2019
Merged
3 tasks
lukpueh added a commit to lukpueh/tuf that referenced this pull request Feb 6, 2020
@lukpueh
Restructure requirements files
0f7383a 
* Use suffixed instead of prefixed sub-requirements files to group
  them alphabetically in the file tree.
* Layer requirements files akin to the in-toto project
  (see in-toto/in-toto#294). The hierarchy is:

  - *requirements.in*
    tuf runtime requirements, including optional requirements
    (pynacl and cyrptography)

  - *requirements.txt*
    pinned tuf runtime requirements, including optional and
    transitive (1 level deep) requirements and their hashes.

    The file is generated semi-automatically using pip-compile
    and a bash script (see document header), based off of
    requirements.in, combining requirements from all supported
    Python versions.

    This file should be auto-updated, by e.g. dependabot, and be used
    for ci/cd tests, to catch issues with new dependencies.

  - *requirements-test.txt*
    additional test runtime requirements

  - *requirements-tox.txt*
    combines requirements.txt, requirements-test.txt and additional
    test tools (for linting and coverage), i.e. everything that is
    needed in each tox environment to run the tests.

  - *requirements-dev.txt*
    lists tox for local development and testing, and also
    requirements-tox.txt and tuf in editable mode to run
    the test suite or individual tests directly.

* Removes an obsolete version constraint on coverage

Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
@lukpueh lukpueh mentioned this pull request Feb 6, 2020
Merged
3 tasks
lukpueh added a commit to lukpueh/tuf that referenced this pull request Feb 6, 2020
@lukpueh
Restructure requirements files
1c039a2 
* Use suffixed instead of prefixed sub-requirements files to group
  them alphabetically in the file tree.
* Layer requirements files akin to the in-toto project
  (see in-toto/in-toto#294). The hierarchy is:

  - *requirements.in*
    tuf runtime requirements, including optional requirements
    (pynacl and cyrptography)

  - *requirements-pinned.txt*
    pinned tuf runtime requirements, including optional
    and transitive (1 level deep) requirements and their hashes.

    The file is generated semi-automatically using pip-compile
    and a bash script (see document header), based off of
    requirements.in, combining requirements from all supported
    Python versions.

    This file should be auto-updated, by e.g. dependabot, and be used
    for ci/cd tests, to catch issues with new dependencies.

  - *requirements-test.txt*
    additional test runtime requirements

  - *requirements-tox.txt*
    combines requirements.txt, requirements-test.txt and additional
    test tools (for linting and coverage), i.e. everything that is
    needed in each tox environment to run the tests.

  - *requirements-dev.txt*
    lists tox for local development and testing, and also
    requirements-tox.txt and tuf in editable mode to run
    the test suite or individual tests directly.

  - *requirements.txt*
    requirements-pinned.txt with the hashes of the dependencies
    as reported by pip at the time of creating the file.
    NOTE: this is not used for testing or  dev-install because pip
    doesn't allow mixed (with and without hashes) installations.

    This file should also be auto-updated, by e.g. dependabot.

* Removes an obsolete version constraint on coverage

Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>

kek
lukpueh added a commit to lukpueh/tuf that referenced this pull request Feb 6, 2020
@lukpueh
Restructure requirements files
15d1e93 
* Use suffixed instead of prefixed sub-requirements files to group
  them alphabetically in the file tree.
* Layer requirements files akin to the in-toto project
  (see in-toto/in-toto#294). The hierarchy is:

  - *requirements.in*
    tuf runtime requirements, including optional requirements
    (pynacl and cyrptography)

  - *requirements-pinned.txt*
    pinned tuf runtime requirements, including optional
    and transitive (1 level deep) requirements and their hashes.

    The file is generated semi-automatically using pip-compile
    and a bash script (see document header), based off of
    requirements.in, combining requirements from all supported
    Python versions.

    This file should be auto-updated, by e.g. dependabot, and be used
    for ci/cd tests, to catch issues with new dependencies.

  - *requirements-test.txt*
    additional test runtime requirements

  - *requirements-tox.txt*
    combines requirements.txt, requirements-test.txt and additional
    test tools (for linting and coverage), i.e. everything that is
    needed in each tox environment to run the tests.

  - *requirements-dev.txt*
    lists tox for local development and testing, and also
    requirements-tox.txt and tuf in editable mode to run
    the test suite or individual tests directly.

  - *requirements.txt*
    requirements-pinned.txt with the hashes of the dependencies
    as reported by pip at the time of creating the file.
    NOTE: this is not used for testing or  dev-install because pip
    doesn't allow mixed (with and without hashes) installations.

    This file should also be auto-updated, by e.g. dependabot.

* Removes an obsolete version constraint on coverage

Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>
lukpueh added a commit to lukpueh/securesystemslib that referenced this pull request Feb 18, 2020
@lukpueh
Revise requirements and dependency monitoring (WIP) [skip CI]
c9a28e9 
TODO:
- Add Dependabot badge
- Compare to theupdateframework/python-tuf#982 and in-toto/in-toto#294
- Grep for renames
- Create commits
@lukpueh lukpueh mentioned this pull request Feb 19, 2020
Merged
3 tasks
@sechkova sechkova mentioned this pull request Mar 2, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@SantiagoTorres SantiagoTorres SantiagoTorres approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@lukpueh @SantiagoTorres