Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 49 additions & 0 deletions .cookiecutter/includes/HACKING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@

Testing Manually
----------------

Normally if you wanted to test a command manually in dev you'd do so through
tox, for example:

```terminal
$ tox -qe dev --run-command 'pip-sync-faster --help'
usage: pip-sync-faster [-h] [-v]

options:
-h, --help show this help message and exit
-v, --version
```

But there's a problem with running `pip-sync-faster` commands in this way: a
command like `tox -e dev --run-command 'pip-sync-faster requirements.txt'` will
run `pip-sync requirements.txt` and `pip-sync` will sync the
current virtualenv (`.tox/dev/`) with the `requirements.txt` file. Everything
in `requirements.txt` will get installed into `.tox/dev/`, which you probably
don't want. Even worse everything _not_ in `requirements.txt` will get
_removed_ from `.tox/dev/` including `pip-sync-faster` itself!

To avoid this problem run `pip-sync-faster` in a temporary virtualenv instead.
This installs the contents of `requirements.txt` into the temporary venv so
your `.tox/dev/` env doesn't get messed up. And it does not install
`pip-sync-faster` into the temporary venv so there's no issue with `pip-sync`
uninstalling `pip-sync-faster`:

```terminal
# Make a temporary directory.
tempdir=$(mktemp -d)

# Create a virtualenv in the temporary directory.
python3 -m venv $tempdir

# Activate the virtualenv.
source $tempdir/bin/activate

# Install pip-tools in the virtualenv (pip-sync-faster needs pip-tools).
pip install pip-tools

# Call pip-sync-faster to install a requirements file into the temporary virtualenv.
PYTHONPATH=src python3 -m pip_sync_faster /path/to/requirements.txt

# When you're done testing deactivate the temporary virtualenv.
deactivate
```
63 changes: 63 additions & 0 deletions .cookiecutter/includes/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@

pip-sync-faster makes
[pip-sync](https://pip-tools.readthedocs.io/en/latest/#example-usage-for-pip-sync)
run faster in the case where there's nothing to do because the virtualenv is
already up to date with the requirements files. On my machine, with my
requirements files, it shaves off over 500ms in the time taken to run pip-sync:

```terminal
$ time pip-sync requirements/foo.txt
Everything up-to-date

real 0m0.569s
user 0m0.525s
sys 0m0.045s

$ time pip-sync-faster requirements/foo.txt

real 0m0.037s
user 0m0.029s
sys 0m0.008s
```

pip-sync-faster does this by saving hashes of the given requirements files in a
JSON file within the virtualenv and not calling pip-sync if the hashes haven't
changed.
If any of the given requirements files doesn't have a matching cached hash then
pip-sync-faster calls pip-sync forwarding all command line arguments and
options.

## You need to add `pip-sync-faster` to your requirements file

A command like `pip-sync-faster requirements.txt` will call
`pip-sync requirements.txt` which will uninstall anything not in
`requirements.txt` from the active venv, including `pip-sync-faster` itself!

You can add `pip-sync-faster` to `requirements.txt` so that it doesn't get
uninstalled.

### Running `pip-sync-faster` directly instead

Alternatively as long as `pip-tools` is installed in the active venv you can
run `pip-sync-faster` directly with a command like:

```bash
PYTHONPATH=/path/to/pip-sync-faster/src python3 -m pip_sync_faster requirements.txt
```

This doesn't rely on `pip-sync-faster` being installed so there's no issue with
`pip-sync` uninstalling it.

## pip-sync-faster doesn't sync modified virtualenvs

If you modify your requirements files pip-sync-faster will notice the change
and call pip-sync. But if you modify your virtualenv without modifying your
requirements files (for example by running a manual `pip install` command in
the virtualenv) pip-sync-faster will *not* call pip-sync because the
requirements files haven't changed and still match their cached hashes.

Calling pip-sync directly in this case would re-sync your virtualenv with your
requirements files, but calling pip-sync-faster won't.

If you can live with this limitation then you can use pip-sync-faster and save
yourself a few hundred milliseconds. If not you should just use pip-sync.
1 change: 1 addition & 0 deletions .cookiecutter/includes/setuptools/install_requires
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
pip-tools
1 change: 1 addition & 0 deletions .cookiecutter/includes/tox/deps
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
lint,tests: pytest-mock
49 changes: 49 additions & 0 deletions HACKING.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,3 +100,52 @@ To change the project's formatting, linting and test dependencies:
```

3. Commit everything to git and send a pull request

Testing Manually
----------------

Normally if you wanted to test a command manually in dev you'd do so through
tox, for example:

```terminal
$ tox -qe dev --run-command 'pip-sync-faster --help'
usage: pip-sync-faster [-h] [-v]

options:
-h, --help show this help message and exit
-v, --version
```

But there's a problem with running `pip-sync-faster` commands in this way: a
command like `tox -e dev --run-command 'pip-sync-faster requirements.txt'` will
run `pip-sync requirements.txt` as a subprocess and `pip-sync` will sync the
current virtualenv (`.tox/dev/`) with the `requirements.txt` file. Everything
in `requirements.txt` will get installed into `.tox/dev/`, which you probably
don't want. Even worse everything _not_ in `requirements.txt` will get
_removed_ from `.tox/dev/` including `pip-sync-faster` itself!

To avoid this problem run `pip-sync-faster` in a temporary virtualenv instead.
This installs the contents of `requirements.txt` into the temporary venv so
your `.tox/dev/` env doesn't get messed up. And it does not install
`pip-sync-faster` into the temporary venv so there's no issue with `pip-sync`
uninstalling `pip-sync-faster`:

```terminal
# Make a temporary directory.
tempdir=$(mktemp -d)

# Create a virtualenv in the temporary directory.
python3 -m venv $tempdir

# Activate the virtualenv.
source $tempdir/bin/activate

# Install pip-tools in the virtualenv (pip-sync-faster needs pip-tools).
pip install pip-tools

# Call pip-sync-faster to install a requirements file into the temporary virtualenv.
PYTHONPATH=src python3 -m pip_sync_faster /path/to/requirements.txt

# When you're done testing deactivate the temporary virtualenv.
deactivate
```
63 changes: 63 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,3 +13,66 @@ For installation instructions see [INSTALL.md](https://github.com/hypothesis/pip

For how to set up a pip-sync-faster development environment see
[HACKING.md](https://github.com/hypothesis/pip-sync-faster/blob/main/HACKING.md).

pip-sync-faster makes
[pip-sync](https://pip-tools.readthedocs.io/en/latest/#example-usage-for-pip-sync)
run faster in the case where there's nothing to do because the virtualenv is
already up to date with the requirements files. On my machine, with my
requirements files, it shaves off over 500ms in the time taken to run pip-sync:

```terminal
$ time pip-sync requirements/foo.txt
Everything up-to-date

real 0m0.569s
user 0m0.525s
sys 0m0.045s

$ time pip-sync-faster requirements/foo.txt

real 0m0.037s
user 0m0.029s
sys 0m0.008s
```

`pip-sync-faster` does this by saving hashes of the given requirements files in a
JSON file within the virtualenv and not calling pip-sync if the hashes haven't
changed.
If any of the given requirements files doesn't have a matching cached hash then
pip-sync-faster calls pip-sync forwarding all command line arguments and
options.

## You need to add `pip-sync-faster` to your requirements file

A command like `pip-sync-faster requirements.txt` will call
`pip-sync requirements.txt` which will uninstall anything not in
`requirements.txt` from the active venv, including `pip-sync-faster` itself!

You can add `pip-sync-faster` to `requirements.txt` so that it doesn't get
uninstalled.

### Running `pip-sync-faster` directly instead

Alternatively as long as `pip-tools` is installed in the active venv you can
run `pip-sync-faster` directly with a command like:

```bash
PYTHONPATH=/path/to/pip-sync-faster/src python3 -m pip_sync_faster requirements.txt
```

This doesn't rely on `pip-sync-faster` being installed so there's no issue with
`pip-sync` uninstalling it.

## pip-sync-faster doesn't sync modified virtualenvs

If you modify your requirements files pip-sync-faster will notice the change
and call pip-sync. But if you modify your virtualenv without modifying your
requirements files (for example by running a manual `pip install` command in
the virtualenv) pip-sync-faster will *not* call pip-sync because the
requirements files haven't changed and still match their cached hashes.

Calling pip-sync directly in this case would re-sync your virtualenv with your
requirements files, but calling pip-sync-faster won't.

If you can live with this limitation then you can use pip-sync-faster and save
yourself a few hundred milliseconds. If not you should just use pip-sync.
3 changes: 3 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,9 @@ ignore = [
branch = true
parallel = true
source = ["pip_sync_faster", "tests/unit"]
omit = [
"*/pip_sync_faster/__main__.py",
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This'll go into the cookiecutter, __main__.py will always be omitted from coverage

]

[tool.coverage.paths]
source = ["src", ".tox/*tests/lib/python*/site-packages"]
Expand Down
4 changes: 3 additions & 1 deletion setup.cfg
Original file line number Diff line number Diff line change
Expand Up @@ -17,13 +17,15 @@ package_dir =
= src
packages = find:
python_requires = >=3.8
install_requires =
pip-tools

[options.packages.find]
where = src

[options.entry_points]
console_scripts =
pip-sync-faster = pip_sync_faster.main:entry_point
pip-sync-faster = pip_sync_faster.cli:cli
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll update this in the cookiecutter: main.py::entry_point() renamed to cli.py::cli()


[pycodestyle]
ignore =
Expand Down
5 changes: 5 additions & 0 deletions src/pip_sync_faster/__main__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
import sys

from pip_sync_faster.cli import cli

sys.exit(cli())
Comment on lines +1 to +5
Copy link
Contributor Author

@seanh seanh Jul 25, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It'll be a good idea to move this __main__.py into the cookiecutter where it can be implemented correctly once and for all. For one thing we can implement __main__.py in the idiomatic way (as here), and we can omit it from test coverage. Also it's actually not as trivial to implement as you might expect:

cli() is used as both the setuptools entry point function (specified in setup.cfg) and the top-level function for __main__.py to call. A setuptools entry point function is expected to return something appropriate for being passed to sys.exit(): None or 0 to exit successfully, an int to exit with that error code, or any printable object (such as a string) that will be printed to stderr before exiting with 1. You can see this by looking at the pip-sync-faster script that setuptools generates when you install the package:

$ cat .tox/dev/bin/pip-sync-faster
#!/home/seanh/Projects/pip-sync-faster/.tox/dev/bin/python
# -*- coding: utf-8 -*-
import re
import sys
from pip_sync_faster.cli import cli
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(cli())

A __main__.py or an if __name__ == "__main__" block should do the same thing: sys.exit(cli()).

This means that cli() itself shouldn't call sys.exit(), it should just return None or 0 to exit successfully or return 1 or return "Some error message" to exit with an error

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TIL about sys.exit with a function, nice.

28 changes: 28 additions & 0 deletions src/pip_sync_faster/cli.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
from argparse import ArgumentParser
from importlib.metadata import version
from subprocess import CalledProcessError

from pip_sync_faster.sync import sync


def cli(_argv=None): # pylint:disable=inconsistent-return-statements
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PyLint will force you to put return 0 or return None in every branch of the function (including at the very bottom) which just seems annoying for an entry point function.

parser = ArgumentParser(
description="Synchronize the active venv with requirements.txt files."
)
parser.add_argument(
"--version", action="store_true", help="show the version and exit"
)
parser.add_argument(
"src_files", nargs="*", help="the requirements.txt files to synchronize"
)

args = parser.parse_known_args(_argv)
Copy link
Contributor Author

@seanh seanh Jul 25, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_argv (which is used by the tests) defaults to None and parse_known_args(None) causes argparse to parse the args from sys.argv


if args[0].version:
print(f"pip-sync-faster, version {version('pip-sync-faster')}")
return

try:
sync(args[0].src_files)
except CalledProcessError as err:
return err.returncode
18 changes: 0 additions & 18 deletions src/pip_sync_faster/main.py

This file was deleted.

50 changes: 50 additions & 0 deletions src/pip_sync_faster/sync.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
import hashlib
import json
import sys
from os import environ
from os.path import abspath
from pathlib import Path
from subprocess import run


def get_hash(path):
"""Return the hash of the given file."""
hashobj = hashlib.sha512()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could probably squeeze a few microseconds switching to sha256, if git considers that future-proof should be enough for this.

It doesn't probably matter.


with open(path, "rb") as file:
hashobj.update(file.read())

return hashobj.hexdigest()


def get_hashes(paths):
"""Return a dict mapping the given files to their hashes."""
return {abspath(path): get_hash(abspath(path)) for path in paths}


def sync(src_files):
cached_hashes_path = Path(environ["VIRTUAL_ENV"]) / "pip_sync_faster.json"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reckon environ["VIRTUAL_ENV"] is the right place to store it. It would be just noise in the project's root.


try:
with open(cached_hashes_path, "r", encoding="utf-8") as handle:
cached_hashes = json.load(handle)
except FileNotFoundError:
cached_hashes = {}

hashes = get_hashes(src_files)

if hashes == cached_hashes:
return

# The hashes did not match the cached ones. This can happen if:
#
# * This is the first time that pip-sync-faster has been called for this venv
# * One or more of the requirements files has changed
# * pip-sync-faster was called with a different set of requirements files

run(["pip-sync", *sys.argv[1:]], check=True)

# Replace the cached hashes file with one containing the correct hashes for
# the requirements files that pip-sync-faster was called with this time.
with open(cached_hashes_path, "w", encoding="utf-8") as handle:
json.dump(hashes, handle)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When you see it split out into its own file like this the actual core logic of pip-sync-faster is very simple (and only about 40 lines of code without comments and docstrings)

Loading