-
Notifications
You must be signed in to change notification settings - Fork 0
Re-add non-boilerplate code and add tests #6
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
dddaf1b
980aabf
ed0fe85
fa275d0
5f1789f
5e7ad22
439c880
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,49 @@ | ||
|
|
||
| Testing Manually | ||
| ---------------- | ||
|
|
||
| Normally if you wanted to test a command manually in dev you'd do so through | ||
| tox, for example: | ||
|
|
||
| ```terminal | ||
| $ tox -qe dev --run-command 'pip-sync-faster --help' | ||
| usage: pip-sync-faster [-h] [-v] | ||
|
|
||
| options: | ||
| -h, --help show this help message and exit | ||
| -v, --version | ||
| ``` | ||
|
|
||
| But there's a problem with running `pip-sync-faster` commands in this way: a | ||
| command like `tox -e dev --run-command 'pip-sync-faster requirements.txt'` will | ||
| run `pip-sync requirements.txt` and `pip-sync` will sync the | ||
| current virtualenv (`.tox/dev/`) with the `requirements.txt` file. Everything | ||
| in `requirements.txt` will get installed into `.tox/dev/`, which you probably | ||
| don't want. Even worse everything _not_ in `requirements.txt` will get | ||
| _removed_ from `.tox/dev/` including `pip-sync-faster` itself! | ||
|
|
||
| To avoid this problem run `pip-sync-faster` in a temporary virtualenv instead. | ||
| This installs the contents of `requirements.txt` into the temporary venv so | ||
| your `.tox/dev/` env doesn't get messed up. And it does not install | ||
| `pip-sync-faster` into the temporary venv so there's no issue with `pip-sync` | ||
| uninstalling `pip-sync-faster`: | ||
|
|
||
| ```terminal | ||
| # Make a temporary directory. | ||
| tempdir=$(mktemp -d) | ||
|
|
||
| # Create a virtualenv in the temporary directory. | ||
| python3 -m venv $tempdir | ||
|
|
||
| # Activate the virtualenv. | ||
| source $tempdir/bin/activate | ||
|
|
||
| # Install pip-tools in the virtualenv (pip-sync-faster needs pip-tools). | ||
| pip install pip-tools | ||
|
|
||
| # Call pip-sync-faster to install a requirements file into the temporary virtualenv. | ||
| PYTHONPATH=src python3 -m pip_sync_faster /path/to/requirements.txt | ||
|
|
||
| # When you're done testing deactivate the temporary virtualenv. | ||
| deactivate | ||
| ``` |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,63 @@ | ||
|
|
||
| pip-sync-faster makes | ||
| [pip-sync](https://pip-tools.readthedocs.io/en/latest/#example-usage-for-pip-sync) | ||
| run faster in the case where there's nothing to do because the virtualenv is | ||
| already up to date with the requirements files. On my machine, with my | ||
| requirements files, it shaves off over 500ms in the time taken to run pip-sync: | ||
|
|
||
| ```terminal | ||
| $ time pip-sync requirements/foo.txt | ||
| Everything up-to-date | ||
|
|
||
| real 0m0.569s | ||
| user 0m0.525s | ||
| sys 0m0.045s | ||
|
|
||
| $ time pip-sync-faster requirements/foo.txt | ||
|
|
||
| real 0m0.037s | ||
| user 0m0.029s | ||
| sys 0m0.008s | ||
| ``` | ||
|
|
||
| pip-sync-faster does this by saving hashes of the given requirements files in a | ||
| JSON file within the virtualenv and not calling pip-sync if the hashes haven't | ||
| changed. | ||
| If any of the given requirements files doesn't have a matching cached hash then | ||
| pip-sync-faster calls pip-sync forwarding all command line arguments and | ||
| options. | ||
|
|
||
| ## You need to add `pip-sync-faster` to your requirements file | ||
|
|
||
| A command like `pip-sync-faster requirements.txt` will call | ||
| `pip-sync requirements.txt` which will uninstall anything not in | ||
| `requirements.txt` from the active venv, including `pip-sync-faster` itself! | ||
|
|
||
| You can add `pip-sync-faster` to `requirements.txt` so that it doesn't get | ||
| uninstalled. | ||
|
|
||
| ### Running `pip-sync-faster` directly instead | ||
|
|
||
| Alternatively as long as `pip-tools` is installed in the active venv you can | ||
| run `pip-sync-faster` directly with a command like: | ||
|
|
||
| ```bash | ||
| PYTHONPATH=/path/to/pip-sync-faster/src python3 -m pip_sync_faster requirements.txt | ||
| ``` | ||
|
|
||
| This doesn't rely on `pip-sync-faster` being installed so there's no issue with | ||
| `pip-sync` uninstalling it. | ||
|
|
||
| ## pip-sync-faster doesn't sync modified virtualenvs | ||
|
|
||
| If you modify your requirements files pip-sync-faster will notice the change | ||
| and call pip-sync. But if you modify your virtualenv without modifying your | ||
| requirements files (for example by running a manual `pip install` command in | ||
| the virtualenv) pip-sync-faster will *not* call pip-sync because the | ||
| requirements files haven't changed and still match their cached hashes. | ||
|
|
||
| Calling pip-sync directly in this case would re-sync your virtualenv with your | ||
| requirements files, but calling pip-sync-faster won't. | ||
|
|
||
| If you can live with this limitation then you can use pip-sync-faster and save | ||
| yourself a few hundred milliseconds. If not you should just use pip-sync. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| pip-tools |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| lint,tests: pytest-mock |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -17,13 +17,15 @@ package_dir = | |
| = src | ||
| packages = find: | ||
| python_requires = >=3.8 | ||
| install_requires = | ||
| pip-tools | ||
|
|
||
| [options.packages.find] | ||
| where = src | ||
|
|
||
| [options.entry_points] | ||
| console_scripts = | ||
| pip-sync-faster = pip_sync_faster.main:entry_point | ||
| pip-sync-faster = pip_sync_faster.cli:cli | ||
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'll update this in the cookiecutter: |
||
|
|
||
| [pycodestyle] | ||
| ignore = | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| import sys | ||
|
|
||
| from pip_sync_faster.cli import cli | ||
|
|
||
| sys.exit(cli()) | ||
|
Comment on lines
+1
to
+5
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It'll be a good idea to move this
A This means that
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. TIL about sys.exit with a function, nice. |
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,28 @@ | ||
| from argparse import ArgumentParser | ||
| from importlib.metadata import version | ||
| from subprocess import CalledProcessError | ||
|
|
||
| from pip_sync_faster.sync import sync | ||
|
|
||
|
|
||
| def cli(_argv=None): # pylint:disable=inconsistent-return-statements | ||
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. PyLint will force you to put |
||
| parser = ArgumentParser( | ||
| description="Synchronize the active venv with requirements.txt files." | ||
| ) | ||
| parser.add_argument( | ||
| "--version", action="store_true", help="show the version and exit" | ||
| ) | ||
| parser.add_argument( | ||
| "src_files", nargs="*", help="the requirements.txt files to synchronize" | ||
| ) | ||
|
|
||
| args = parser.parse_known_args(_argv) | ||
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
|
|
||
| if args[0].version: | ||
| print(f"pip-sync-faster, version {version('pip-sync-faster')}") | ||
| return | ||
|
|
||
| try: | ||
| sync(args[0].src_files) | ||
| except CalledProcessError as err: | ||
| return err.returncode | ||
This file was deleted.
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,50 @@ | ||
| import hashlib | ||
| import json | ||
| import sys | ||
| from os import environ | ||
| from os.path import abspath | ||
| from pathlib import Path | ||
| from subprocess import run | ||
|
|
||
|
|
||
| def get_hash(path): | ||
| """Return the hash of the given file.""" | ||
| hashobj = hashlib.sha512() | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. You could probably squeeze a few microseconds switching to sha256, if git considers that future-proof should be enough for this. It doesn't probably matter. |
||
|
|
||
| with open(path, "rb") as file: | ||
| hashobj.update(file.read()) | ||
|
|
||
| return hashobj.hexdigest() | ||
|
|
||
|
|
||
| def get_hashes(paths): | ||
| """Return a dict mapping the given files to their hashes.""" | ||
| return {abspath(path): get_hash(abspath(path)) for path in paths} | ||
|
|
||
|
|
||
| def sync(src_files): | ||
| cached_hashes_path = Path(environ["VIRTUAL_ENV"]) / "pip_sync_faster.json" | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I reckon |
||
|
|
||
| try: | ||
| with open(cached_hashes_path, "r", encoding="utf-8") as handle: | ||
| cached_hashes = json.load(handle) | ||
| except FileNotFoundError: | ||
| cached_hashes = {} | ||
|
|
||
| hashes = get_hashes(src_files) | ||
|
|
||
| if hashes == cached_hashes: | ||
| return | ||
|
|
||
| # The hashes did not match the cached ones. This can happen if: | ||
| # | ||
| # * This is the first time that pip-sync-faster has been called for this venv | ||
| # * One or more of the requirements files has changed | ||
| # * pip-sync-faster was called with a different set of requirements files | ||
|
|
||
| run(["pip-sync", *sys.argv[1:]], check=True) | ||
|
|
||
| # Replace the cached hashes file with one containing the correct hashes for | ||
| # the requirements files that pip-sync-faster was called with this time. | ||
| with open(cached_hashes_path, "w", encoding="utf-8") as handle: | ||
| json.dump(hashes, handle) | ||
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. When you see it split out into its own file like this the actual core logic of pip-sync-faster is very simple (and only about 40 lines of code without comments and docstrings) |
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This'll go into the cookiecutter,
__main__.pywill always be omitted from coverage