hatch-pip-compile --upgrade

[juftin]

Currently updating requirements can be tough because of the way the plugin is implemented:

Current Implementation

1. Create a temporary requirements.in file with a project's dependencies
2. Use that requirements.in file as the input to pip-compile with the output being the respective lock file
3. Since the input is actually a requirements.in file and not the actual pyproject.toml - some post-processing occurs replacing lines like # via -r /var/folders/qk/mw5mftjj0qz42nrxrnhctlzw0000gp/T/tmpo85wegym/default.in with # via hatch-pip-compile (pyproject.toml) as well as to add the custom hatch-pip-compile header

Because of that implementation there isn't currently a command you could run locally that would update the file in-place. Ideally something like this would just work:

hatch dep show requirements --all | \
  hatch run python -m piptools compile - \
    --output-file requirements.txt \
    --generate-hashes \
    --no-header \
    --upgrade-package certifi

However running that above command will blow away the hatch-pip-compile header.

At this point this issue is just a placeholder and place to brainstorm ideas for the ideal workflow with pip-compile

[juftin]

🤔 Maybe hatch-pip-compile could also be a CLI that maintainers could invoke to do something like this

[juftin]

🤔 Maybe we should stop with the requirements.in file and simply pipe the self.dependencies via stdout similar to the CLI command in the description

[oprypin]

Maybe hatch-pip-compile could also be a CLI that maintainers could invoke to do something like this

That was my first thought, yeah.

Another thing could be to include a large command like the one that I had in the original post with hatch env show but I don't think any reasonably short universal command could be made.
pypa/hatch#522

Maybe we should stop with the requirements.in file and simply pipe the self.dependencies via stdout similar to the CLI command in the description

I'm not sure if I fully understood this comment, but yeah, a temporary file is not necessary (also demonstrated in my original post). But it'll be -r - otherwise, which can make my next point risky (an imprecise string match)

You could probably do a plain find&replace on the filename to avoid the ugly parts, right?
Ah right yes that's what you're doing. I think that's fine

[oprypin]

Actually.. don't mind my comment, it seems I'm just saying the same stuff that you already said at the top.

[oprypin]

Well.. one universal command that we already have is

rm requirements/requirements-envname.txt; hatch run envname:true

[juftin]

Yeah - the more I'm thinking about this the more I think it's right for a CLI.

Probably something similar that combines something like this with the same post-processing that the tool does on the backend.

hatch dep show requirements --all | \
  hatch run python -m piptools compile - \
    --output-file requirements.txt \
    --generate-hashes \
    --no-header \
    --upgrade

Similarly, rm requirements/requirements-envname.txt; hatch run envname:true is usually all I need but it would be nice to get some additional control.

[juftin]

@oprypin what would you think about a workflow like this for upgrading dependencies instead of a CLI? I was writing a CLI for the tool but figured this would be easier and has the benefit of the plugin handling the lockfile generation instead of a CLI external to the hatch environment.

(taken from the README on a feature branch):

Updating Dependencies

Passing arguments like --upgrade / --upgrade-package to pip-compile is controlled by two environment variables: PIP_COMPILE_UPGRADE and PIP_COMPILE_UPGRADE_PACKAGE. When either of these environment variables are set hatch will assume that the environment is out of date and automatically run pip-compile the next time the environment is activated.

The below command runs pip-compile --upgrade on the default environment.

PIP_COMPILE_UPGRADE=1 hatch env run --env default -- python --version

The below command runs pip-compile --upgrade-package=mkdocs --upgrade-package=mkdocs-material on the docs environment.

PIP_COMPILE_UPGRADE_PACKAGE="mkdocs,mkdocs-material" hatch env run --env docs -- python --version

The above commands call python --version on a particular environment to trigger the lockfile regeneration but the same behavior applies to any command that activates the environment.

[oprypin]

Well one thing is, I think a CLI would run into a lot of issues being accessed, because yeah, if it's used through pipx, you pretty much just can't access the CLI. So, sure, maybe the environment variables are what has to be done.

[juftin]

Well one thing is, I think a CLI would run into a lot of issues being accessed, because yeah, if it's used through pipx, you pretty much just can't access the CLI. So, sure, maybe the environment variables are what has to be done.

I got a CLI working but there are two not so-ideal options that it made me decide between:

• Use a Python interpreter outside of the hatch environment (i.e pipx) to call pip-compile
• Use the hatch environment's interpreter by calling hatch env run --env docs -- python -m piptools compile... - but this has the issue of the plugin running pip-compile if it's out of date before running the requested pip-compile --upgrade command from the CLI.

I feel like an in-plugin implementation makes the implementation much more straightforward by only having one way that pip-compile is ever invoked.

[oprypin]

• Use the hatch environment's interpreter by calling hatch env run --env docs -- python -m piptools compile... - but this has the issue of the plugin running pip-compile if it's out of date before running the requested pip-compile --upgrade command from the CLI.

Actually I have an idea for this. What if, instead of this being an issue, you turn it into the place where the upgrade actually happens. Just set some kind of override (such as the environment variable as being implemented elsewhere) and make the hatch invocation itself perform the upgrade rather running the actual pip-compile separately again.

But I still didn't get how the previous issue I mentioned would be solved - the CLI would probably be isolated by pipx and inaccessible by default?

[juftin]

Actually I have an idea for this. What if, instead of this being an issue, you turn it into the place where the upgrade actually happens.

I'm not completely following here. Is the idea for a CLI tha handles setting the PIP_COMPILE_UPGRADE variable under the hood and just invokes hatch env run --env default -- python --version so the lock is updated?

[oprypin]

Yes

[juftin]

Yes

That works - the way I'm thinking about it there might be four ways of doing an update. I'm curious which workflows are pleasant and if any aren't worth documenting.

---

Removing the lock and invoking the env:

rm requirements.txt
hatch env run --env default -- python --version

---

Setting the env var and invoking the env:

PIP_COMPILE_UPGRADE=1 hatch env run --env default -- python --version

---

Installing the CLI outside of hatch - it sets the env var and invokes the env under the hood:

pipx install hatch-pip-compile
hatch-pip-compile default --upgrade

---

Creating a hatch script to set the env var and invoke the env (this might be an interesting thing to do with an Environment Collector plugin too):

[tool.hatch.envs.requirements]
detached = true

[tool.hatch.envs.requirements.scripts]
upgrade = "hatch env run --env {args} -- python --version"

[tool.hatch.envs.requirements.env-vars]
PIP_COMPILE_UPGRADE = "true"

hatch run requirements:upgrade default

[oprypin]

Ah interesting regarding the latest idea.

The sad thing about an environment collector plugin though is that it needs to be explicitly mentioned in the config in order to kick in. As in this example

Anyway I'm just rambling. These are all very cool ideas but in the end probably only the first two from the above list will be viable. In any case, sure, only the "delete" option and one other option of choice should be the documented one.

[juftin]

🎉 This issue has been resolved in version 1.2.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀