Replace pipenv?

[jameslzhu]

At this point I'm starting to be convinced pipenv is not the right solution for deploys, for a few reasons:

1. It does not respect existing venvs well. We'd prefer to have a cached venv for the production instance, and unless we specifically source the venv before running pipenv, or symlink a .venv link to the shared/venv folder and set the PIPENV_VENV_IN_PROJECT env variable, it will create a different venv for each timestamped release, because they are at different paths.
2. It is under active development. Occasionally this has caused breaking changes (for me, personally) that cause pipenv itself to fail to run, and require upgrading pipenv.
3. dependency = "*" by default. This wreaks havoc on dependency pinning, in particular because if left unpinned before another person installs the dependency, version mismatches will ensue in the Pipfile.lock, with the resulting merge conflict requiring a brand-new pipenv lock to fix or manual intervention. (Admittedly, this can be fixed by pinning a version at the start, but pip does this with pip freeze automatically.)

During development, pipenv install will install versions beyond those specified in the lockfile, even when not prompted to upgrade versions.

With that said, there are major features of pipenv that remove pain points in development:

1. Automatic venv installation with dependency installation. This cuts out all of the venv / virtualenv / source bin/activate/ / pip install -r requirements.txt nonsense, and installs everything with pipenv install.
2. Dependency resolution. Pip's dependency resolution is apparently not very sophisticated; but pipenv is not the only dependency resolver (see poetry below).
3. Automatic lockfiles. Pip versioning typically requires a requirements.txt with full versioning and dependencies, and a requirements-minimal.txt with the minimal requirements (roots of the graph). Pipenv saves the generated dependency list with hashes (a+ for security), usable with pipenv install --deploy.
4. Editor integration. It integrates with Sublime, VS Code, PyCharm, and a couple other services (Github vulnerabilities).

Alternatives might include poetry, the leading competitor to pipenv.

[jameslzhu]

Is conda a viable alternative?

@adithyaj

[jameslzhu]

May also consider hatch, which appears not to use any other "special" files other than requirements.txt and setup.py (which are already used by pip).

However it doesn't manage dependencies, a major downside.

[jvperrin]

I think this setup is decent, although I'm biased since I'm used to how it's set up and it originated where I currently work :)

The upside is that it's fast, easy to work with once you know how it works, has cached venvs, a lockfile of sorts (pinned requirements in a file suitable for deploys anyway), etc. However, the downsides are that it needs at least 4 files (requirements-minimal.txt for unpinned prod dependencies like anything the application actually imports, requirements.txt for pinned prod dependencies generated from requirements-minimal.txt, requirements-dev-minimal.txt for unpinned dev dependencies like anything used for testing and pre-commit hooks, and finally requirements-dev.txt for pinned dev dependencies). It also can be confusing to know what to edit starting out, and it needs some tools to work with it (like upgrade-requirements to easily update pinned requirements). I've found it works pretty well overall and addresses all my concerns around dependencies and pinning though, which is nice, and it's also something the OCF uses.

[bri25yu]

Resolved by #272