pin *all* dependencies in requirements.txt

[candlerb]

Environment

• Python version: 3.5.2
• NetBox version: 2.7.6

Proposed Functionality

In my opinion, netbox should pin the version of all packages it uses - not just direct dependencies but all descendant dependencies as well.

Use Case

People are repeatedly running into issues where Netbox breaks because a child or grandchild dependency package is at an unexpected version.

I personally hit it in #3460. I hit it again with PyYAML installing the wrong version under Ubuntu 16.04. However there are numerous recent cases in google groups where people have broken systems and the fixes involve upgrading or downgrading random packages (e.g. markdown, django-restframework).

I think Netbox has become unsupportable unless one of two things happen:

1. Release an official docker container, and this is the only supported artefact - i.e. if you choose to install from pip then you are a developer and get to fix your own problems; or
2. Netbox pins all dependencies to fixed versions in requirements.txt

There may also be some improvement by getting all users to install in a fresh virtualenv (not polluted by any packages they've already installed, or any OS-supplied python packages), but that doesn't solve the problem completely.

Database Changes

None

External Dependencies

This is to some extent caused by a long-standing bug in pip which does not handle recursive dependencies properly. If A installs B and C, and B and C both depend on D (but with different version requirements), a version of D may be installed which meets one requirements but not both, even though a suitable version which meets both is available.

However even if that bug did not exist, pip3 could still end up installing a version of D which meets the declared requirements of B and C, but for some reason does not work in the context of Netbox. This could be because of some missing dependency (Netbox actually uses D directly but doesn't declare it); or it could be because of some other unexpected interaction.

To take one specific example see this post, where the output of "pip3 list" from an end user is very different to what my own system shows - and by extension, is likely to be different to what the Netbox developers have.

If this happens, it means that (a) developers aren't going to be aware of problems; (b) end-users are going to experience these problems first; and (c) we end up trying to support people's systems with problems we haven't seen before and whose systems are in indeterminate states.

Note: Netbox pins most of its direct dependencies to specific versions, but not Django which it pins to a range.

[chicks-net]

Have you considered pipenv ? It seems to solve the virtualenv creation and python package pinning that you're looking for.

[jeremystretch]

In my opinion, netbox should pin the version of all packages it uses - not just direct dependencies but all descendant dependencies as well.

Are you asserting that there's something unique to NetBox that requires this, or that this is true for all projects which employ pip for dependency management?

[candlerb]

The latter. This is what "pip freeze" exists for.

[jeremystretch]

Release an official docker container, and this is the only supported artefact

Just want to go on record to point out that this is a non-starter: We won't require users to implement an entire layer of abstraction and deploy external services simply to run the application.

I'm not necessarily opposed to adopting this as policy, but we should consider the implications. First, this would effectively require users to run NetBox in a virtual environment (if not using a dedicated server already). This isn't a bad thing IMO, but it does amount to a new installation requirement that needs to be communicated and documented. Incidentally, this initiative has already been captured as #3949.

Second, it means the maintainers must now vet a specific release version for every dependent package: There are currently almost 60, as opposed to the 25 direct dependencies we currently maintain. I don't think this is too big a deal given that we typically update pinned versions only during minor releases (e.g. from 2.6 to 2.7), but it is worth noting that we'd now be managing an entire tree of dependencies.

We'd also be asserting that all dependent packages can be updated only during a minor release. The only exception to this is when a bug or vulnerability is identified in a dependent package (which requires a peer bug report to be submitted against NetBox to track the change).

And finally, I'll point out that this largely prevents bundling NetBox as a distribution package, as pointed out in #2977. I don't think this is a substantial argument though, especially given that it's already largely frustrated by the current pinnings.

So to summarize:

• Officially require installation within a virtual environment (or dedicated instance)
• Extend requirements.txt to pin a specific version for all ~60 dependent packages
• Extend base_requirements.txt to document all ~60 packages and why they are needed
• Update all pinned versions on every 2.x release
• Update the pinned version of a specific package as needed per bug reports

Does that sound right?

Note: Netbox pins most of its direct dependencies to specific versions, but not Django which it pins to a range.

IMO if we're going to pin everything we might as well pin Django too. The downside is that new installations won't get the most recent stable release.

[candlerb]

Does that sound right?

Yes, I believe so, thank you.

• Update the pinned version of a specific package as needed per bug reports

Speaking as a user of Netbox, that would be a big plus point. I am unlikely to be monitoring for security alerts on all of Netbox's recursive package dependencies; but if Netbox comes out with a new release because a dependency had to be updated, I will notice that and will upgrade.

Regarding updating all pinned versions on minor (i.e. 2.x) release: I'd say that's a good time to do it.

IMO if we're going to pin everything we might as well pin Django too. The downside is that new installations won't get the most recent stable release.

I think it's right to pin it. If a new Django release doesn't do anything which fixes a problem observable by Netbox users, then there's no need to upgrade. Having a known working version is more important. But as above: if Django makes a point release because of a security issue, then having a corresponding bump in Netbox gives a useful notification of the need to upgrade. Plus, hopefully the Netbox developers have done at least a quick regression test first.

[tyler-8]

There are some common tools out there that can help maintain these workflows and track requirements more intelligently than what requirements.txt provides

• pyup for automated dependency updates - which could hook in to an automated CI/CD like travis
• poetry which helps maintain a better list of requirements (using lock files and pyproject.toml) - that is also exportable to requirements.txt for users.

[jeremystretch]

It occurs to me to point out that our CI tests rebuild a virtual environment per requirements.txt on every run, and never seem to encounter a problem with dependency management. Since we're looking at moving to recommend a virtualenv-based installation anyway (see #3949), this may be easily resolved by simply rebuilding the virtual environment from scratch on each upgrade.

[jeremystretch]

I've just implemented #3949, which moves the official NetBox implementation into a Python virtual environment. During an upgrade, the virtualenv is destroyed and completely rebuilt, effectively replicating the same process we use for our CI tests. With this approach in place, it's difficult to imagine a dependency issue that would persist for a user but escape our integration tests.

I'm going to close this issue as I believe the us of virtual environments negates the concerns it has raised. We can revisit if any further dependency issues arise.