Different dependency versions installed in dev versus final Docker images

[glennmatthews]

Environment

• Python version: any
• Nautobot version: 1.1.1

Steps to Reproduce

1. Build and run dev docker image and run pip list
2. Build and run final docker image and run pip list
3. Compare outputs

Expected Behavior

Installed packages should be identical except for the additional dev-only dependencies present in the dev image.

Observed Behavior

In several cases final has newer versions of various dependencies than dev, for example:

• dev has idna 2.10 whereas final has idna 3.2
• dev has packaging 20.9 whereas final has packaging 21.0
• dev has requests 2.25.1 whereas final has requests 2.26.0
• etc.

This is problematic as it means that final and dev may show different behavior, have different bugs or fixes, etc.

The root cause appears to be that the dev image installs its dependencies based on exact versions pinned in poetry.lock, whereas the final image installs its dependencies based on the looser version constraints defined in pyproject.toml.

Related Poetry FR and PR: python-poetry/poetry#2778, python-poetry/poetry#3341

[glennmatthews]

One possibility might be to use https://bneijt.nl/pr/poetry-lock-package/ to build a "locked" wheel and install that one in the final Docker image so as to ensure that it has the same dependencies as the dev image. The potential downside is that this will prevent any consumers of the final image from doing a pip install --upgrade of any dependencies, even for security updates and such, since the dependency versions will be locked.

Another option would be to aggressively ensure that for our release version images, at least, that we are consistently running poetry update prior to release so as to minimize the delta between what's in dev images and what are the latest acceptable versions of our dependencies installed in final images.

[glennmatthews]

Related: #1097.

[glennmatthews]

Reopening as #1323 had to be rolled back by #1325.

[glennmatthews]

There seem to be quite a few open issues against Poetry involving incorrect dependency declarations - any or all of these may be causing the issues we encountered with #1323, but dropping support for Python 3.6 (#1268) will at least in the short term allow us to avoid the currently encountered issue as it'll let us remove the split dependencies that we currently have for Python 3.6 versus all later versions, which are causing the current failures. However there is a concern that at some point in the future we may find ourselves in this same position (e.g. with respect to Python 3.7) so getting the issues fixed in Poetry would still be highly desirable.

• poetry export compose conditions with and, but should use or python-poetry/poetry#4428
• poetry export makes faulty conditions for package python-poetry/poetry#4461
• Incorrect locking of platform-specific wheel dependencies python-poetry/poetry#4810
• Incorrect combination of environment markers python-poetry/poetry#4209
• Poetry v1.1.6 does not respect markers or version specifiers when generating lockfile python-poetry/poetry#3959
• export omits implicit markers python-poetry/poetry-plugin-export#25
• requirements.txt file generated by poetry export causes conflicting dependencies in pip python-poetry/poetry-plugin-export#32

[glennmatthews]

Fixed in next by #1620.