"ImportError: No module named jaraco.functools" raised in app frozen by py2exe

[marunguy]

• Windows 8.1 64bit
• Python 2.7.15 32bit
• py2exe 0.6.10a1
• CherryPy 16.0.2
• Cheroot 6.3.2.post0
• tempora 1.13
• jaraco.functools 1.20

"ImportError: No module named jaraco.functools" raised in my app frozen by py2exe.

D:\prj\test\dist_py2exe>test.exe
----------------------------------------
main : Unhandled error
Traceback (most recent call last):
...
  File "cherrypy\process\servers.pyo", line 126, in <module>
  File "zipextimporter.pyo", line 74, in load_module
  File "portend.pyo", line 18, in <module>
  File "zipextimporter.pyo", line 74, in load_module
  File "tempora\timing.pyo", line 12, in <module>
ImportError: No module named jaraco.functools

jaraco.functools seems the namespace package.
This issue may be same to jaraco/jaraco.classes#3

Is allowed the namespace package in CherryPy?
#1673

[benjaoming]

Our project also broke despite having pinned the CherryPy version. Essentially, the issue comes from a new version of tempora, 1.13, which is a dependency of portend (a dependency of cherrypy).

cherrypy doesn't put any upper limits on requirements, which is a bit of an issue in itself, supposing that these requirements use semantic versioning. Which my impression is that they do.

Issues are:

• backports.functools_lru_cache (3 watchers) is a namespaced dependency of cheroot.
• jaraco.functools>=1.20 (1 watchers) is a namespaced dependency of tempora (1 watchers), a dependency of portend (1 watchers).

Apart from the complex dependency tree, these trickle-down regressions are also inherent of utilizing tiny, obscure libraries (perhaps namespaced) from unwatched repos. In order to offset this, I would propose to at least put upper bounds on minor versions. Supposing that semantics are followed, then minor versions allow for dependency changes, while patch versions do not.

So the dependencies of cheroot and portend should be changed to:

install_requires = [
    'six>=1.11.0',
    'cheroot>=6.2.4,<6.3',
    'portend>=2.1.1,<2.2',
]

...and portend has to be changed to:

install_requires=[
    'tempora>=1.8,<1.9',
],

...and tempora has to be changed to:

install_requires=[
    'six',
    'pytz',
    'jaraco.functools>=1.20,<1.21',
],

Please also read this Wikipedia article, since it is in fact a real issue. I'm not being "cheeky" or anything, I just want us all to have less of this sort of Sisyphus work: https://en.wikipedia.org/wiki/Dependency_hell

[webknjaz]

cc @jaraco

[jaraco]

You're right - the additional dependency of jaraco.functools from tempora has resurfaced the issue.

minor versions allow for dependency changes

Normally, I consider dependency changes to be non-breaking, so unnecessary to pin. Maybe I should consider the introduction of a new dependency a backward-incompatible change (that cascades to every depending library), but that seems extreme. Maybe the addition of a (first) namespace package dependency should be considered backward-incompatible, due to the known bugs in the ecosystem.

So the dependencies of cheroot and portend should be changed to:
...and portend has to be changed to:
...and tempora has to be changed to:

I think you're illustrating my main objection to setting upper bounds on dependencies. I don't want to take on the responsibility of managing cascades of version bumps.

Moreover, if we enact the bounds you suggest, and then cherrypy needs a fix to portend or tempora, it can't have it because it's stuck on an old version. Adding arbitrary upper bounds creates as many problems as it solves, but also takes the conservative approach of disallowing automatic fixes and updates.

CherryPy's tests pass against the latest dependencies, meaning they're viable as declared. The defect lies in the (admittedly unintentional) adoption of namespace packages.

In my opinion, the value of exporting and presenting this functionality as encapsulated, purposefully-organized dependencies far outweighs the costs of copy/pasting the functionality into a dozen or more applications and maintaining the inevitable divergence that comes from not honoring DRY.

Perhaps for the short term, I can vendor (aka copy/paste) the function from jaraco.functools into tempora and save the dependency.

For the long term, I'll cut a new release of CherryPy that explicitly pins the dependencies above, and then CherryPy will drop support for py2exe (or other packaging systems without namespace package support).

[jaraco]

Perhaps for the short term, I can vendor the function

On second thought, there's no value in that. Let's focus on the long-term solution.

[jaraco]

I'm afraid I don't understand why the presence of backports.functools_lru_cache in cheroot didn't trigger this same issue.

[jaraco]

I see - cheroot only attempts to import backports.functools_lru_cache on older Pythons, so that's why it didn't affect py2exe (Python 3.4).

[jaraco]

tiny, obscure libraries

I should also say too - if there is a more widely-used library supplying the same functionality, I'm happy to deprecate the functionality, especially in the jaraco.* packages, in favor of the more popular packages.

[benjaoming]

For the long term, I'll cut a new release of CherryPy that explicitly pins the dependencies above, and then CherryPy will drop support for py2exe (or other packaging systems without namespace package support).

First of all, thanks for responding to the issue.

We can pin these versions in our own packages, so you don't have to do another release for that reason. That is not a solution, but a work-around. What we were interested in, was to be able to depend on CherryPy and its future updates, with a healthy dependency tree, using semantic versioning and upper bounds derived from those semantics.

I'll have to consider what the deprecation means to our distribution and using CherryPy, but I have to say (as an open source enthusiast) that it's problematic that you are reluctant to take advice or receive help when it's offered. Other than that, thanks for your efforts.

[webknjaz]

@benjaoming we cannot afford to support outdated envs, which don't even try to mitigate their issues. Supporting them is a workaround or even hack itself.
We don't have an army of maintainers and tons of CI resources to be able to fine tune stuff every 5 minutes, so we decide to at least concentrate efforts on supporting widely used things, which also care about being up-to-date.
The best course of action for you is to suggest py2exe folks fix their issues and release an update which would better match modern reality. We need to move forward and supporting broken things only blocks us from focusing on important things.

[benjaoming]

@webknjaz the fundamental issue is the absence of upper bounds in the dependency tree.

[webknjaz]

Putting them there will create another fundamental issue: having to constantly maintain updates of deps.
While it's a good practice for applications, for libraries this is harmful, since it limits their users who might want to combine newer version of another lib, for example, with older version of CherryPy which heavily restricts that lib. That's why requirements.txt (and/or -c constraints.txt) based approach for apps exists in the first place, so that apps could rely on stable environment, where you are expected to actually pin versions.

[benjaoming]

@webknjaz I agree that for instance single environment deployments, like web site projects, should pin versions to achieve 1:1 between dev/test/staging and production environments.

However, when a library is used in another library -- for instance other distributed apps depending on CherryPy -- then you are effectively off-loading the responsibility to discover and maintain dependency version updates at the level of each individual installation of your upstream dependencies (and their dependencies etc). To put things on the edge, you are saying "what we do not intend to do about upper bounds to dependency versions, we expect everyone else to be doing, either by pinning or by upper bounds".

Not being a CherryPy maintainer, it takes a huge amount of my time to discover and understand your dependency tree when things break.

I realize that it's extra work on your side to maintain upper bounds to versions of your dependencies and their dependencies, but I would like to point out two things here: 1) You guys are already maintaining those libraries, so you'll know first-hand when updates are out, and what the version bumps mean, and 2) most of this dependency issue can be solved by internalizing the tiny functions that have been split unto separate libraries.

[webknjaz]

We have CI which runs tests on daily basis, so we notice when something needs fixing/adjustments, but still in your case you need to pin versions when you ship software. It's your responsibility as a packager of application. While our responsibility is to not limit on what you can combine with our library.
Ideally you should run your tests against CherryPy's master automatically traversing dependencies, like we do with Cheroot, for example. This ensures forward compatibility.

As for third-parties we use we just don't want to have a lot of unmanageable copy-paste across all projects it's used in.
When we discover some popular libraries that have same features we switch to them of course. But the primary motivation is to track and version reused code in one place.

With that said CherryPy has always been trying to give maximum flexibility to its users on low level. If you want to restrict yourself it's fine for your use-case, but might be unfortunate for other people.

I'm pretty sure that your build system should resolve these things. If you wanted something opinionated you could go for flask and django, but CherryPy promotes different Zen.
I can agree that our docs could be better at describing these cases and providing examples of how one could solve this situation. Feel free to contribute that if you have something on your mind :)