New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"ImportError: No module named jaraco.functools" raised in app frozen by py2exe #1722

Closed
marunguy opened this Issue Jul 5, 2018 · 20 comments

Comments

Projects
None yet
5 participants
@marunguy

marunguy commented Jul 5, 2018

  • Windows 8.1 64bit
  • Python 2.7.15 32bit
  • py2exe 0.6.10a1
  • CherryPy 16.0.2
  • Cheroot 6.3.2.post0
  • tempora 1.13
  • jaraco.functools 1.20

"ImportError: No module named jaraco.functools" raised in my app frozen by py2exe.

D:\prj\test\dist_py2exe>test.exe
----------------------------------------
main : Unhandled error
Traceback (most recent call last):
...
  File "cherrypy\process\servers.pyo", line 126, in <module>
  File "zipextimporter.pyo", line 74, in load_module
  File "portend.pyo", line 18, in <module>
  File "zipextimporter.pyo", line 74, in load_module
  File "tempora\timing.pyo", line 12, in <module>
ImportError: No module named jaraco.functools

jaraco.functools seems the namespace package.
This issue may be same to jaraco/jaraco.classes#3

Is allowed the namespace package in CherryPy?
#1673

@benjaoming

This comment has been minimized.

Contributor

benjaoming commented Jul 5, 2018

Our project also broke despite having pinned the CherryPy version. Essentially, the issue comes from a new version of tempora, 1.13, which is a dependency of portend (a dependency of cherrypy).

cherrypy doesn't put any upper limits on requirements, which is a bit of an issue in itself, supposing that these requirements use semantic versioning. Which my impression is that they do.

Issues are:

  • backports.functools_lru_cache (3 watchers) is a namespaced dependency of cheroot.
  • jaraco.functools>=1.20 (1 watchers) is a namespaced dependency of tempora (1 watchers), a dependency of portend (1 watchers).

Apart from the complex dependency tree, these trickle-down regressions are also inherent of utilizing tiny, obscure libraries (perhaps namespaced) from unwatched repos. In order to offset this, I would propose to at least put upper bounds on minor versions. Supposing that semantics are followed, then minor versions allow for dependency changes, while patch versions do not.

So the dependencies of cheroot and portend should be changed to:

install_requires = [
    'six>=1.11.0',
    'cheroot>=6.2.4,<6.3',
    'portend>=2.1.1,<2.2',
]

...and portend has to be changed to:

install_requires=[
    'tempora>=1.8,<1.9',
],

...and tempora has to be changed to:

install_requires=[
    'six',
    'pytz',
    'jaraco.functools>=1.20,<1.21',
],

Please also read this Wikipedia article, since it is in fact a real issue. I'm not being "cheeky" or anything, I just want us all to have less of this sort of Sisyphus work: https://en.wikipedia.org/wiki/Dependency_hell

@webknjaz

This comment has been minimized.

Member

webknjaz commented Jul 10, 2018

cc @jaraco

@jaraco

This comment has been minimized.

Member

jaraco commented Jul 10, 2018

You're right - the additional dependency of jaraco.functools from tempora has resurfaced the issue.

minor versions allow for dependency changes

Normally, I consider dependency changes to be non-breaking, so unnecessary to pin. Maybe I should consider the introduction of a new dependency a backward-incompatible change (that cascades to every depending library), but that seems extreme. Maybe the addition of a (first) namespace package dependency should be considered backward-incompatible, due to the known bugs in the ecosystem.

So the dependencies of cheroot and portend should be changed to:
...and portend has to be changed to:
...and tempora has to be changed to:

I think you're illustrating my main objection to setting upper bounds on dependencies. I don't want to take on the responsibility of managing cascades of version bumps.

Moreover, if we enact the bounds you suggest, and then cherrypy needs a fix to portend or tempora, it can't have it because it's stuck on an old version. Adding arbitrary upper bounds creates as many problems as it solves, but also takes the conservative approach of disallowing automatic fixes and updates.

CherryPy's tests pass against the latest dependencies, meaning they're viable as declared. The defect lies in the (admittedly unintentional) adoption of namespace packages.

In my opinion, the value of exporting and presenting this functionality as encapsulated, purposefully-organized dependencies far outweighs the costs of copy/pasting the functionality into a dozen or more applications and maintaining the inevitable divergence that comes from not honoring DRY.

Perhaps for the short term, I can vendor (aka copy/paste) the function from jaraco.functools into tempora and save the dependency.

For the long term, I'll cut a new release of CherryPy that explicitly pins the dependencies above, and then CherryPy will drop support for py2exe (or other packaging systems without namespace package support).

@jaraco

This comment has been minimized.

Member

jaraco commented Jul 10, 2018

Perhaps for the short term, I can vendor the function

On second thought, there's no value in that. Let's focus on the long-term solution.

@jaraco

This comment has been minimized.

Member

jaraco commented Jul 10, 2018

I'm afraid I don't understand why the presence of backports.functools_lru_cache in cheroot didn't trigger this same issue.

@jaraco jaraco closed this in 2496611 Jul 10, 2018

@jaraco

This comment has been minimized.

Member

jaraco commented Jul 10, 2018

I see - cheroot only attempts to import backports.functools_lru_cache on older Pythons, so that's why it didn't affect py2exe (Python 3.4).

@jaraco

This comment has been minimized.

Member

jaraco commented Jul 10, 2018

tiny, obscure libraries

I should also say too - if there is a more widely-used library supplying the same functionality, I'm happy to deprecate the functionality, especially in the jaraco.* packages, in favor of the more popular packages.

@benjaoming

This comment has been minimized.

Contributor

benjaoming commented Jul 10, 2018

For the long term, I'll cut a new release of CherryPy that explicitly pins the dependencies above, and then CherryPy will drop support for py2exe (or other packaging systems without namespace package support).

First of all, thanks for responding to the issue.

We can pin these versions in our own packages, so you don't have to do another release for that reason. That is not a solution, but a work-around. What we were interested in, was to be able to depend on CherryPy and its future updates, with a healthy dependency tree, using semantic versioning and upper bounds derived from those semantics.

I'll have to consider what the deprecation means to our distribution and using CherryPy, but I have to say (as an open source enthusiast) that it's problematic that you are reluctant to take advice or receive help when it's offered. Other than that, thanks for your efforts.

@webknjaz

This comment has been minimized.

Member

webknjaz commented Jul 10, 2018

@benjaoming we cannot afford to support outdated envs, which don't even try to mitigate their issues. Supporting them is a workaround or even hack itself.
We don't have an army of maintainers and tons of CI resources to be able to fine tune stuff every 5 minutes, so we decide to at least concentrate efforts on supporting widely used things, which also care about being up-to-date.
The best course of action for you is to suggest py2exe folks fix their issues and release an update which would better match modern reality. We need to move forward and supporting broken things only blocks us from focusing on important things.

@benjaoming

This comment has been minimized.

Contributor

benjaoming commented Jul 10, 2018

@webknjaz the fundamental issue is the absence of upper bounds in the dependency tree.

@webknjaz

This comment has been minimized.

Member

webknjaz commented Jul 10, 2018

Putting them there will create another fundamental issue: having to constantly maintain updates of deps.
While it's a good practice for applications, for libraries this is harmful, since it limits their users who might want to combine newer version of another lib, for example, with older version of CherryPy which heavily restricts that lib. That's why requirements.txt (and/or -c constraints.txt) based approach for apps exists in the first place, so that apps could rely on stable environment, where you are expected to actually pin versions.

@benjaoming

This comment has been minimized.

Contributor

benjaoming commented Jul 11, 2018

@webknjaz I agree that for instance single environment deployments, like web site projects, should pin versions to achieve 1:1 between dev/test/staging and production environments.

However, when a library is used in another library -- for instance other distributed apps depending on CherryPy -- then you are effectively off-loading the responsibility to discover and maintain dependency version updates at the level of each individual installation of your upstream dependencies (and their dependencies etc). To put things on the edge, you are saying "what we do not intend to do about upper bounds to dependency versions, we expect everyone else to be doing, either by pinning or by upper bounds".

Not being a CherryPy maintainer, it takes a huge amount of my time to discover and understand your dependency tree when things break.

I realize that it's extra work on your side to maintain upper bounds to versions of your dependencies and their dependencies, but I would like to point out two things here: 1) You guys are already maintaining those libraries, so you'll know first-hand when updates are out, and what the version bumps mean, and 2) most of this dependency issue can be solved by internalizing the tiny functions that have been split unto separate libraries.

@webknjaz

This comment has been minimized.

Member

webknjaz commented Jul 11, 2018

We have CI which runs tests on daily basis, so we notice when something needs fixing/adjustments, but still in your case you need to pin versions when you ship software. It's your responsibility as a packager of application. While our responsibility is to not limit on what you can combine with our library.
Ideally you should run your tests against CherryPy's master automatically traversing dependencies, like we do with Cheroot, for example. This ensures forward compatibility.

As for third-parties we use we just don't want to have a lot of unmanageable copy-paste across all projects it's used in.
When we discover some popular libraries that have same features we switch to them of course. But the primary motivation is to track and version reused code in one place.

With that said CherryPy has always been trying to give maximum flexibility to its users on low level. If you want to restrict yourself it's fine for your use-case, but might be unfortunate for other people.

I'm pretty sure that your build system should resolve these things. If you wanted something opinionated you could go for flask and django, but CherryPy promotes different Zen.
I can agree that our docs could be better at describing these cases and providing examples of how one could solve this situation. Feel free to contribute that if you have something on your mind :)

@webknjaz

This comment has been minimized.

Member

webknjaz commented Jul 11, 2018

@jaraco what if we additionally published dists with .post0 version and all dependencies strictly pinned?

@benjaoming

This comment has been minimized.

Contributor

benjaoming commented Jul 11, 2018

Hi @webknjaz - thanks for the answer.

Unfortunately, we agree very much that pinned versions are not the solution for a distributed re-usable app/library. Pinned versions and unbounded dependencies are bad in each their way. With pinned dependencies, a patch release (for instance a security fix) would prompt all depending packages to re-release with changed dependencies. This completely breaks distribution. If there was a CherryPy release with pinned dependencies, you would have to either 1) ignore if dependencies were patched or 2) re-release every time dependencies were patched. Neither seems desirable.

@benjaoming

This comment has been minimized.

Contributor

benjaoming commented Jul 11, 2018

I don't want to discourage the development of CherryPy by being a critique, so I will leave this be. Thanks for sharing your views and opinions.

If time permits, I'll write a more general blog post about using semantic versioning, reusable libraries, their dependency trees, and the security issues inherent from having complex and/or unreliable dependency trees.

@jaraco

This comment has been minimized.

Member

jaraco commented Jul 11, 2018

What we were interested in, was to be able to depend on CherryPy and its future updates, with a healthy dependency tree, using semantic versioning and upper bounds derived from those semantics.

Agree this is worthwhile and I also believe it's our intention. I think our perspectives diverge on what those semantics communicate. Normally, I wouldn't consider adding or changing a dependency as a backward-incompatible change and so I don't typically shepherd a project through a major version bump for such a change (as long as that project's interface remains compatible assuming the dependency is met as declared). Or if we agree that a minor version bump accurately reflects the intention of a dependency change, and we could then start pinning against minor versions out of the potential instability that could bring, but that approach brings its own maintenance burden.

I don't think there's an obvious answer here that doesn't have tradeoffs. I appreciate your patience when we encounter situations like these, which I believe are rare.

@benjaoming

This comment has been minimized.

Contributor

benjaoming commented Jul 11, 2018

@jaraco I think a set of best-practices for semantic versioning of Python packages distributed on PyPi would be ideal. Not sure if anyone did that. For instance, these best practices could include:

  • Bumping the upper bound of a minor version of a dependency, having done a reasonable level of regression testing: Release with a patch update

  • Bumping the upper bound of a minor version of a dependency that also bumped minor versions of its dependencies: More likely to aspire for a minor version bump in the package itself

  • Bumping the upper bound of a major version of a dependency: Definitely a minor version bump in the package itself. For instance, introducing Django 2.0 support.

  • Bumping lower bounds: Same as the recommendations for upper bounds.

But since the above is rather hard to communicate (the horribly unreadable sentences), I would have to take a lot of time to actually write it down :)

@wkerzendorf

This comment has been minimized.

wkerzendorf commented Aug 16, 2018

just wanted to bump this thread: current version on conda-forge exhibits the same behaviour under python 3. 6

fix can be to downgrade tempora to 1.8

@webknjaz

This comment has been minimized.

Member

webknjaz commented Aug 16, 2018

Conda seems to not support namespaced packages properly: https://github.com/conda-forge/staged-recipes/wiki/Namespace-packages

They need to fix this to follow PEPs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment