extensions: allow extensions in namespace packages

[oliver-sanders]

This allows Jupyter extensions to be developed in Python namespace packages.

Related to: #222 (comment)

See PEP420.

Currently Jupyter Server extensions developed in Python namespace packages fall over with some nasty traceback:

$ t_ns_jps_ext
[I 2021-05-19 12:40:39.457 ServerApp] t_namespace | extension was successfully linked.
Traceback (most recent call last):
  File ".../t_ns_jps_ext", line 33, in <module>
    sys.exit(load_entry_point('jps-namespace-extension', 'console_scripts', 't_ns_jps_ext')())
  File ".../python3.9/site-packages/jupyter_server/extension/application.py", line 518, in launch_instance
    serverapp = cls.initialize_server(argv=args)
  File ".../python3.9/site-packages/jupyter_server/extension/application.py", line 488, in initialize_server
    serverapp.initialize(
  File ".../python3.9/site-packages/traitlets/config/application.py", line 87, in inner
    return method(app, *args, **kwargs)
  File ".../python3.9/site-packages/jupyter_server/serverapp.py", line 1855, in initialize
    point = self.extension_manager.extension_points[starter_extension]
KeyError: 'test_namespace_extension'

Here's an example extension developed in a Python namespace package: https://github.com/oliver-sanders/jupyter-server-namespace-extension-test

[welcome]

Thanks for submitting your first pull request! You are awesome! 🤗

If you haven't done so already, check out Jupyter's Code of Conduct. Also, please make sure you followed the pull request template, as this will help us review your contribution more quickly.

You can meet the other Jovyans by joining our Discourse forum. There is also a intro thread there where you can stop by and say Hi! 👋

Welcome to the Jupyter community! 🎉

[codecov-commenter]

Codecov Report

Merging #523 (4a82af5) into master (f29bb11) will increase coverage by 0.04%.
The diff coverage is 83.33%.

@@            Coverage Diff             @@
##           master     #523      +/-   ##
==========================================
+ Coverage   76.30%   76.34%   +0.04%     
==========================================
  Files         110      110              
  Lines        9814     9835      +21     
  Branches     1067     1069       +2     
==========================================
+ Hits         7489     7509      +20     
- Misses       1944     1946       +2     
+ Partials      381      380       -1     

Impacted Files	Coverage Δ
jupyter_server/extension/application.py	71.63% <20.00%> (-1.05%)	⬇️
jupyter_server/tests/test_utils.py	100.00% <100.00%> (ø)
jupyter_server/utils.py	64.77% <100.00%> (+1.62%)	⬆️
jupyter_server/services/kernels/kernelmanager.py	80.79% <0.00%> (+0.72%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f29bb11...4a82af5. Read the comment docs.

[Zsailer]

Thanks, @oliver-sanders. This is awesome!

This was briefly mentioned in the Jupyter Server meeting today. The "committers" around Jupyter Server have been a little low on bandwidth the past couple of weeks, so review has been a little slower than normal. You should start getting some review over the next week. 😎

[oliver-sanders]

Rebased and switched to a nicer way of detecting namespace packages by using the __file__ attribute, this is part of the Python language specification. Updated the PR description with explanation.

[Zsailer]

Thanks, @oliver-sanders. 🚀

I think this change is good as-is, even though we duplicate some importing in the ExtensionPoint API

jupyter_server/jupyter_server/extension/manager.py

Lines 50 to 55 in f29bb11

	self._module = importlib.import_module(self._module_name)
	except ImportError:
	raise ExtensionModuleNotFound(
	"The submodule '{}' could not be found. Are you "
	"sure the extension is installed?".format(self._module_name)
	)

In the long term, I believe there is some work to do around extension distribution/packaging/discovery. The get_extension_package classmethod is pretty brittle. It’s particularly annoying that extensions loaded from the current working directory (i.e. not from site-packages; usually happens when you’re developing the extension) are given the name __main__ by the ExtensionManager. I just haven’t had a change to revisit this yet.

[oliver-sanders]

Two One test failures which appears unrelated.

[edit] fixed in rerun
Warning: Failed to download action 

ERROR: Could not find a version that satisfies the requirement jupyter_packaging<2,~=0.9 (from versions: none)

[oliver-sanders]

[update] This PR is hanging on:

• Conclusion of the implementation discussion above
  • No especially nice solutions, I've outlined two options, need to pick one.
• Likely unrelated documentation build failure
  • ERROR: No matching distribution found for jupyter_packaging<2,~=0.9

[blink1073]

Hi @oliver-sanders, can you please see yesterday's meeting notes about this topic? Thanks again for your patience on this one!

[oliver-sanders]

@blink1073 sure, no problem, I'm happy to implement this however is desired.

Currently, PR relies on private APIs, which is concerning.

It's not ideal, but I don't think it's a big issue because this implementation is brittle (i.e. will fail if the internals change). So if we ran the CI test battery against the latest dev version of Python (currently 3.10-dev) we would get ample early warning to patch any issues that crop up.

Can we build a more "defensive" way to discover namespace packages?

• Make get_namespace_package try a series of steps to find a NS package
• Try importing the private standard lib API to inspect the namespace package (how it's currently implemented in the PR)
• If those private APIs are not found (i.e. change in the future), fallback to trying to import the namespace package directly.
• If that fails, bail gracefully.

It would appear there is no especially nice solution to this problem so it comes down to picking a nasty method and ensuring that it is well tested.

I don't think implementing two nasty methods would be better than implementing one. The failure of the first method (which should never happen) would be silently suppressed which is nasty. The second method (which would never be run unless the first fails which it shouldn't) then becomes dead code and a potential bug in waiting.

[Zsailer]

Ok, I’m convinced. 😄

Let’s stick with option 1 (as implemented). We can add Python latest dev version (3.10.x) to our Github actions in a separate PR, perhaps #541.

@oliver-sanders, this is great stuff. Thank you for many of your recent contributions. If you’re ever available to make it to our weekly Jupyter Server meetings, we’d love to meet you!

[oliver-sanders]

Cheers, added 3.10-dev in #541, Linux/MacOs tests pass, Windows fails to pip install (waiting on 3.10 support for pywin32?). I'll try and pop by a meeting sometime to say hi.

[welcome]

Congrats on your first merged pull request in this project! 🎉

Thank you for contributing, we are very proud of you! ❤️