Using python entrypoints for notebook server extensions?

[yuvipanda]

Heya!

Notebook server extensions now need an extra step after being pip installed to be enabled. However, since they're purely python, we could just discover them via a well known Python Entry Point - and then just pip install would work.

I tried looking for discussion about this in this repo, and couldn't find any. #116 was the closest I could find, but that seems a much bigger discussion than just serverextensions. Server Extensions are always Python, they can be notebook specific, etc - so all those would be side stepped.

I found http://amir.rachum.com/blog/2017/07/28/python-entry-points/ to be a very good description of entrypoints in python, and I really like how pytest uses them.

If this seems like a good idea, I might take a shot at writing up a patch. If there was discussion about this elsewhere, telling me where to search would also be highly appreciated!

[maartenbreddels]

I was thinking about this.. maybe a few hours ago :)
But regarding nbextensions, see also #2824
I like the idea!

[takluyver]

I think for server extensions it makes sense to use entry points. It's less clear for nbextensions: they are usually installed and served through Python machinery, but in principle there's nothing Python specific about them, and we may one day regret tying them to Python mechanisms. On the other hand, if server extensions are loaded by entry points, it would be a nice symmetry for nbextensions to use the same, and it would mean that extensions could be cleanly uninstalled with the same tools used to install them.

This also ties into our discussion on #2824 about moving to a model where installed extensions are automatically loaded unless they have been disabled, rather than requiring an enable step as we currently do.

See also my entrypoints module, which we're already using in nbconvert.

[takluyver]

I'm leaning towards using entry points for both server extensions and nbextensions:

1. There are real user benefits: easier installation and uninstallation.
2. Using the same system for both kinds of extensions makes things simpler for extension authors.
3. We have no plans to move away from a Python-based server, as far as I know.
4. If one day we do switch to a server written in e.g. node or Go, I don't think it would be all that hard to update nbextensions for another kind of discovery.

[juhasch]

👍 For server extensions using entry points. This would also allow installation while the notebook server is running.

For extensions themselves that sounds good, too. Definitely worth considering IMO.

[maartenbreddels]

I like that it is tied to the Python environment, it will no leak out of the Python environment like it currently does (if you don't include --sys-prefix, at least for nbextensions)

[juhasch]

Hm, but that may the thing that breaks the use case for notebook extensions ?
Using entry points, all notebook extensions need to be installed in the notebook server environment.

Now when I use a different Python environment as my kernel, how do I get nbconvert support or additional templates tied to the Python package ?
Also I really like being able to have notebook extensions per environment and not per notebook server instance.

[bollwyvl]

+1, big fan of entry_points as the least-worst solution for extensibility, especially with the non-setuptools version by @takluyver, and a win from a development, packaging, documentation, user experience point of view.

Presumably, we'd just duplicate part of the traitlets config namespaces:

setup(
    # ...
    entry_points={
        'notebook.NotebookApp.nbserver_extensions: 'my-serverextension = my_serverextension'
    }
)

Where, because nbserver_extensions is a dict of importable locations of the magic-named function.

Anyhoo, thinking about that, though: some of the more powerful/evil "extensions" out there aren't serverextensions (or aren't just server extensions), and need to touch other parts of (multiple) jupyter_[*_]config.

Having a "one entry point to rule them all" for config would be pretty concise:

setup(
    # ...
    entry_points={
        'jupyter_notebook_config': 'my-contents-kernel-auth-manager-serverextension = mckam:my_custom_jupyter_config'
    }
)
# ...
def my_custom_jupyter_config(c):
    c.NotebookApp.contents_manager_class = Foo
    c.NotebookApp.nbserverextensions["foo"] = Bar
    ...

Presumably this would take effect right before the on-disk $PREFIX/etc/jupyter, such that it could still be disabled/whatever down the line by file configuration.

As an extension developer, you're still writing a function, but it's not a magic-named function, and the magic name appears in the least-worst place (setup.py), but you can configure whatever you need to affect the change.

[yuvipanda]

w00t, this is all awesome :) Looks like there's enough other people more experienced & invested in this than me to make this happen, so I'm unlicking this cookie :D I'm happy with anything y'all come up with as long as I can just do 'pip install' and it just works :) <3

[takluyver]

For reference, Donald Stufft mentioned some problems with entry points in a thread on distutils-sig. I don't think it should necessarily stop us using them (especially because I'm not sure there's any alternative without similar issues), but it's worth being aware of the drawbacks:

One that I was helping someone debug just the other day is that they’re super non-debuggable and the behavior when you have two things providing the same entry point is basically ???? (If I remember correctly, the behavior is that the first thing found is the one that “wins”, which means the ordering of sys.path and the names of the projects supply it is what determines it). This got exposed to the end user that they installed something that they thought was going to add support for something, but which silently did nothing because two different project happened to pick the same name for their entry point (not the group, it was two things providing plugins for the same system).

Of course there is the perennial entrypoints are super slow, which is partially the fault of pkg_resources does a bunch of import time logic, but also because scanning sys.path for all installed stuff is just slow.

They’re also somewhat fragile since they rely on the packaging metadata system at runtime, and a number of tools exclude that information (often times things that deploy stuff as a tarball/zipfile) which causes regular issues to be opened up for these projects when they get used in those environments.

Those are the ones I remember because they come up regularly (and people regularly come to me with issues with any project related to packaging in any way even for non packaging related features in those projects). I’m pretty sure there were more of them that I’ve encountered and seen projects encounter, but I can’t remember them to be sure.

I have some plans to do caching to mitigate the slowness issue (that's partly what the distutils-sig thread is about) - but that's inevitably going to lead to a new kind of problem, with caches getting out of date. We'll have to wait for the implementation to know how big a problem that will be in practice.

[maartenbreddels]

FWIW, I'm trying to use entry points as well, and I am not sure how good support is on conda. The meta.yaml has an entry_points entry, but that assumes they are console_scripts. In an old thread, it is adviced to put in

build:
  preserve_egg_dir: True

@takluyver could you explain what entrypoints does? I am not sure why it is needed next to setuptools/pkg_resources?

[takluyver]

I think it should work through conda so long as you use pip to build the package. We don't appear to need any other options when doing that - I've just checked the IPython package from conda-forge, and it has the expected entry points in the right place (both console_scripts and pygments.lexers)

entrypoints is designed to do basically the same thing as pkg_resources, without being part of setuptools. There are some practical benefits - pkg_resources does a load of operations on import, which can make startup slow (can be an extra 5 seconds on a spinning HDD) - but it's also because I've been bitten so many times by setuptools that i will go to some lengths to avoid it.

[blink1073]

The jupyterlab recipe also includes entry points but without using pip (by passing extra flags to setup.py).

[maartenbreddels]

@takluyver thanks, i'll take that up with the conda-forge people to see what their preference is for non-console-entry points. Could you maybe explain the entrypoints goals on the documentation page, because I think it is useful to know, and a useful package :).

@blink1073 I was referring to non-console entry points, or did I miss your point?

[blink1073]

Ah, I haven't used those before in conda, but I suspect they should work the same with that invocation.