Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using python entrypoints for notebook server extensions? #2894

Open
yuvipanda opened this issue Oct 3, 2017 · 20 comments

Comments

@yuvipanda
Copy link
Contributor

commented Oct 3, 2017

Heya!

Notebook server extensions now need an extra step after being pip installed to be enabled. However, since they're purely python, we could just discover them via a well known Python Entry Point - and then just pip install would work.

I tried looking for discussion about this in this repo, and couldn't find any. #116 was the closest I could find, but that seems a much bigger discussion than just serverextensions. Server Extensions are always Python, they can be notebook specific, etc - so all those would be side stepped.

I found http://amir.rachum.com/blog/2017/07/28/python-entry-points/ to be a very good description of entrypoints in python, and I really like how pytest uses them.

If this seems like a good idea, I might take a shot at writing up a patch. If there was discussion about this elsewhere, telling me where to search would also be highly appreciated!

@maartenbreddels

This comment has been minimized.

Copy link
Contributor

commented Oct 3, 2017

I was thinking about this.. maybe a few hours ago :)
But regarding nbextensions, see also #2824
I like the idea!

@takluyver

This comment has been minimized.

Copy link
Member

commented Oct 3, 2017

I think for server extensions it makes sense to use entry points. It's less clear for nbextensions: they are usually installed and served through Python machinery, but in principle there's nothing Python specific about them, and we may one day regret tying them to Python mechanisms. On the other hand, if server extensions are loaded by entry points, it would be a nice symmetry for nbextensions to use the same, and it would mean that extensions could be cleanly uninstalled with the same tools used to install them.

This also ties into our discussion on #2824 about moving to a model where installed extensions are automatically loaded unless they have been disabled, rather than requiring an enable step as we currently do.

See also my entrypoints module, which we're already using in nbconvert.

@takluyver

This comment has been minimized.

Copy link
Member

commented Oct 3, 2017

I'm leaning towards using entry points for both server extensions and nbextensions:

  1. There are real user benefits: easier installation and uninstallation.
  2. Using the same system for both kinds of extensions makes things simpler for extension authors.
  3. We have no plans to move away from a Python-based server, as far as I know.
  4. If one day we do switch to a server written in e.g. node or Go, I don't think it would be all that hard to update nbextensions for another kind of discovery.
@juhasch

This comment has been minimized.

Copy link
Contributor

commented Oct 3, 2017

👍 For server extensions using entry points. This would also allow installation while the notebook server is running.

For extensions themselves that sounds good, too. Definitely worth considering IMO.

@maartenbreddels

This comment has been minimized.

Copy link
Contributor

commented Oct 3, 2017

I like that it is tied to the Python environment, it will no leak out of the Python environment like it currently does (if you don't include --sys-prefix, at least for nbextensions)

@juhasch

This comment has been minimized.

Copy link
Contributor

commented Oct 3, 2017

Hm, but that may the thing that breaks the use case for notebook extensions ?
Using entry points, all notebook extensions need to be installed in the notebook server environment.

Now when I use a different Python environment as my kernel, how do I get nbconvert support or additional templates tied to the Python package ?
Also I really like being able to have notebook extensions per environment and not per notebook server instance.

@bollwyvl

This comment has been minimized.

Copy link
Contributor

commented Oct 3, 2017

+1, big fan of entry_points as the least-worst solution for extensibility, especially with the non-setuptools version by @takluyver, and a win from a development, packaging, documentation, user experience point of view.

Presumably, we'd just duplicate part of the traitlets config namespaces:

setup(
    # ...
    entry_points={
        'notebook.NotebookApp.nbserver_extensions: 'my-serverextension = my_serverextension'
    }
)

Where, because nbserver_extensions is a dict of importable locations of the magic-named function.

Anyhoo, thinking about that, though: some of the more powerful/evil "extensions" out there aren't serverextensions (or aren't just server extensions), and need to touch other parts of (multiple) jupyter_[*_]config.

Having a "one entry point to rule them all" for config would be pretty concise:

setup(
    # ...
    entry_points={
        'jupyter_notebook_config': 'my-contents-kernel-auth-manager-serverextension = mckam:my_custom_jupyter_config'
    }
)
# ...
def my_custom_jupyter_config(c):
    c.NotebookApp.contents_manager_class = Foo
    c.NotebookApp.nbserverextensions["foo"] = Bar
    ...

Presumably this would take effect right before the on-disk $PREFIX/etc/jupyter, such that it could still be disabled/whatever down the line by file configuration.

As an extension developer, you're still writing a function, but it's not a magic-named function, and the magic name appears in the least-worst place (setup.py), but you can configure whatever you need to affect the change.

@yuvipanda

This comment has been minimized.

Copy link
Contributor Author

commented Oct 3, 2017

w00t, this is all awesome :) Looks like there's enough other people more experienced & invested in this than me to make this happen, so I'm unlicking this cookie :D I'm happy with anything y'all come up with as long as I can just do 'pip install' and it just works :) <3

@takluyver

This comment has been minimized.

Copy link
Member

commented Oct 20, 2017

For reference, Donald Stufft mentioned some problems with entry points in a thread on distutils-sig. I don't think it should necessarily stop us using them (especially because I'm not sure there's any alternative without similar issues), but it's worth being aware of the drawbacks:

One that I was helping someone debug just the other day is that they’re super non-debuggable and the behavior when you have two things providing the same entry point is basically ???? (If I remember correctly, the behavior is that the first thing found is the one that “wins”, which means the ordering of sys.path and the names of the projects supply it is what determines it). This got exposed to the end user that they installed something that they thought was going to add support for something, but which silently did nothing because two different project happened to pick the same name for their entry point (not the group, it was two things providing plugins for the same system).

Of course there is the perennial entrypoints are super slow, which is partially the fault of pkg_resources does a bunch of import time logic, but also because scanning sys.path for all installed stuff is just slow.

They’re also somewhat fragile since they rely on the packaging metadata system at runtime, and a number of tools exclude that information (often times things that deploy stuff as a tarball/zipfile) which causes regular issues to be opened up for these projects when they get used in those environments.

Those are the ones I remember because they come up regularly (and people regularly come to me with issues with any project related to packaging in any way even for non packaging related features in those projects). I’m pretty sure there were more of them that I’ve encountered and seen projects encounter, but I can’t remember them to be sure.

I have some plans to do caching to mitigate the slowness issue (that's partly what the distutils-sig thread is about) - but that's inevitably going to lead to a new kind of problem, with caches getting out of date. We'll have to wait for the implementation to know how big a problem that will be in practice.

@maartenbreddels

This comment has been minimized.

Copy link
Contributor

commented Oct 25, 2017

FWIW, I'm trying to use entry points as well, and I am not sure how good support is on conda. The meta.yaml has an entry_points entry, but that assumes they are console_scripts. In an old thread, it is adviced to put in

build:
  preserve_egg_dir: True

@takluyver could you explain what entrypoints does? I am not sure why it is needed next to setuptools/pkg_resources?

@takluyver

This comment has been minimized.

Copy link
Member

commented Oct 25, 2017

I think it should work through conda so long as you use pip to build the package. We don't appear to need any other options when doing that - I've just checked the IPython package from conda-forge, and it has the expected entry points in the right place (both console_scripts and pygments.lexers)

entrypoints is designed to do basically the same thing as pkg_resources, without being part of setuptools. There are some practical benefits - pkg_resources does a load of operations on import, which can make startup slow (can be an extra 5 seconds on a spinning HDD) - but it's also because I've been bitten so many times by setuptools that i will go to some lengths to avoid it.

@blink1073

This comment has been minimized.

Copy link
Member

commented Oct 25, 2017

The jupyterlab recipe also includes entry points but without using pip (by passing extra flags to setup.py).

@maartenbreddels

This comment has been minimized.

Copy link
Contributor

commented Oct 25, 2017

@takluyver thanks, i'll take that up with the conda-forge people to see what their preference is for non-console-entry points. Could you maybe explain the entrypoints goals on the documentation page, because I think it is useful to know, and a useful package :).

@blink1073 I was referring to non-console entry points, or did I miss your point?

@blink1073

This comment has been minimized.

Copy link
Member

commented Oct 25, 2017

Ah, I haven't used those before in conda, but I suspect they should work the same with that invocation.

@blink1073

This comment has been minimized.

Copy link
Member

commented Oct 25, 2017

Yeah it looks like you don't have to specify the non-console entry points in meta.yaml. The nbconvert recipe also only includes the console script, but adds nbconvert.exporters.

In [2]: import pkg_resources

In [3]: for entry_point in pkg_resources.iter_entry_points('nbconvert.exporters'
   ...: ):
   ...:     print(entry_point)
   ...:
rst = nbconvert.exporters:RSTExporter
custom = nbconvert.exporters:TemplateExporter
html = nbconvert.exporters:HTMLExporter
...
@maartenbreddels

This comment has been minimized.

Copy link
Contributor

commented Jan 5, 2018

I think this can be closed due to #3116 right?

@bollwyvl

This comment has been minimized.

Copy link
Contributor

commented Jan 5, 2018

@blink1073

This comment has been minimized.

Copy link
Member

commented Jan 5, 2018

Also, entry_points imply that you must have a Python package, while data_files can be populated by anything (e.g a non-Python conda package).

@blink1073

This comment has been minimized.

Copy link
Member

commented Jan 5, 2018

But I suppose with server extensions you are already a Python package...

@bollwyvl

This comment has been minimized.

Copy link
Contributor

commented Jan 5, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants
You can’t perform that action at this time.