Confusing behavior of hv.extension()

[ceball]

(in an env with matplotlib)

>>> holoviews.extension('no way does this exist')
>>>

(in an env without matplotlib)

>>> holoviews.extension('no way does this exist')
Traceback (most recent call last):
  File "/Users/cball/code/ioam/holoviews4/holoviews/util/__init__.py", line 228, in __call__
    __import__(backend)
ModuleNotFoundError: No module named 'matplotlib'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/cball/Eunectes/mc3/envs/hvdev/lib/python3.6/site-packages/param/parameterized.py", line 1891, in __new__
    return inst.__call__(*args,**params)
  File "/Users/cball/code/ioam/holoviews4/holoviews/ipython/__init__.py", line 106, in __call__
    super(notebook_extension, self).__call__(*args, **params)
  File "/Users/cball/code/ioam/holoviews4/holoviews/util/__init__.py", line 231, in __call__
    "to activate %s extension." % (backend, backend))
TypeError: not enough arguments for format string

[jbednar]

I agree that's confusing. PR?

[ceball]

I made a branch to fix the obvious warning message error, but unfortunately the function confused me more than the error message, so I stopped there. I don't understand the intent...maybe it has changed over time?

[ceball]

class extension(param.ParameterizedFunction):
    """
    Helper utility used to load holoviews extensions. These can be
    plotting extensions, element extensions or anything else that can be
    registered to work with HoloViews.
    """

But when I squint at the code in there, it seems to be about plotting only, maybe?

[ceball]

If someone confirms the docstring is right, or supplies an updated docstring, I'll make a PR for the function itself...

[jbednar]

Yes, it has definitely evolved over time, changing names once or twice and changing how it works. Long ago, it was called notebook_extension(), and it was only for installing ipython-notebook support. When we added multiple plotting backends (now more accurately considered frontends), we decided that we should just have one extension loading mechanism, and shortened the name. So the intent is for it to cover any extensions, even though currently what it handles is bokeh, matplotlib, plotly, and ipython (I think?).

[ceball]

To me it seems like it only handles plotting (judging from it defaulting to loading matplotlib, and attempting to import holoviews.plotting.x where x can be mpl, bokeh, plotly).

    _backends = {'matplotlib': 'mpl',
                 'bokeh': 'bokeh',
                 'plotly': 'plotly'}

    def __call__(self, *args, **params):
        # Get requested backends
        config = params.pop('config', {})
        util.config.set_param(**config)
        imports = [(arg, self._backends[arg]) for arg in args
                   if arg in self._backends]
        for p, val in sorted(params.items()):
            if p in self._backends:
                imports.append((p, self._backends[p]))
        if not imports:
            args = ['matplotlib']
            imports = [('matplotlib', 'mpl')]

        args = list(args)
        selected_backend = None
        for backend, imp in imports:
            try:
                __import__(backend)
            except:
                self.warning("%s could not be imported; %s must be importable "
                             "to activate the %s extension." % (backend, backend, backend))
            try:
                __import__('holoviews.plotting.%s' % imp)
            [...]

[jbednar]

Right, but there's another level beyond that, due to some monkeypatching in holoviews/__init__.py:

try:
    import IPython                 # noqa (API import)
    from .ipython import notebook_extension
    extension = notebook_extension # noqa (name remapping)
except ImportError as e:
    class notebook_extension(param.ParameterizedFunction):
        def __call__(self, *args, **opts): # noqa (dummy signature)
            raise Exception("IPython notebook not available: use hv.extension instead.")

That way user code is the same whether in or out of the notebook environment, but it does make this code horrible to reason about, and it also makes error detection very awkward.

[jbednar]

To me it seems like this code could be inverted to have one, normal function "extension()" with no monkeypatching, just a try/except import that runs some code in the try, and then runs other stuff in the except.

[GeoVizNow]

Not sure if this could be related, but I experience a very annoying thing.
When using holoviews in a venv it seems that proper autocomplete is disabled after loading the hv.extension('bokeh') or after running hvplot.pandas (that also loads the extension I guess)
This image shows that np.random.TAB (pressing 'tab') will list available options for tab auto completion. Note that here hv.extension('bokeh') is commented out.

When hv.extension('bokeh') is not commented then
for numpy np.TAB will work for autocompletion, but only in the first level. np.random.TAB does not show any options.
I might have some lacking dependencies in the jupyter install, but have tried a few times with same result all the time.
my requirements file content below:

holoviews==1.15.4
shapely
bokeh==2.4.3
panel>=0.14.1
pandas==1.5.2
hvplot==0.8.2
datashader
jupyter
jupyter-core
jupyter-client
nbconvert
xarray
plotly
pyarrow
# tslearn
# h5py
# geoviews  # no, now wheel for cartopy
pyproj
shapely
spatialpandas>=0.4.4
pooch  # for scipy datasets
jedi
pyreadline
pyparsing

The auto-completion also fails for other libraries, e.g. panel in the level of hierarchy after pn.pane.TAB.

[Hoxbro]

The problem is that hv.extension disables Jedi competition by default. Try running: hv.extension("bokeh", allow_jedi_completion=True). The reason for this is given here:

holoviews/holoviews/ipython/__init__.py

Lines 102 to 106 in 04ba54a

	allow_jedi_completion = param.Boolean(default=False, doc="""
	Whether to allow jedi tab-completion to be enabled in IPython.
	Disabled by default because many HoloViews features rely on
	tab-completion machinery not supported when using jedi.""")

@jlstevens do you have any objections in making Jedi completion true by default?

[jlstevens]

No objections as long as:

1. It works
2. The behavior is now more sensible than it used to be

I'm happy to defer to your judgement on this but I do remember jedi completion did not work as expected and broke a lot of tab-completion suggestions we had put effort into generating. That was a long time ago though and it is entirely possible that now it is fixed and works well.

[GeoVizNow]

Thanks for this useful trick and the background information. I actually swapped into using jupyterlab because there tab auto-completion worked well. However, after now going back to check in ordinary jupyter notebook it magically seems to work there too with no further modifications. I notice however, that when starting jupyter notebook from command line using 'jupyter notebook' that the terminal is reporting 'Loading JupyterLab as a classic notebook (v6) extension.' so I wonder if somehow using the jupyterlab in classic notebook mode (as opposed to using jupyter with no jupyterlab available) is making a difference.

[jlstevens]

The way jedi sorts its completions does still annoy me and multi-level completions used the work fine without jedi but this looks it is no longer the case.

Given this situation, I think we should allow jedi to be on by default despite my distaste for it.