Auto determine and configure the environment

[MarcSkovMadsen]

If you go to https://github.com/holoviz/panel you can see that Panel can be used in many different environments: Notebook, Lab, VS Code, Colab, Panel Server, Voila Server etc.

I am switching back and forth between most of these. Especially Jupyter Labs, Notebook running in VS Code, Panel Server and now potentially Voila Server. C.f. #2241

Right now I have to add/ remove pn.extension(comms='vscode'), pn.extension(comms='colab'), pn.extension(comms='ipywidgets') depending on the environment.

It is not efficient that I have to change/ update the code depending on the development or deployment environment. Furthermore it can confuse new users and lead to support questions on how to use Panel in XYZ environment. Or just users leaving because they don't think it works in their favorite environment. Furthermore it makes it complicated to develop example notebooks for a community of users with different preferences for using those example notebooks. For example for Panel or for work.

Solution

Find a way to auto detect the environment and automatically do the configuration.

[MarcSkovMadsen]

I've asked the VS-Code Python developers if there is a way to determine if I am running inside VS Code Interactive Notebook.

microsoft/vscode-python#16034

[philippjfr]

We have asked Jupyter again and again whether there is some way to determine the frontend and the answer each time was that the backend should stay completely agnostic as to the frontend and therefore no support for identifying the environment would ever be provided. So while I agree this would be nice I'm strongly inclined to close this issue since it is out of our hands and anything we do implement will be hacky and/or very brittle.

[jbednar]

What if we provided pn.extension('auto'), and explicitly documented that 'auto' may be useful but it is hacky and completely unsupported and we recommend specifying a specific extension instead?

[MarcSkovMadsen]

Thanks @philippjfr . I feel your frustration.

If ever they (Jupyter, VS Code, Colab etc) should provide something. What should it be? An imported module?

[MarcSkovMadsen]

The goal is not to need to specify pn.extension() as it is not needed on the server unless we specifically need to import an extension. Alternatively to always specify the same pn.extension(...). The code needs to be the same across environments and no matter how it is served. Otherwise the promise of Panel of going from exploration to production with the same code base (especially notebook) does not hold in practice.

[philippjfr]

What if we provided pn.extension('auto'), and explicitly documented that 'auto' may be useful but it is hacky and completely unsupported and we recommend specifying a specific extension instead?

That supposes we have a mechanism (hacky or not) to detect the frontend, which we don't.

The goal is not to need to specify pn.extension() as it is not needed on the server unless we specifically need to import an extension.

Panel itself (and Bokeh) is the extension that has to be loaded in the notebook, so there's no way around it and since running the extension on the server isn't harmful it should generally be included.

[jbednar]

Using the same code is already possible between server and notebook contexts; pn.extension('bokeh') should work in both cases. It's all the other contexts ('ipywidgets', 'vscode', etc.) that currently have to be specified explicitly.

In any case, the code to install the javascript that's required in some environments has to be invoked somehow, and so if not through a call to pn.extension() or some similar method, it would have to be done automatically on import. hvPlot is an example of using a module in this way, e.g. import hvplot.pandas implicitly calls the equivalent of panel.extension('bokeh'). But even in hvPlot, importing hvplot alone doesn't do this, because otherwise there would be no way to avoid executing this sometimes brittle code, and I think the same would be true of Panel. I.e., even if auto-detection code were available, you'd still need to do a special import like import panel.auto to invoke that code, so that the library itself would behave politely and not try to mess with stuff as this auto code would do.

So such a request comes down to (1) accepting that invoking the auto-detection would still need to be done explicitly, and (2) actually implementing the autodetection. If you're ok with (1), seems like it would be ok to try to achieve (2), though it is not at all straightforward (by design of Jupyter).

[philippjfr]

To be clear pn.extension('bokeh') is not something that exists, you're thinking of hv.extension where the arguments load the plotting backend. For pn.extension the arguments load optional libraries like deck.gl, MathJax, echarts, vega etc.

[jbednar]

Good point; I was confusing about that. But here we are not talking about deck/echart/vega/mathjax support, but about environments: vscode, collab, jupyter, jupyterlab, bokeh server, voila.

[MarcSkovMadsen]

For VS Code running locally on a laptop VSCODE_PID is injected into the kernel. My estimate is that it would catch +90% of VS Code use cases. For more info see this discussion microsoft/vscode-python#16034 (comment)

Would you accept a PR that sets comms='vscode' if VSCODE_PID is set and no comms is provided by the user @philippjfr ?

[MarcSkovMadsen]

And for Google Colab something similar could probably be done. For example based on whether COLAB_GPU is set or not?

[MarcSkovMadsen]

I believe the only one outstanding is colab. Would be really great to get supported such that Sophia, I and other blogstars 😄 can communicate easily.

We would like to get rid of the explanations

and support on discourse when users ask about this.

[philippjfr]

We already auto-determine colab here: https://github.com/holoviz/panel/blob/master/panel/config.py#L556

[jbednar]

Sounds like the docs need updating?

[MarcSkovMadsen]

https://github.com/holoviz/panel#google-colab

[jbednar]

I'm pretty sure that I wrote that and then Philipp made it obsolete. PRs to update the docs strongly encouraged!

[philippjfr]

I think we now auto-detect all common environments and docs have been updated.