Merge @jupyterlab/debugger into JupyterLab core #75
Comments
|
I support this proposal |
|
I support
…On Wed, Jul 15, 2020 at 10:15 AM Steven Silvester ***@***.***> wrote:
I support this proposal
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#75 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAAGXUHCOBMHOUS5LL3V5PTR3XP35ANCNFSM4O2XB7HQ>
.
--
Brian E. Granger
Principal Technical Program Manager, AWS AI Platform (brgrange@amazon.com)
On Leave - Professor of Physics and Data Science, Cal Poly
@ellisonbg on GitHub
|
|
I can understand the pressure to include this in core, especially since VS Code does now. And I am very thankful to all the work put in from folks to push this feature along, it will obviously have huge benefits to users. However, I wanted to articulate one potential objection. If we merge this in and release a 3.0 blog post announcing this feature, then users of JupyterLab will be presented with a choice:
This will lead, I believe, to some pressure from users to move the community to a place where they can have a default kernel that will both have backwards compatible support and the debugger support. So that means at some point in the future, if we merge this PR, we will either want to:
When we (at Quansight, including @Carreau) explored that question for a client of ours, we tentatively came to the conclusion that we should support bring xeus-python up to ipykernel support, because no one really wanted to keep maintaining ipykernel and adding debugger support would require a large rework. But it required more funding than we had to even explore that question in a comprehensive way. If anyone has already done this exploration, it would be really helpful to share any findings from that to understand how much money and time that would require. So my preference at the moment would be to wait on including debugger support until our default kernel we include has support for it. If we include it, since we are a high visibility project, we will effectively have to announce it. And it's unclear what win including it brings, if we aren't advertising it as a new feature. Of course, it's more troublesome to install a third party extension, but that is true for many other extensions as well which have higher usage than this one. |
|
It's still unclear how much work is needed in ipykernel to get it to the right point, there are also a couple of things in the current jupyter implementation protocol that are incorrect/impossible, for example the control channel message receiving is blocked by the GIL, so it would need to be moved to a thread which has its own implications – xeus does that. It for example affects killing and restarting the kernel and there is hack in the notebook server that kill the process instead of sending a graceful shutdown message. Also AFAICT after talking with Sylvain is that Xeus will not (and cannot) be 100% compatible with IPykernel (WRT configuration & extensions), and is not yet packaged on all platforms because it has to be compiled; So we will likely always have a need for an ipykernel. I've worked with sylvain to make the IPython utils better delineated so that xeux en If core of JupyterLab want to merge support that's your decision, but I do think this may fracture the ecosystem, and that kernels maintainer (me), are stating to get tired of being single maintainer of projects no one want to spend critical time on (IPython, IPykernel, Traitlets). I did work with Sylvain to make IPython more modular and importable from Xeus, but I'm at the limit to the number of support request I can handle. The client @saulshanabrook talked about really want an official way of using a debugger, but they do have a number of extensions/magics, and having their user switch kernels depending on the features supported will not make them happy. There have already been a big fracture on the Python ecosystem with the move to JupyterLab, as people are still using classic notebook, and there is little time spent on it. So I would be mindful. |
|
I am currently maintaining Other kernel authors are working on adding debugger support. I don't think it is fair to say that we are causing a rift by offering a feature that is standard in IDEs. |
|
The way I think about rolling out new features is based on these simple heuristics:
If our own tools are shipping in other products, I submit they should ship with ours as well. |
|
Quick comments on xeus:
|
I still see the magic issue open - jupyter-xeus/xeus-python#63
So, this means it has a dependency on anaconda? Or is pip install also supported? When I looked at the last year Python Developer Survey conda was used by only about 23% of python developers. |
Cool.
No. Why would you think so?
Cool. Pip install is supported on osx and linux. We build windows wheels as well in our CI but as I said, I have not uploaded them yet. |
👍 I agree and I support the proposal. |
|
Does xeus have this critical feature of ipykernel?
Without the ability to launch from an existing python process it doesn't have much utility for those of us with large, custom python deployments. Also the idea of having certain python kernels support certain features and not others seems bad. If you try to install non "big 3" kernels the user experience degrades substantially and it becomes very confusing very quick. |
Not quite. You are free to use ipykernel if xeus-python does not fit your need. Note that this issue is about incorporating the debugger front-end in lab. The debugging protocol was devised in a language agnostic fashion and other kernel authors are already looking at supporting it in their implementation. |
|
Would it be possible to leave it out of the assets so as to not bloat the build with something we cannot use? If it's not going to work for most cases it should probably stay separate. |
You can certainly use the debugger if you install a kernel that support debugging. |
|
But if i don't install such a kernel because it doesn't work due to missing features, I'd like to not have the debugger assets included in my build. Basically if it's included in the main build, it should work with the standard environment. If it requires extra things or customization, it should probably stay separate (I think resource utilization breaks this since nbresuse is required). |
|
I think that the inclusion of the front-end in the main bundle will
FYI we are currently also exploring the support of debugging in ipykernel. Xeus was primarily developed to make other kernels (R, C++, SQLite, and others), and going with xeus simplified things, but we are still very invested in the ipykernel project. |
|
I just think more divergent kernels is a bad thing. Having to maintain a matrix of which python kernels support which features is going to lead to building feature-specific kernels whether we like it or not. "If you want to use this new shiny feature, install this other kernel!" One kernel for the debugger, one kernel for matplotlib magics, maybe another kernel for RTC, another kernel for the next feature. I think everyone wants the debugger, it's just the extra externalities of building kernel-specific features into the mainline that could be problematic. |
As a question: completely hidden of grayed-out ?
If it gets merged I think it should be grayed out; so that users can at least discover that there might be debugger option. The assets are likely small, and there are many other options people are not using in JupyterLab. You can still probably rebuild w/o said assets. I don't like having non usable visible options by default, but I think hidden one are worse.
It's not a rift because of the debugger UI itself, it's a rift as I'm afraid you'll end up with two slightly incompatible types of Python notebooks, and the complexity of understanding that you need to get an extra kernel and share notebook that won't really work. Heck even GvR was confused as to why Jupyter need kernels in a recent discussion and why it had to be installed separately and registered to work (PR about binder to test Python 3.10 Match expression). It ends up badly reflecting on Jupyter because it does not work "out of the box", adn leads to more support request. One option that was not mentioned is to also make xeus-python a dependency of Jupyter metapackage, and have it installed for (most) users by default. It's ways simpler to tell user "just switch kernel" than go to this long complex speech about what xeus-python is. That's not the best solution IMHO, but I think that's a better one that only merging the debugger UI, announcing it and letting users potentially struggle to figure out why. A metaphor would be that I'll prefer to build either a full bridge or no bridge at at all, but definitely not 1/2 a bridge. |
Better, but it would still be confusing xpython vs python. Not sure that solves the ux problem sufficiently |
|
We have not yet built the behavior of the debugger front-end when it exists in an environment with no debugger-capable kernels. The behavior we were planning to build was to hide it in order to prevent users from being frustrated by features they cannot use. That seemed like the UX equivalent of: "first, do no harm". But this user experience is certainly something we can discuss and iterate on. |
|
At the moment, installing the debugger extension requires rebuilding the jupyterlab bundle which is where we lose many users, having it by default in the bundle will significantly lower the bar for getting debugging support in jupyterlab (just installing a pip or conda package). We are currently working with a potension funder to implement the Jupyter debug protocol in ipykernel, and are pretty close to getting it too. Having the front-end already installed will also simplify working with that improved kernel. The absence of a debugger is one of the main reason why people move to other tools like VSCode, and that is why I think it is important enough to be in the main jupyterlab bundle. |
|
My vote is to:
|
|
Is there a spec documentation on how the debugging protocol behaves (requirements, channels, messages....)? This is needed for others to migrate their kernels. Also, is there a description of the features that IPykernel is missing to support the debugging protocol? |
|
Thx @Carreau. I would say if the debugger goes into core, a |
|
Regarding the integration in ipykernel, we propose to do it in two stages:
Even if we do both (1) an (2) in one go or ipykernel, I think we will want to improve the UI for kernels that only support (1). For example, by having a visual indication that a breakpoint was input, but is not yet added on the kernel side. That will help with kernels providing a naive implementation, and with high-latency scenarios. Note that we are already making strides on having common code between xeus-python and ipykernel (input preprocessors, display mechanism, magics). |
|
IMHO the kernel icon should reflect the language more than the implementation. As as user I just care what language I am running. Right now there is an "optional" feature that is debugging, but that is because it is novel. It will soon be standard, and then as I user I shouldn't need to care "which" Python kernel I am using in the UI, unless I ask for the "about" information. |
I am happy to change the xeus-python kernel icon to be Python. Using the language icon is what we did for xeus-sqlite (SQLite), xeus-cling (C++). It totally makes sense. |
This has really not been our experience. We work on many C++ codebases and recieve numerous high-quality contributions from the community. The ipykernel/IPython combo is a very complex Python codebase (which has this baggage for good historical reasons), and xeus is simpler in my opinion. |
That was originally put in Jupyter Notebook because a lot of the logic does not support having no kernels available.
Well, a lot of other things in the kernel depend on which implementation. Xeus support most IPython magics, but IPython has many features that are used by people at the language level: Autocall (no parenthesis for function call), autoawait, Those are non-python syntax, after there are also kernel specific behavior like how to configure and extend. If you don't know which type of kernel you are running, or just put them under "Python" then you get the bug reports or complaint "The Python kernel does not work". So I think there should be some indication that a kernel is IPython/Xeus, and not just Python. It should of course be relatively clear the language is Python of course, but in the presence of multiple implementation with different behavior, people should know which one they are using. |
|
I am +1 for bringing the debugger in at 3.0 I am happy to work with the debugger team on the UX of how it should act when |
|
I am concerned as an end user and someone who has to support Jupyter for a larger number of people what it will be like to have two Python kernels installed by default. Adding the This I hope gets to the heart of what I believe to be the problem with this proposal. This thread at times seems to be conflating two related issues:
I think everyone agrees that if we have a kernel that support the debugger preinstalled, the debugger is a much needed project and adds a lot of value. It's a huge missing feature for lab. The problem is there is only one kernel installed today and it forces us to ask should |
|
My take is that we should not require the xeus-python kernel by default (and probably not any kernel by default but that is another discussion). Adding the debugger front-end to lab is not consequential for kernels that don't support debugging yet, and will not change their experience. It will lower the bar to the debugger experience. (In the mean time, VSCode already documents how to debug with xeus-python). If D.E. Shaw really needs debugging support in ipykernel, they can wait for us to do that work before offering it to their internal users, or even fund us to develop the debugger support in ipykernel and get it quickly. We have a very clear idea of what is required to get there, as we already implemented the backend in xeus-python, and co-developed the front-end. |
|
Note also that it is possible to disable a core extension using page_config, something we do at Amazon for a couple of the core extensions that we provide alternate implementations for. |
|
Merging into core would also increase the visibility of the project and attract contributors who would like to improve the debugger frontend too.
The blog post and the JEP posted in the previous comments are indeed good resources for this. There is also this issue in the By merging the debugger extension into core, we can then add the Debug Protocol documentation to the main JupyterLab documentation. This will increase visibility and make it easier to find. The documentation can also serve as the reference point for kernel authors who want to add support for debugging to their kernels. |
Thx @jtpio to complement the information given by @Carreau. One of the important point IMHO is to consolidate those various sources of information into a proper doc. As you suggest, a first step would be to host the Once enough polished, a second step would be to move that to a pure kernel repo. The first candidate I see would be to move it to I am not sure if we need to doc before a potential merge or if is is fine to leave that for after a merge. I tend to prefer before if possible. |
Absolutely, the plan is to add the new messages to the messaging spec documentation. |
|
Having an entrypoint in the JupyterLab documentation would be useful since this is where the UI is. Then yes it makes sense to keep the lower-level details in a repo such as |
|
I think there are still some question to resolve as to how/how much we advertise that, but I think we can move on with getting the UI code in. It can still be refined later on anyway if needs be. |
|
I appreciate being able to have the discussion here around this! It seems like we are mostly in consensus that merging in now is best, so I am in support of that. The next longer term step after that would be figuring out how to make the kernel we ship by default works with the debugger, which it sounds like we have multiple paths to achieve that we can work towards. |
|
If you put it in the default UI, will it entice kernel maintainers to adopt the debugging functionality? Otherwise, you could just indicate that the kernel does not support the debugger. |
|
We can also give a spot in the new Settings UI (if it gets into 3.0) with a spot to enable/disable the debugger with a nice warning that it requires a supported kernel. |
|
Because it's possible for the debugger to detect whether any kernels support debugging without any user action necessary, I would suggest we take advantage of that to provide the best automatic experience possible. |
|
From a UX perspective, it seems like a good idea to allow debugging for kernels that support debugging, so it's a yes vote from me. I am also happy to help work out the experience and UI for this, particularly the experience for what the debugger does when no kernels that support debugging are installed since that seems to be one of the main concerns. |
|
When I think as a When I put myself as a user, seeing a widely requested feature such as debugging capability available in JupyterLab, would tease me to use it, but I would quickly realize I can't debug my existing notebooks as is. IMHO, I still think that this should be enabled as part of the widely adopted/used kernels such as IPython kernel. A compromise being discussed here involves remediation steps such as: clear documentation about how the debugger can be used, and the limitations around the kernel(s) that support the debugging protocol and also that, by default, this would be hidden from the user that does not have proper requirements to run the debugger. Based on the compromises described above, I am +1 on merging the debugger into JupyterLab core. |
Also, just a clarification on the topic of debugging notebooks with VSCode, my understanding is that it requires VSCode Insiders + the above github repository extension in order to enable the debugging UI with Xeus code, but when I followed the documented steps with the latest VSCode Insiders and Xeus release, the UX was ok, but actually debugging was causing multiple exceptions. |
|
Note that the latest release of It can be tested on Binder: https://mybinder.org/v2/gist/jtpio/27533014e25ca65c85d0bafff96f38b9/master?urlpath=%2Flab (from this gist: https://gist.github.com/jtpio/27533014e25ca65c85d0bafff96f38b9) Then install
|
|
Thanks for the screencast, Jeremy! |
|
I support merging the debugger into core, especially with the debugger JEP basically being approved at this point, and the debugging protocol becoming a Jupyter standard. |
retracting this since |
Yes, the xpython_extension makes xeus-python somewhat closer to ipykernel, but we aren't 100% sure yet if this should be the default way to launch it - but we have been iterating on streamlining that usecase, for the purpose of the PyPI wheel. |
The JEP has been merged, and the Jupyter Debugger Prototol is now described in https://jupyter-client.readthedocs.io/en/latest/messaging.html. |
|
Closed in jupyterlab/jupyterlab#8747 |
We propose merging
@jupyterlab/debuggerinto JupyterLab core as a 3.0 feature that only appears for users who have a kernel that supports debugging. Because the end-user experience is transparent, this feature should not cause disruption.As of July 2020, the only kernel that supports the Debug Adapter Protocol is the
xeus-pythonkernel, but we are eager for other kernels to support debugging. The front-end debugger is kernel-agnostic and will only show up for users who have installed at least one kernel that supports DAP.cf. VSCode support for debugging Jupyter notebooks:
xeus-pythonexampleThe text was updated successfully, but these errors were encountered: