Spike: Investigate common ground between I.W. and Python REPL

[anthonykim1]

This document intends to capture information that pertains to or intends to distinguish between I.W. and upcoming Python native REPL. For information that is focused more solely or in a broader spectrum towards native Python REPL can be tracked here: #22674

(More to be added) Conducting spike into learning more about the current state of Interactive Window that uses iPython kernel, including target users, complains about I.W, and how I.W and notebook components and attributes can be applied towards building VS Code native Python REPL experience.

This spike plans to display and document some of the finding that represent current state of Interactive Window, Notebook and plans in moving forward for move I.W. and native Python REPL experience.

---

We should also note of the positive and mutually inclusive components/design/feature we would want to potentially apply in designing and implementing VS Code Native REPL.

---

Components/design/features that may be present in notebook, interactive window, but rather should be avoided for native VS Code REPL experience should also be listed here.

• If in a case that the cell like structure is used to represent VS Code native REPL, we should not allow users to execute different cell components back and forth in a somewhat randomized way that is allowed in Jupyter notebook and Interactive Window without keeping the sole, uninterrupted history of execution order that is visible to users. We want to make it explicitly clear which exact order user has executed their commands, and index number per each cell is not sufficient to provide the full history of execution order.
• If in a case that the cell like structure is used, user should be able to clearly see which order they have executed their commands. On a instance where user wish to re-execute their previous command that they have previously executed in previous cell like structure, we may allow them to copy or re-execute in the promise of placing the re-execution to the very latest location of the REPL instance.

---

Questions that needs more research:

• If we decide to go with more of horizontal structure that still uses cell-like components that basically sends commands from the left side to the right, I'm not sure if notebook editor or notebook API can be used since it is on top of builtin list/tree view renderer

• How does I.W. get IntelliSense, syntax highlighting? Is it via Pylance or other LSP.
• Presence and extent of debugging: What does debugging look like for I.W.? How should debugging look like in the upcoming native REPL?
• Since VS Code Native REPL won't be using iPython Kernel, what would it take to replace that with? Is it a custom interactive interpreter (custom REPL)? If so what should we keep in mind in building this other than the storage of command history?
• How generalized is Interactive Window and Notebook to outside of Python. How applicable is it for other languages?
• If in a case of using cell-like structure, how do I keep the state of each individual cell component to share same instance of my customized interactive interpreter?
• Do I need to build my own "kernel" if I use notebook structure/API?
• Explore a ways to embed variable view in the current interactive window vertical structure.

---

Actionable items going forward (more to be added):

• Build custom Python REPL (interactive interpreter)
• Build node.js application to that hooks up to custom interactive interpreter that can send message back and forth.
• Once I have package to launch/communicate with Python REPL then I would hook it up to VS Code API.
• Message would mean the Python command users are trying to execute, and also the result of those. This would be ideal to see if correct communication is happening between the TypeScript side and the Python side later when the code from the above node js project gets embedded inside Python extension repository; And also to be part of/connect to notebook controller, so then the custom Python REPL/interactive interpreter can be used as a kernel, and not have to require iPython kernel which were a overhead/complaints in users staying away from the current interactive window experience.

[amunger]

From your mockup screenshot, I was initially thinking of this suggestion
To clarify what you were going with on that editor layout -
Are those two seperate tab groups? Is this in the editor area or the panel?
Is the left side one big input?
Would there still be the option to execute code from a .py file?

[anthonykim1]

It is sort of like a diff editor view. If we go with horizontal like view, it should be configurable into a verticle view if user also wants to view their content from Python file on side side of the VS Code, from feedback from Python standup today.

The link is surprisingly similar to my updated mockup as I was thinking more of:

after the screenshot above.

I was thinking left side should be chunked up and match with the output boxes on the rightside on 1:1 ratio for better readability.

Yes, there would be way to trigger from .py.

[anthonykim1]

Attaching meeting notes from March 7th, 2024 (feel free to add more):

• Approach we may take to build the REPL: Define target user, list out features the native REPL should have, and try to design the UI based on those features whether that means replacing/removing/redesigning interactive window.
• At this moment, we are trying to understand how far we can go with the existing infrastructure we have. Either its notebook API, controller API, or interactive window.
• VS Code Native REPL makes sense: thinking about interactive window possibly being the subset of what the Native REPL would be?
• Native REPL can have superset of features and knobs that could be modified by whoever the language provider is, or whichever extension is using it. Interactive Window may go away when REPL becomes superset of interactive window.
• Limit test how far we can go with existing Notebook API, interactive window would be better start than re-inventing the wheel right off the bat with defining new editor type.
• UI for this new editor can come later: Current stage is more so learning which features should be included in language experience no matter the language provider, what's possible with interactive window. UI can come after/out of that.
• Mock-up (see horizontal split view image in this issue) looked familiar: While building notebook, users had same requests of how can we have outputs on the right side, other than top down. Didn't go that path since small fraction of users tend to prefer this. (May be useful to conduct user studies here)
• Additional user study may be helpful later down the road when we are setting the default UI for REPL experience.
• REPL seems to be the right concept to have in VS Code. The difference between REPL and interactive window is that we are talking to different compute. Interactive window talks to Jupyter kernel, which can generate both text output and rich output. iPython generate all text.
• If we build new editor type and it take long time, we still need to make sure Pylance can understand new editor and give feedback/functionality. This is something we have already for interactive, hence limit test and see how far we can go with existing interactive window API, and then see what limitation, complains we still have with the UI/UX.
• We may still use different UI, but still use notebook as the "backing" document and notebook controllers as the "compute",
• First good step would be linking notebook, interactive with "normal" customized Python REPL to see how everything works.
• Current workflow of interactive window: When you try to open interactive window, you create notebook and input box. Then we try to find if we have notebook kernels or notebook controllers for it, Jupyter kernel can be a controller for interactive window. What I (Anthony) needs to do is to create notebook controller, but behind that is not a Jupyter kernel, it is Python, regular Python interpreter. This way, we can first give something people can play around first and also get the benefit that interactive window, notebook currently has (ruff support? Pylance understanding the editor type).
• Emphasis on making sure history is READONLY and whatever we try to re-run is inserted to bottom is important for nature of REPL. Important that command history is immutable. Once something is run, they should not be able to change that. They can pull down something if they want to re-run cell in place, but adding to the bottom of the list sequence is crucial.
• User's don't want to use interactive window because it requires the overhead of the Jupyter extension, iPython kernel. (From user studies). Creating notebook controller with custom Python REPL aka. interactive interpreter will tackle this overhead problem. Source would be from the Python extension, instead of Jupyter kernel.
• Controller requires viewtype, notebook type. Only two builtin viewtype: jupyter notebook, interactive window?????
• It may be confusing to users that you can use Python REPL or Python interpreter as a Jupyter.
• For interactive window, if we want to save to normal Python file, what is the separation between each execution? TODO: Look more into #%%
• Variable view) New controller created from the Python side to run with notebook. If we attach variable provider to that controller, then core can reach out to the variable provider and ask for the variables. Then that will fill out the variable view. Very possible to extend this.
• Later down the road, possibly building new API and domain model for REPL. We may be still able to use notebook and leave the execution/compute to notebook controller on the back side.
• What does it mean for core to be in the Notebook: When you see execution history in REPL, we always need a document model for it. That could just be a notebook concept we have in VS Code API. Still fitting into notebook model. It won't use multiple output and multiple items for single output. It won't use everything available to notebook model, but still would fit well since it's a history of commands that have been run and output.
• Notebook controller API: gets input, returns output. Output have different run type, in Python's case, it will always be text for now.
• How would communication occur: node.js project that spawns Python process => send code input in notebook and send them over to process. Listen to stdout on that subprocess? Whatever you get from stdout, construct notebook output and send that to vs code. WARNING: STDIN and STDOUT to communicate with Python not recommended. Communication over stdin, stdout should not be the baseline. Reason: There are some packages that plug into stdout for logging and they would instantly crash your server if you are reading only stdin and stdout. This is the reason why we migrated language server to use named pipe and also migrating testing platform we have to use named pipe.
• Essentially have to build a REPL server: Listens to user input, spits out whatever it ran, maintain the state. Each session is unique session. Nightmare scenario: Two REPL window talking to the same session. Need unique isolated session for each one of them. It's a server that completely isolate the session however you want to deal with it.

[amunger]

It may be confusing to users that you can use Python REPL or Python interpreter as a Jupyter.

Should be "as a controller/kernel". To expand on this, the current categories that the jupyter extension provides are these:

Jupyter allows users to pick an interpreter to be used as a controller (in the python environments category) and we ensure that the required packages are installed - ipykernel/IPython.

If the REPL also uses that same kernel picker (rather than perhaps just forcing the user to use the active interpreter), it would likely provide the same list of interpreters that are in Jupyter's python environments category, but selecting them would be a different experience.

But I think the Python extension will have to select which controller to use when opening the IW anyway, so the user won't be in this situation unless they want to change it.

[anthonykim1]

MVP
intellisense - aaron
execute (enter/shift+enter/ctrl+enter) - aaron
add to python extension behind experiment - anthony
Jupyter's IW shouldn't break - aaron
handle interpreter not selected gracefully - anthony
interrupt - anthony
support input() - anthony
repl file association - anthony???
------------------------ Further Improvements ----------------------

re-add cell if executed in history - aaron
reduce padding in cells microsoft/vscode#175295 - aaron
variable provider - anthony
restart - anthony
IW/"repl notebook" with pure notebook model - aaron
Native IW API - aaron
Multiple REPLs - anthony
smart send - anthony
using core component for text output - aaron
recovering an active REPL on reload - anthony

[anthonykim1]

Adding issue that contains items needed for MVP: #23332

[anthonykim1]

Discussed May 9th:
We may need additional notebook API that resembles pre-execute handler of some kind.

If Python extension were to strive for "enter" to be the default execution mechanism for user trying to execute their Python command using IW UI (with REPL controller contributed from Python extension), we need to be able to get user's Python command inside user's text input box inside the IW UI BEFORE execute handler.
This is because cell output UI seem to already be attached by the time execute handler gets called, but we would need user's input before in order to figure out whether they have typed complete command or incomplete command and hence need to wait for more input.