Respond to ptvsd changes to get debugging cells to work

[rchiodo]

Essentially this is:

1. Create hashes and input counts for each cell
2. Updating cell values as the user edits a file - need to handle adds/removes
3. Rewiring the debug adapter so that launching a cell calls back into the Extension instead of launching python

[DonJayamanne]

Click for details to create/edit diagram

Instructions for editing diagram:

• Go to https://swimlanes.io
• Paste below content

title: Jupyter Debugging

VS Code ->  Extension: Click `Debug Cell` code lens

Extension ->  VS Code: Start Debug Session

TS Debug Adapter <-  VS Code: Send `launch request`

TS Debug Adapter -->  Extension: Send `custom event` with `port`
note:
**Custom Event** This is a fire and forget message requesting the extension (DS) to start the python program (in this case Jupyter code).

TS Debug Adapter ->  TS Debug Adapter:  **Wait for PTVSD to connect**
note:
**Wait for connection** At this point TS Adapter waits for PTVSD to connect to it. When it does, TS Adapter will connect PTVSD with VS Code.

Jupyter <-  Extension: Start `ptvsd` passing in the `port`


group: _**More communication...**_

end

PTVSD ->  TS Debug Adapter: Connection Established
note:
**Connect** PTVSD connects to TS Adapter, and then this connection is propogated to VS Code.  
_Simple process of connecting the streams/pipes_.


TS Debug Adapter ->  VS Code:  Debugger has started

@int19h @karrtikr @rchiodo @IanMatthewHuff /cc

[DonJayamanne]

Currently the PVSC Debugger launches user code (in a python process), this needs to be changed (as discussed) to launch the Jupyter stuff.

Here's an overview of the solution and suggested changes:

• Remember, looking at the diagram, launch request is what's triggering the TS Adapter to start the python program. Hence that's the request that needs to be intercepted and modified to launch the jupyter stuff.
• The launch request is handled in the PythonDebugger.launchRequest
  • All we're doing here is checking whether the program (python file) exists on disc.
  • TODO: This check will have to be conditional based on a setting in the launch request argument (i.e. the json that's basically entered in launch.json. Add some custom stuff that's specific to DS but not available to the user).
  • The method launchPTVSD basically launches the python process with ptvsd inside it.
  • When that finishes, TS Adapter (this process) waits for PTVSD to connect to it.
  • When that's successful, then we fire an event debugger_launched that'll connect PTVSD with VS Code... (no changes here)...

    protected launchRequest(response: DebugProtocol.LaunchResponse, args: LaunchRequestArguments): void {
        const fs = this.serviceContainer.get<IFileSystem>(IFileSystem);
        if ((typeof args.module !== 'string' || args.module.length === 0) && args.program && !fs.fileExistsSync(args.program)) {
            return this.sendErrorResponse(response, { format: `File does not exist. "${args.program}"`, id: 1 }, undefined, undefined, ErrorDestination.User);
        }

        this.launchPTVSD(args)
            .then(() => this.waitForPTVSDToConnect(args))
            .then(() => this.emit('debugger_launched'))
            .catch(ex => {
                const message = this.getUserFriendlyLaunchErrorMessage(args, ex) || 'Debug Error';
                this.sendErrorResponse(response, { format: message, id: 1 }, undefined, undefined, ErrorDestination.User);
            });
    }
    private async launchPTVSD(args: LaunchRequestArguments) {
        const launcher = CreateLaunchDebugClient(args, this, this.supportsRunInTerminalRequest);
        this.debugServer = launcher.CreateDebugServer(this.serviceContainer);
        const serverInfo = await this.debugServer!.Start();
        return launcher.LaunchApplicationToDebug(serverInfo);
    }

Changes:

• Basically all we need to do is, change launcher.LaunchApplicationToDebug
  • That's the method that'll launch the application with PTVSD in it.
• Instead of running the standard python code with ptvsd, we need to launch jupyter.
• Option 1:
  • Note: CreateLaunchDebugClient is a factory function. You can modify that to create a custom DebugClient class.
  • Today we have 2 debug clients, one that starts code without debugging and the other that starts code with debugging. This would be a 3rd, that lets the extension start the code, instead of the debug adapter doing it...
  • Create a new debug client inheriting the existing LocalDebugClient class.
  • This can be generic to the extension. Basically the extension is responsible for launching the debugee (application with ptvsd).
  • Suggestion: Lets not make this about Jupyter or DS, basically a generic name where extension will be responsible for launching the PTVSD process instead of the debug adapter...
  • Override the LaunchApplicationToDebug method of this new class to send the custom event to the extension, as follows:
    • Use this.debugSession.sendEvent(new Event('<Event Name>', {cusotm json object})) to send an event to the extension (this is fire and forget, as its across processes).
    • You can get hold of the host and port from the argument dbgServer.port and dbgServer.host respectively.
  • On the extension side, use the method IDebugService.onDidReceiveDebugSessionCustomEvent to add an event handler to receive this custom event.
    • This event handler is the only code thats data science specific.
    • This way, rest of the code can be generic functionality.
• Option 2:
  • I'm sure there are other ways to skin this cat...

Notes & Warning:

• All code inside debugAdapter folder does NOT have access to VS Code. Its running in a completely separate process Debug Adapter process. Hence the name of the folder.
• All code inside [extension] is debugger specific code that runs in the extension (with full access to VS Code API).
• I believe DS Code specific to debugger can live in the DS folder itself, no need to keep it in here (up to you)
• There's a fixed time the TS Adapter waits for PTVSD to connect to it. If PTVSD doesn't connect within 20 seconds. This time is configurable, and you might want to set it when testing/debugging. All you need to do is add a property timeout in launch.json request arguments with the value you need (time in milli seconds).
  • This happens in waitForPTVSDToConnect
  • If there's a timeout, on VS Code side you'll get a message indicating debugger timeed out, etc... (something we tell VS Code to display).

[greazer]

We are looking for an MVD (Minimum viable demo) by July 8th.