Explore proxy controller in notebooks #146942
Comments
|
I attempted to register an async notebook controller to represent lazy controller / controller resolver and quite a few UX and technical questions occurred to me which I might or might not have answers for yet, some notes for myself:
|
Initial connection: select and runBased on exploration/mockups @misolori put together, here is a prototype of how the kernel resolver would work with integration with Jupyter
In the recording:
To archive above, we would need following APIs
export interface NotebookProxyController {
readonly id: string;
readonly notebookType: string;
label: string;
kind?: string;
resolveHandler: () => NotebookController | string | Thenable<NotebookController | string>;
readonly onDidChangeSelectedNotebooks: Event<{ readonly notebook: NotebookDocument; readonly selected: boolean }>;
dispose(): void;
}
export namespace notebooks {
export function createNotebookProxyController(id: string, notebookType: string, label: string, resolveHandler: () => NotebookController | string | Thenable<NotebookController | string>): NotebookProxyController;
}
export interface IJupyterServerUri {
baseUrl: string;
token: string;
authorizationHeader: any; // JSON object for authorization header.
expiration?: Date; // Date/time when header expires and should be refreshed.
displayName: string;
}
export type JupyterServerUriHandle = string;
export interface IJupyterUriProvider {
readonly id: string; // Should be a unique string (like a guid)
onDidChangeHandles: Event<void>;
getHandles(): Promise<JupyterServerUriHandle[]>;
getServerUri(handle: JupyterServerUriHandle): Promise<IJupyterServerUri>;
}
export interface IExtensionApi {
registerRemoteServerProvider(serverProvider: IJupyterUriProvider): void;
addRemoteJupyterServer(providerId: string, handle: JupyterServerUriHandle): Promise<void>;
}Lazy kernel resolver extension is responsible for setting up remote Jupyter servers and contributing to Jupyter (through Todos/notes
Reload and reconnectWorkspace reload/reconnect is more complicated than the initial connection. Let's say the first time users open notebook, the kernel quick pick has two options After users pick The
All 3 options are valid and the 3rd one meets users' expectation the most, but it has high chances to fail as there is no guarantee the same kernel users use last time is still available (like the server is released). As we discussed offline, we would want to conditionally show option 1 and 3, it would require the github extension and the Jupyter extension coordinate with each other
|
|
Extension snippet code for contributing a lazy kernel which can launch a local jupyter server. The code demonstrates how the lazy kernel integrates with VS Code and Jupyter extension:
interface ICachedServer {
serverProcess: any;
baseUrl: string;
token: string;
}
function registerLazyKernels(context: IExtensionContext, api: IExtensionApi) {
const servers = new Map<String, ICachedServer>();
api.registerRemoteServerProvider({
get id() {
return 'github';
},
getQuickPickEntryItems: () => {
return [] as QuickPickItem[];
},
handleQuickPick: (_item, _back) => {
return Promise.resolve(undefined);
},
getServerUri: (handle: string) => {
// github to do the auth
const server = servers.get(handle);
if (server) {
// resume the machine/vm
const token = server.token;
return Promise.resolve({
authorizationHeader: { Authorization: `token ${token}` },
displayName: 'GitHub',
baseUrl: server.baseUrl,
token: token
});
}
return Promise.reject();
}
});
const createServer = () => {
return new Promise<ICachedServer>(resolve => {
const server = spawn('C:\\Users\\rebor\\.conda\\envs\\testenv\\python.exe', ['-m', 'jupyter', 'lab', '--no-browser']);
const handleData = (data: string) => {
const buffer = data.split(/\r|\n|\r\n/g) as string[];
for (let i = 0; i < buffer.length; i++) {
const matches = /(http(s)?:\/\/(.*))\/\?token=(.*)/.exec(buffer[i]);
if (matches) {
const baseUrl = matches[1];
const token = matches[4];
resolve({
serverProcess: server,
baseUrl,
token
});
break;
}
}
}
server.stdout.on('data', (data: any) => {
handleData(data.toString());
});
server.stderr.on('data', (data: any) => {
handleData(data.toString());
});
})
}
const controller = notebooks.createNotebookProxyController('lazy_kernel', 'jupyter-notebook', 'GitHub Server', async () => {
return new Promise(async (resolve, reject) => {
const server = servers.size === 0 ? (await createServer()) : [...servers.values()][0];
servers.set(server.baseUrl, server);
commands.executeCommand(
'jupyter.selectjupyteruri',
false,
{ id: 'github', handle: server.baseUrl },
window.activeNotebookEditor?.document
).then(p => {
if (p) {
resolve(p as NotebookController);
} else {
reject()
}
});
});
});
controller.kind = 'GitHub';
context.subscriptions.push(controller);
} |
|
@kieferrm and I had offline discussions of the two common scenarios: initial connection and reload. The initial connection is simple as the proxy kernel will spin up the jupyter server on a remote machine, provides a jupyter server url to Jupyter extension and lastly a notebook controller is created and used to execute code. The "Reload" scenario is not that straight forward. When the window/page reloads (e.g., install an extension which requires reloading the window), VS Code workbench restores the same notebook editor opened before, we need to figure out if we should allow users to run code against the jupyter server/kernel just used before. In all 3 options described in #146942 (comment), the 3rd one is the most natural one:
The catch here is the cached jupyter server might be deallocated/released. When that happens, Jupyter won't be able to connect to the server anymore and execution will fail. A nicer experience will be the GitHub extension launch a new jupyter server under the hood with the same kernel specs. An alternative is GitHub provides the availability of jupyter servers to Jupyter extension in advance, and Jupyter extension can clear the cache if the jupyter servers are no longer available. We also found that the same concept/UX could be expanded to remote jupyter server support in Jupyter on VS Code desktop:
|
|
|
Be sure to account for this issue. |
We want to explore how we can build extensions which contribute a kernel/controller resolver that when selected provides functional notebook controllers for execution. The resolver handles necessary authentication for users and should be responsible for spinning up the real kernel/servers and work with controller extensions (e.g. Jupyter) to register notebook controllers (or register controllers themselves)
The text was updated successfully, but these errors were encountered: