rfc: Prompt API support

[jkleinsc]

The Prompt API is part of Built-in AI capabilities in Chromium based browsers.

With the Prompt API developers can send natural language requests from the renderer for local processing.

This proposal adds support for this API in Electron.

📄 Read rendered document

[felixrieseberg]

I love the simplicity of directly setting the handler on Session, I think that's really easy to understand.

The main drawback I see is that with this design, the main process is always an intermediary between the utility process and the renderer in which window.ai is involved. We already have similar shortcomings in our network API, so this proposal isn't any worse than equivalent APIs - but the optimal solution would establish a message channel directly between the utility process and the rendered process (via Mojo), thus avoiding having to bother the main process at all.

In @electron/llm, I'm going with something that combines both approaches - exchanging ports on prompt and then streaming additional responses directly to the renderer. I wonder if we can make that easier for devs here, for instance by providing and established port in the streaming API that you can just post to.

[itsananderson]

How do we expect to get a prompt response back out of the utility process? Will it postMessage back to the main process?

[itsananderson]

If we directly pass a construction function, I think it'll make it easier to more flexibly swap out the implementation, and will also make it easier to adapt to changes in the upstream API

Suggested change

	session.defaultSession.setPromptAPIHandler(() => { ElectronAI });
	session.defaultSession.setPromptAPIHandler((webContents, options) => new ElectronAI(webContents, options));

[dsanders11]

Some non-substantive comments, and a note that we should probably make the source.chromium.org links permalinks for historical reference.

[dsanders11]

Are there other imagined use cases at the moment for the WebContentsargument to setPromptAPIHandler, or is the only imagined use case currently to distinguish different WebContents instances to avoid conversation bleed over between them?

Exposing WebContents to utilityProcess would be a rather large side effect of this API, so I wonder if we want to start with the minimal surface area and only provide webContents.id to start?

[MarshallOfSound]

Exposing WebContents to utilityProcess would be a rather large side effect of this API, so I wonder if we want to start with the minimal surface area and only provide webContents.id to start?

It would also expose a bit of a security hole, utility process's shouldn't have arbitrary mojo access to webContents. And building a whole system where we pass handles back and forth to ensure that only the given utility process can call those methods without a strong reason for utility process's to need that access is gonna be a hard sell (at least to me).

Agree with david, would love to avoid exposing any part of the webContents API to Utility. An ID seems good, or if you want to make security decisions passing a securityOrigin

[jkleinsc]

@dsanders11 @MarshallOfSound Good points on passing the webContents. I've adjusted the design to pass a webContentsId instead and I've also added the securityOrigin.

[dsanders11]

I've got some questions about the lifetime of the associated UtilityProcess.

It looks like the Prompt API allows for explicit session destruction, and I see you've got the destroy method in the handler script. Do we want destroy handled at that level, and leave the UtilityProcess alive, or do we also kill the UtilityProcess on destroy?

Do we support multiple LanguageModel.create calls, and/or the clone operation? Does each one get its own UtilityProcess, or is it a single UtilityProcess with multiple ElectronAI instances?

Do we automatically restart the UtilityProcess if it crashes, like network service in Chromium? Or for now should we simply let the process die and throw errors for any LanguageModel associated with the dead process?

[jkleinsc]

@dsanders11 in regards to your questions:

It looks like the Prompt API allows for explicit session destruction, and I see you've got the destroy method in the handler script. Do we want destroy handled at that level, and leave the UtilityProcess alive, or do we also kill the UtilityProcess on destroy?

I think this should be left up to those implementing the API as to whether they want to leave the UtilityProcess running or not.

Do we support multiple LanguageModel.create calls, and/or the clone operation? Does each one get its own UtilityProcess, or is it a single UtilityProcess with multiple ElectronAI instances?

Again, I think this is up to those implementing the API. You could in theory register multiple local AI Handler Scripts/Utility processes or you could use one utility process.

Do we automatically restart the UtilityProcess if it crashes, like network service in Chromium? Or for now should we simply let the process die and throw errors for any LanguageModel associated with the dead process?

This is a good question that I'd like the API WG to weigh in on, but I think we'd want to let the process die and throw errors because if someone wrote a utility process that continually crashes we wouldn't want to repeatedly restart it only to have it die again.

[samuelmaddock]

Maybe we can structure this in a way that allows looking up more details of the sender similar to how preload realms introduced senderType to IPCs. That way the handler can more easily validate the initiator.

rfcs/text/0008-preload-realm.md

Lines 319 to 329 in d89000c

	```diff
	interface IpcMainEvent {
	// Breaking change
	- sender: WebContents;
	+ sender: WebContents | ServiceWorkerMain;
	// Non-breaking change
	+ senderType: 'frame' | 'service-worker';
	+ senderServiceWorker: ServiceWorkerMain;
	}
	```

Could be something like

type HandlerDetails = { senderType: 'frame', webContentsId: number, frameId: number } |
  { senderType: 'service-worker', versionId: number, scopeUrl: string } |
  { senderType: 'process', processId: number }

[samuelmaddock]

Also, should the handler allow rejecting the prompt API request?

[itsananderson]

Will this call the destroy() method on any prompt API handlers, or does that only happen if the prompt instance is destroyed?

Are there any other considerations we should make for how the utility process should handle shutdown, or is that already covered by existing utility process APIs?

[itsananderson]

Would it be possible to make this async? Since we're just getting a webContentsId, it might be necessary to make an IPC call to the Main process to gather other info.

[itsananderson]

We may eventually want to add other parameters in future, so it might be more future-proof if we pass a dictionary to the handler, rather than positional paramters