RFC-0029: Perfetto UI: Intelletto - AI Assistant (coordination)

[github-actions[bot]]

📄 RFC Doc: 0029-intelletto.md

---

Perfetto UI: Intelletto - AI Assistant (coordination)

Authors: @stevegolton

Status: Draft

Introduction

Reading a trace well takes deep, domain-specific knowledge of the system being
traces - plus a fair amount of Perfetto UI fluency to get at it - a barrier that
keeps most of Perfetto's power in the hands of a few experts. This doc proposes
an AI assistant, and the LLM framework behind it, that lets anyone investigate a
trace and drive the UI in natural language: ask why a frame was janky, turn a
question into SQL, build a Data Explorer graph, or jump to the right point in
the timeline.

It is the coordination doc for the work involved with integrating this new
tooling into the Perfetto UI: it owns the overall motivation, the component
breakdown, the cross-cutting risks, and the roadmap, and links out to a
dedicated RFC for each component. It does not specify any component itself -
each lives in its own doc (see "What we're actually building" below). For
concrete examples of what the assistant can do, and the case for embedding it in
the UI, see the embedded-assistant doc
(RFC-0032).

Framing

RFC-0025 establishes the
motivation and demand for AI tooling in Perfetto - who it serves, the problems
it addresses, and the evidence behind it. This doc and its children deliberately
take that case as given and cover the implementation in the UI. Review
comments about whether to build this belong on that doc; these are about how.

There is also prior art within the codebase: com.google.PerfettoMcp already
offers similar but simpler chatbot-style functionality, and will likely be
replaced by the tool described in these docs (see the migration subsection
under Roadmap, and RFC-0032).

Risks

Adding potentially controversial features to a well-established tool carries
risk:

• Privacy reputation. AI and LLMs have a poor reputation on privacy - long
  one of Perfetto's strengths - so any integration risks undermining the trust
  we've built there. To put minds at ease up front: Perfetto OSS will always
  remain model-agnostic, with no model or provider baked in - the user is always
  free to bring their own and decide where their data goes (see the last point
  below).
• Useless features degrade the experience. AI bolted onto established
  software without clear value has repeatedly drawn criticism and, in many cases,
  degraded the very experience it set out to improve.
• Hallucinations and the cost of being wrong. LLMs can state falsehoods
  confidently, and a plausible-but-wrong answer can erode confidence in the tool
  as a whole - cutting against Perfetto's long-standing principle of only
  showing what we know to be true. While hallucinations can't be eliminated
  entirely, the goal is to make the value the assistant adds clearly outweigh
  the harm done by the occasional falsehood.
• Hosted users can't opt out by forking. Perfetto is open source so folks
  can in theory strip out any feature they don't want, but a lot of users use
  the hosted build at ui.perfetto.dev. Thus, every feature has to earn its place
  on the value it genuinely adds, rather than simply being an attempt to
  shoe-horn in the latest tech trends.
• Disablable, possibly off by default. There should be a way to turn off all
  AI related tooling entirely. We still want to advertise these features - but
  only to make users who'd benefit aware of them and drive adoption, not to push
  AI on anyone.
• The value is the tools and skills, not the model. The real value of this
  feature isn't in the weights of the model used: it's the tools and the library
  of domain-specific skills the model draws on. The model itself is largely
  interchangeable, and Perfetto stays model-agnostic - users are always free to
  bring whatever model they prefer.

Note on the name. Intelletto is a codename for the assistant, used
in the plugin id (dev.perfetto.Intelletto); these docs otherwise just say
"the assistant". The name may be subject to change.

Scope

These docs cover the implementation in the OSS codebase - the provider-agnostic
plumbing. They make no judgement about which backends are used or what data is
acceptable to send to them: trace contents are sent to whichever endpoint the
user configures, and any data-egress / privacy policy is a deployment concern
layered on top, out of scope here. Likewise, API key handling is up to the
user or specific deployment.

These docs also don't cover classic ML models for uses such as classifying
traces. While the assistant could certainly make use of ML-powered tools in
the future, this work is focused on the assistant, which will leverage LLMs.

What we're actually building

The proposal breaks down into four components, each with a different value
proposition and a different degree of novelty. This doc specifies none of them
directly; each has its own RFC, linked below. In brief:

1. The embedded assistant — the sidebar chat itself, and what embedding buys:
   click-to-context and the ability to act on the same view the user is looking
   at. RFC-0032 (UX, agent loop, system prompt,
   conversation management).
2. The LLM framework — the provider-agnostic Provider → Config → Model
   gateway in the UI core: a shared backend any plugin or core feature can
   request a model from, with providers (and keys) pushable by extension servers
   so managed deployments are plug-n-play. RFC-0033.
3. The extensible surface — context, tools and skills registered by plugins
   and the core, so the assistant's capability grows with the codebase instead of
   a hand-maintained list. RFC-0034 (context
   injection) and RFC-0035 (tools & skills).
4. External harness integration — the same tool surface exposed to Gemini
   CLI / Claude Code / Cursor-style agents over a bridge via trace processor.
   RFC-0036.

Roadmap

Status: a working prototype exists. The screenshots in the child docs are
taken from it; it covers most of Phase 1 (the Provider/Config/Model stack with
Gemini and OpenAI-compatible providers, the sidebar with context chips and
tool use, the core tool surface). Phase 1 is therefore largely a landing
plan - review, hardening, and upstreaming existing code - rather than
speculative work, and the designs in the child docs are informed by it rather
than hypothetical.

See: #6209

• Phase 1 — core plumbing (mostly prototyped): Provider/Config/Model
  config layers, Gemini provider, agent sidebar, basic context injection
  (page + selection + viewport), core tool surface (SQL queries, selection,
  timeline navigation, Data Explorer state). Embedded assistant only - no
  bridge or external-harness support of any kind. Done when: a user with a
  configured provider can run the intro's example prompts end-to-end in the
  sidebar.
• Phase 2 — richer context & extensibility: click-to-context on anything,
  additional providers, tools from other plugins, skills integration,
  external agent conduit via TP, merge with and deprecate PerfettoMcp. Done
  when: a third-party plugin can register a tool, a skill, and a context
  provider without core changes, and an external harness can drive the UI
  through the TP conduit.
• Phase 3 — advanced: context compaction, more provider types, extension
  server integration, first-party extensions for external harnesses (if
  warranted), richer tool surface as plugins add their own.

Migration from com.google.PerfettoMcp

PerfettoMcp is folded into this design and deprecated once the assistant's
tool surface covers it (Phase 2, alongside the external-agent conduit). The
details live in RFC-0036.

Open questions

The open questions now live in the docs that own each area:

1. Build vs. library for the provider layer - decided (build); see
   RFC-0033.
2. Tools vs. commands - whether tools are their own registration mechanism
   or commands grow optional schemas + an allowlist; see
   RFC-0035.
3. External-agent conduit - the whole of
   RFC-0036 is the least settled part of the
   proposal.

---

💬 Discussion Guidelines:

• This discussion is automatically synced with the RFC document
• Please provide constructive feedback and suggestions

[github-actions[bot]]

📝 RFC Document Updated

View changes: Commit History

[stevegolton]

PTAL @LalitMaganti @primiano

[primiano]

Ok wow there is a lot. Thanks a lot for starting this.
Before we go into the details, my main advice is to split this RFC into a set of sub-rfcs.
The good news is that I feel you are spot on on the main areas to tackle. If you asked me I would have come up probably with the same set of areas.
However, having everything into one big rfc will make commenting and decision making very hard.
I would prefer if we turn this RFC into a top-level tracker (maybe just for the overall UX discussion), and then link out to sub- RFCs splitting out

• 1 RFC to discuss plugin architecture (including Procider/Model/Protocol): split of plugin responsibility, debate on core vs a plugin dependency, debate on details of Provider/Model/Protocol
• 2 Context: this is a super interesting topic (IMHO the biggest unknown / most handwavy) and deserves its own separate discussion (but also it's not a blocker because we could make progress meanwhile and punt this to a 2nd interation)
• 3 Tools and skills: I think these sections are less controverials. There are a few details to discuss, but I think they can easily go into their own RFC

My overall feedback

UX

UX wise I'm very aligned.
it would be nice if there was a way to "select" portions of the UI / set context (a bit in the way the chrome llm extension does, although i'm not a huge fan of that ux where you have to always first Add soemthing to context). What I have vaguely in mind is something where just above the chat there area bunch of hints of possible contexts (based on what you have on the page).
EDIT: after reading your Context injection section, we are aligned. More discussion below.

Plugin Architecture

Shouldn't dev.perfetto.llm actually be part of core (app.agents or something similar)? In my mind there shoudl be some "bonding" logic to hold the registry of possible backends. I totally buy that "intelletto" itself is a plugin. BUt i feel some of the upper logic belongs more to core

AsyncGenerator<StreamEvent> -> this feels a bit too limiting. IMHO this shoul return an object, so you can have methos like .cancel(), or sideband .btw(...)

Provider/Model/Protocol

On Provider/Models i'm mostly there, but I feel i'm a bit off by few details when I look more closerly. The way I see it is like this:

• A Provider is an abtract TS interface (NOT a pure data json struct). The provider is what a plugin (e.g Gemini, Claude etc) injects into core (`agents.registerProcvider(geminiProvider))).
• A provider takes a configuration (which is pure JSON struct) in input. This mainly contains API key, http endpoint override and few bits like this.
• What comes from the extension server is the provider config not the provider itself.
• The XxxProviderImpl comes from the various plugins

Then I'm not sure I fully understand what a "protocol" is in your description. Maybe is just a naming thing, but in my mind the next abstraction is a Model (another TS interface). So a Provider gives a list of models, which concrete classes that implement the model.
Essentially I can see a GeminiProvider that provides two implementations of Model: a LocalFastModel based on chrome's JS API, and a GeminiPro model which is based on using the Gemini API key.

Context inection

I am very aligned on the spirit of this. But I feel I have some different (and still handwavy) view on the details.

Plugins can supply additional custom context by registering context providers with the assistant plugin

I think this is where instead I feel core should have the responsibility of maintaining the abstraction of injectable context for AI agents, not depending on the assistant plugin.
So I disagree on assistantPlugin.registerCotnextForProvider, i feel that shoudl be a core API.

Not sure about <ui_context> wrapping at the DOM level. I think the key is to make sure we never have to look at the DOM, as:

• It's a nightmare to parse for AI
• it's slow to edit in JS

The way I'm thinking about this problem is that you have a UiContext wrapper element in mithril.
At the dom level, this mithril component does absolutely nothing (it just passed through it's children).
However, as a side effect, this wrapper component builds somewhere some sort of Shadow JSON DOM for Agents.

So in the code you do something like

return m(AiContext(
   m(Track, ...),
   {aiDescription: 'this element does ...'}
)),

and then as you render the whole page, it builds a shadow JSON dom like

{
  "view": "timeline",
  "components": [
    {
      "id": "track_1",
      "type": "slice track",
      "summary": "Scheduling slices for CPU 0",
      "raw_data_endpoint": "/ai/tracks/track_1"
    },
    {
      "id": "details_panel",
       "summary": "this panel shows the selected slice details"
      "available_commands": [
        {"intent": "query similar slices", "endpoint": "/ai/actions/..."},
      ]
    }
  ]
}

The only thing is that this feels liek a lot of code to maintain... I am still very torn and handwavy on this. Defo should discuss more

Tools

100% aligned. Again my only objection is that "registerTool" should be on core not on AssistantPlugin

[stevegolton]

FYI @dreveman

[dreveman]

This all looks good to me. The only points of feedback I have at this time are:

• same concern as @primiano, should the tools registration (and emulation when native tools are not supported) be at a layer below the AssistantPlugin? I'd like to use this at the GpuCompute plugin level and have support for a run_query tool. If the expectation is that I depend on the dev.perfetto.Intelletto plugin then maybe the current layering is right but my current assumption is that I'd use dev.perfetto.Llm directly for this? Maybe we should have a dev.perfetto.Agent plugin on top of dev.perfetto.Llm that provides an agent loop with tools?
• I'd like us to consider allowing one LlmProtocol impl to potentially route to N other implementations. E.g. a LlmProtocolCorpProxy that can route to both Gemini and Claude. Ie. make capabilities such as nativeToolCalling and streaming something that is model specific and not protocol specific. It's not critical as you can always create multiple downstream plugins for this use-case but I think something we should consider and be intentional about.

[github-actions[bot]]

📝 RFC Document Updated

View changes: Commit History

[stevegolton]

@dreveman thanks for taking a look.

same concern as @primiano, should the tools registration (and emulation when native tools are not supported) be at a layer below the AssistantPlugin? I'd like to use this at the GpuCompute plugin level and have support for a run_query tool. If the expectation is that I depend on the dev.perfetto.Intelletto plugin then maybe the current layering is right but my current assumption is that I'd use dev.perfetto.Llm directly for this? Maybe we should have a dev.perfetto.Agent plugin on top of dev.perfetto.Llm that provides an agent loop with tools?

Good question. In the doc I put this on the Intelletto plugin which means that other users of the LLM plugin will not have access to the tools. However I'm not convinced about this and I specifically mentioned it as open question:

Open question: whether tool registration lives in the llm plugin or the assistant plugin - other users of the llm plugin may or may not want to use tools.
...and...
3. Where tool registration lives (see Tools): the LLM gateway plugin or the assistant plugin - other gateway consumers may not want the assistant's tool semantics.

What's changed since is we've decided to merge the LLM plugin into the core - effectively making the language model interface a first class citizen. So we either register the tools/skills with the core or we register them with the Intelletto plugin only. Since I'm leaning more towards making tools and commands the same thing I think it makes sense to register tools with the core, and make them available to all users of the language model API if they want them.

What I'm sure about is that we don't want to push tools onto every user of the language model API - e.g. code completion probably has no business calling tools as this might just slow it down and/or leave side effects. We probably need some way of limiting the number of tools that are available depending on the use case like a tool tagging system and a filter defined when you create a new language model session.

What did you have in mind for the GPU compute plugin? Is this the auto-generated summaries in the details panel?

I'd like us to consider allowing one LlmProtocol impl to potentially route to N other implementations. E.g. a LlmProtocolCorpProxy that can route to both Gemini and Claude. Ie. make capabilities such as nativeToolCalling and streaming something that is model specific and not protocol specific. It's not critical as you can always create multiple downstream plugins for this use-case but I think something we should consider and be intentional about.

What use case are you envisioning for this?

If extension servers / the embedders file can define multiple configs then you can just configure two provider configs, e.g. one for anthropic and one for gemini, then the user can switch between providers/models on the fly.

If you're planning on using a multi model router type backend (e.g. openrouter), then you can just choose the openai compatible protocol and let the router handle any tool conversion or streaming adapter for us.

For example - the idea is you can configure as many providers as you like and switch between the models at will. In the screenshot(s) - I've configured Gemini and Chrome's on device model and the user can switch between them.