Development roadmap

[JasonTheAdams]

I think it would be helpful to take the requirements doc and derive a development roadmap. I think what exactly is the MVP can be defined as this is thought through, but isn't my initial intent to define. The phases are prioritized based on the greater WP AI initiative, with the goal being that we can move to other projects within that effort faster, without having to wait for this to be done in its entirety. Lastly, this is intended to be iterative with no backwards compatibility issues.

I'm not sure if we want to update the requirements doc with any/all of this, or if it's best to consider them separate. I'm good either way.

Phase 1: Basic Universal Client

Goal: Client is capable of registering providers, models, providing a uniform UI regardless of the model being used, and make the fundamental requests with the model.

• Client provides a unified API for interacting with various models
• Basic model parameters (e.g. temperature and top P) are standardized
• Providers and models can be registered
• Basic AI modalities
  • Text
  • Image

Phase 2: Agent Flows

Goal: Client is capable of providing a chat experience

• Response streaming is supported to provide fluid UX
• Client can receive a chat history

Phase 3: Agentic tooling

Goal: The client is capable of dynamic agentic capabilities with the introduction of tooling, opening it up to WP capabilities

• Tools can be defined and passed to the model
• Resources can be defined and passed to the model
• Registered models can specify whether they support tools and/or resources
• Model prompts support tool calls and results

Phase 4: Advanced features

Goal: Expand the capabilities of the client to include additional modalities and multi-modal

• Add more advanced modalities
  • Audio
  • Video
• Long-running operations are supported
• Structured output from AI

No Gos

The following are ideas and directions we are intentionally not taking at this time. Some indefinitely and others in the meantime

• The client is not responsible for persisting chat history in any way. It can receive chat history for context, but it's up to the implementing code to persist it.

[Jameswlepage]

For both phase 2 and 3, I'd drop the "agent" terminology.

Phase 2 seems like it should be Phase 1, in that it's not much of a larger scope, and passing message back scroll is an essential part of any chat-completion/instruct model nowadays (ie; system prompts). I can see streaming adding some complexity though.

Phase 3 to me is more about structured outputs / function calling. Within WP, as part of this, we can pass in Abilities (via Abilities API) to models as functions. It's up to the implementation on if these should be used agentically.

In Phase 4, "Structured output from AI" is an item, though to me, it's the same as "Model prompts support tool calls and results", which is a phase 3 item.

[JasonTheAdams]

For both phase 2 and 3, I'd drop the "agent" terminology.

You're right that "agent" is probably too specific of a term. What I'm really going for in my thinking is iterating up to a chat UI within WP — as I'm imagining that will most likely be the primary UI introduced in the feature plugin. Correct me if I'm wrong here, @Jameswlepage. My hope for WCUS is to have a chat UI that uses tools and a non-chat feature such as a button to get alt text for an image.

I wonder if "AI Assistant" or, more generically "AI Chatbot" is better used here. It feels useful in my mind to make that a clear target for his roadmap, as we're effectively meaning to iterate on the client until these goals are achieved.

Phase 3 to me is more about structured outputs / function calling. Within WP, as part of this, we can pass in Abilities (via Abilities API) to models as functions. It's up to the implementation on if these should be used agentically.

In Phase 4, "Structured output from AI" is an item, though to me, it's the same as "Model prompts support tool calls and results", which is a phase 3 item.

You might be right here. I'm still thinking on this. Tool usage isn't the same as structured output. Tool usage is a concept of the model itself; if it doesn't support tools then we simply won't even try it. But structured output, like JSON or something specific, varies from model to model, and may even be something the client itself takes a stab at (such as parsing or repairing JSON). That's why they were separate things in my mind, and something we don't necessarily need to accomplish our initial WP goals. But maybe I'm wrong! 😆

Curious what you think, too, @felixarntz!

[felixarntz]

@JasonTheAdams Thank you for putting together this proposed outline!

Before sharing my thoughts, I have to say I'm not entirely clear what the scope of this development roadmap is. Is this purely for this AI client SDK, or more broadly for the WordPress AI development tooling efforts?

For now, I'm going to interpret it as the former, and based on that I agree with @Jameswlepage to drop the agent terminology. More broadly, I'm not sure about the idea of assigning any higher-level terms that are not directly related to the client functionality to the roadmap phases, as that can lead to confusion. After all, this SDK's scope is only to provide the client, so while that by definition also includes support for things that agents typically use (e.g. function calling) and that chatbots typically use (e.g. chat history), the actual implementation of chatbots or agents should not happen in this SDK, as that would add a deeper level of opinionated logic that would therefore be better to live elsewhere (e.g. an eventual WordPress/ai-agent-sdk built on top of this SDK).

For the proposed sequencing, I agree for the most part, except what @Jameswlepage already mentioned, extending a bit on that:

• I'd say we should put chat history into phase 1. There is almost zero complexity to it because that's how almost all providers work.
• I'd say streaming should come in a later phase. It is a very important feature to support, but it's worth highlighting that it does not provide any actual feature per-se - it's a UX improvement. A very important UX improvement admittedly, but implementing streaming is very complex, and I think we should leave that for until we got all the essential features available. Whatever you can do with streaming, you can also do without streaming.
  • When developers start using the MVP and don't find streaming included, they may wonder about that, so I think we should call out very visibly later that we will add it.
  • But I think it's more important for the MVP to give developers the flexibility and tools to be able to leverage all common AI capabilities (including e.g. function calling) than having streaming in there right away.
  • I don't mean to say that streaming should only come after all other actual features. I'd say it's a sufficiently important UX improvement to consider it over more edge-case capabilities like audio or video generation. But I'd slot it in later than where it is currently.
• In other words, I would drop phase 2 as it currently stands, and instead go with only the 3 other phases.

My only other feedback is: Instead of focusing on modalities to support in this list, we should focus on capabilities. Supporting different modalities is super simple. Video, audio, or even documents work no different from images, so there's no good reason IMO to delay those. What does require a lot more effort is to implement the actual capabilities to generate content in these modalities (i.e. audio generation, video generation). For those, I agree they should only come in the last phase.

[felixarntz]

One additional thought, as I'm thinking through the implementation breakdown: We may want to leave out image generation as well and focus the MVP purely on text generation.

Image generation is probably the second most important capability, but every new capability is another time investment to implement, and if we want to get something out the door sooner than later, I think we should keep image generation out of the MVP. So maybe that on its own could become a new phase 2?

[JasonTheAdams]

Thank you both for the feedback! I'll get this updated shortly!

Before sharing my thoughts, I have to say I'm not entirely clear what the scope of this development roadmap is. Is this purely for this AI client SDK, or more broadly for the WordPress AI development tooling efforts?

I've been thinking of it as the roadmap for this client influenced by the needs of the greater WP project. I don't think it benefits us to completely isolate our thinking for this client at this time. After we have the feature plugin and this is doing what we need it to for our WP purposes, then I think this is more likely to take on a life of its own.

My only other feedback is: Instead of focusing on modalities to support in this list, we should focus on capabilities. Supporting different modalities is super simple. Video, audio, or even documents work no different from images, so there's no good reason IMO to delay those.

My reason for punting video and the like was to also delay async operations. Text and images will be synchronous, so I figured that kept it a bit simpler.

Also, what capabilities would you recommend calling out to defer that aren't modalities, @felixarntz?

[felixarntz]

I've been thinking of it as the roadmap for this client influenced by the needs of the greater WP project. I don't think it benefits us to completely isolate our thinking for this client at this time. After we have the feature plugin and this is doing what we need it to for our WP purposes, then I think this is more likely to take on a life of its own.

That makes a lot of sense 👍

My only other feedback is: Instead of focusing on modalities to support in this list, we should focus on capabilities. Supporting different modalities is super simple. Video, audio, or even documents work no different from images, so there's no good reason IMO to delay those.

My reason for punting video and the like was to also delay async operations. Text and images will be synchronous, so I figured that kept it a bit simpler.

My comment wasn't to say that we should pull ahead video or audio generation, but that modality is not the same as capability, hence that roadmap point being confusing. "video" and "audio" are modalities, "video generation" and "audio generation" are capabilities.

The "video generation" and "audio generation" capabilities are the things that are more complex (often long-running) and require a whole set of additional work, and therefore should be done later.

On the other hand, "video" and "audio" modalities are basically zero extra work. It just means that we support content of those modalities in e.g. the Message object, which works the same way as "image". If a provider supports receiving "video" or "audio" input, there's no difference in implementation. If the provider doesn't, it simply would be marked as not having support for these modalities. So we can do this right away, together with the other modalities.

[JasonTheAdams]

Hmm... I think I get what you're saying with regards to modalities versus capabilities, @felixarntz. For the client to merely have the concept of an image modality, for example, isn't a big deal. It's just a concept. But plugging that in to the model such that it generates said modalities (especially kinds like video and audio) is more complicated — which you're suggesting we defer.

One thing that's a bit unclear to me is that it seems like you're saying that supporting many modalities as an input to the modal is fairly trivial. Either it supports receiving it or it doesn't, and it doesn't change much from the client perspective. I would still view this as a capability — that is, the model is capable of receiving audio — but maybe I'm following? On the other hand, generating all the modalities (i.e. as an output) is more complicated, especially when you get into operations and the like.

Am I following you correctly, here? You're suggesting that:

• Defining the modalities and supporting their input is reasonably trivial
• Handling the generation of certain modalities (e.g. video and audio) is more complex and should be deferred?

[felixarntz]

Exactly, your two bullet points capture what I mean.

One thing that's a bit unclear to me is that it seems like you're saying that supporting many modalities as an input to the modal is fairly trivial. Either it supports receiving it or it doesn't, and it doesn't change much from the client perspective. I would still view this as a capability — that is, the model is capable of receiving audio — but maybe I'm following?

That's a fair point, considering that as a capability. Thinking that further though, as we already defined the concept of capabilities vs options, so far we were leaning towards this being part of an option. Capabilities, in our terms, would be only the overarching "features", like text generation and image generation. Everything else is an option. Whether a model supports e.g. temperature could also be considered a capability in colloquial terms, but then arguably everything is a capability.

A key difference in our definition so far is that capabilities are just strings that are either supported or not, whereas options may have additional constraints in what is supported for a specific option.

Let me give an example:

• Can I generate text using this model? -> capability
• Can I pass temperature to the model? -> option (because not an overarching feature)
• Can I pass outputMimeType => image/webp to the model? -> option (because the specific value image/webp may or may not be supported by a model, even if they generally support the outputMimeType option
• Can I pass outputModalities => [ 'text', 'image', 'audio' ] to the model? --> option, which effectively enables multimodal output of these two modalities (and again, may not work for every model with that concrete value, e.g. most only support one at a time, and the others today only support [ 'text', 'image' ], without audio
• Can I pass inputModalities => [ 'text', 'image' ] to the model? --> option (same consideration as above; technically this never needs to be passed because it can be inferred from the Message`s given)

I hope this helps clarify the approach so far.

[JasonTheAdams]

Thanks for breaking that down, @felixarntz! If I'm following, we have three related terms:

• Capability
• Option
• Requirement

The first two are in the glossary, while the latter is an internal term we introduced recently. As I follow, both capabilities and options translates as requirements when we determine which model can perform the given set of defined work (e.g. prompt).

So, if I'm tracking, it's trivial for us to define the modalities and options as something to pass as input to the models. It doesn't really make a difference as it's just "here, I have more data for you." Capabilities is where things get more complicated because how the process works (results vs operations) and is returned from the model gets more interesting. Therefore, in the roadmap, we can start with many options and modalities, but we should defer the capabilities themselves — first prioritizing text, then images, and then the rest as we see fit.

Am I tracking? 😄

[felixarntz]

Yes, that sounds right. Both capabilities and options map to model requirements. That's mostly relevant for the extender API.

For the implementer, a capability (which they don't need to know about as a term) maps to a specific prompt builder terminate method (or set of terminate methods), such as generateText for the text generation capability. An option is something that they provide to alter how that method works, either explicitly (via something like usingTemperature) or implicitly (via specific modalities in in the Message).

Your point about modalities and options being straightforward and capabilities being the bulk of work and therefore being deferred is spot-on, that's a great way to summarize it.

[felixarntz]

@JasonTheAdams We haven't looked at this for a while, I think some of this needs to be revised based on what we're doing for our MVP.

I just marked all points with a checkmark that I think will be done by the time we release a first version (likely next week). So we should probably move all that into "Phase 1", and then rethink the subsequent phases.

Two other aspects I think we should clarify:

• Tools and resources, while being different things from an architectural perspective in ways that affect Abilities API and MCP, for the sake of communicating with LLMs, both are simply function declarations (think doThis -> tool, getThis -> resource), so I think we should combine those into simply saying "functions" or "function declarations".
• The points about modalities are not really about the modalities, they are about generating output in these modalities. Right now that's not clear, it could be misinterpreted as "we don't support passing audio input", which we actually do. It's only about the modality output / generation.

Let me know if you have any feedback or additional thoughts. Would be great to bring this up to date soon, as it would be a great place to point people to when they ask "what's next?".