How do you measure the success of AX in a quantitative or qualitative manner?

[sean-roberts]

I have some ideas on this but I'm really curious what others are thinking about. Convex's evals is an inspiring start to this consideration

[quarterback]

who’s evaluating AX—the user or the system owner?

[sean-roberts]

How would the system owner know that their investments in AX are impactful

[mbcrosiersamuel]

This is a super interesting question!

I feel like we'll probably see some level of sort of super high-level grading system, similar to lighthouse or an SEO grader today, e.g. for things like:

• Load time
• Convenient availability of semantic data (or maybe % coverage of llms.txt if something like that becomes more standard the way sitemaps or SEO metadata are today)
• Access/auth compatibility (ensuring desired agents can access and undesired bots/agents aren't able to access)
• Analytics on how many and which agents are visiting/buying/etc. -- the next gen of web/product analytics

And then at a higher sophistication level, maybe something like 'Agent Journey Tests' where you automate runs of the X most popular agents and evaluate % successful completion of core user workflows (e.g. what % of the time can the agent successfully make a purchase) and get aggregated results. Convex Evals is definitely good inspiration for this; the hard part, though, would be figuring out what the right tasks & tests are for each site (or each agent <> site combo if agents become more specialized)

[sean-roberts]

You're spot on @mbcrosier! This is precisely the direction I'm exploring with some prototypes. Other clear areas that come to mind...

• content/token size
• AX capabilities
• eval based quality checks
• cross model evals

I'm opening a PR soon around the principles of AX, I'd love your feedback on that and I think it will help ground some of these areas of validation.. if the principles are what are important, that's what should be verified.

[mbcrosiersamuel]

Those are great additions -- lots to think about.

And sounds super valuable as a way to ground the discussion. Happy to take a look!

[esthor]

Hi. 👋 Hope it's ok to join the party a little late...

I've been hacking on a little framework for tackling some of this and would love to join the convo!

Agent Journey Tests

Yes! I've been calling something like this "Intent Tests". (In my mental model, they sit above e2e tests, and can also apply to human users.)

figuring out what the right tasks & tests are for each site

I split into a 'supervised' and 'unsupervised' flows, with some different agent orchestrations and tools:

• Supervised: Declare a core user intent you care about (and we wrap some magicks to turn them into evals).
• Unsupervised: Figure out what the intent of the website is or even figure out which website to go to. Very meta, but also extremely practical as this tends to get at fundamental AX problems straight away and "AX-as-a-blocker" for sites (i.e., it's such bad AX every agent will 'take their business someplace else').

(I feel like we're at the "nothing is really built for these users" super early days, but the northstar imo is a world where unsupervised flows would make the most sense for Agentic Discovery , but would depend on more mature AX sites/services and registries/directory services tailored to agents.)

each agent <> site combo if agents become more specialized

I think the variance is probably so large between models and families, prompts, sites, and tasks that it's almost a requirement to have concurrent runs of various models for the different combinations of other variables. I haven't yet done enough runs of these combos to have anything more than anecdotal...but if someone wants to throw me a bunch of GPU compute, I will be happy to create an open leaderboard! 😄

[sean-roberts]

Overfitting and underfitting will be a critical part of this conversation as context from one system can be over/under, impacting the context to other parts of systems. For example, context can suggest using one provider over all others which obviously impacts other tools ability to play their role.