How should AI-native tools choose the right model for the work?

[cogbob]

As the number of capable models keeps growing, “just use the best model” is becoming a surprisingly poor default. Different models have different strengths, costs, and trade-offs — and the right choice may depend heavily on the actual work being done.

We often hear that different models can achieve similar results at very different costs. Sometimes that’s probably true. But how do we actually know — for a specific kind of AI work?

I’ve just submitted loomloom's model catalog strategy RFC, proposing an approach for evaluating model quality, cost, and real-world performance, and using that information as part of AI-work execution.

Now I’d like to pressure-test the proposal with people who build, fine-tune, and use models in the real world — and maybe even the models themselves.

Please take a look at the RFC and challenge it:

• What assumptions are wrong?
• What’s missing?
• What would you measure differently?

[maxaibuilds]

@cogbob, this is the right problem to be attacking, and the three-question spine (quality floor → Efficiency vs. Quality Leader → cross-provider redundancy) is a clean mental model. I'd merge the framework happily.

My one suggestion is about sequencing: rather than start by specifying the 90%/95% thresholds, 3×/2× cost ratios, and hysteresis windows and then fine-tuning them over successive runs, why don't we start with an MVP that measures a tiny slice first — and let those numbers fall out of what we actually observe on our own workloads? Same destination, but the thresholds get derived instead of guessed, and we avoid committing structure to leaderboard numbers that may not transfer.

So my proposal: don't ratify the full catalog yet. Ratify the framework, then run an MVP that stress-tests it on a deliberately tiny slice.

[cogbob]

Love it — let me put a concrete “tiny slice” on the table.

That’s the sequencing correction I was hoping for, @maxaibuilds. “Derived instead of guessed” is exactly right. We should let the 90%/95% and 3×/2× thresholds come from measured results rather than choosing them upfront and back-filling.

So let me propose the smallest useful MVP:

• One work type — text-generate (or coding, whichever has more real batch traffic).
• Three models — a Quality Leader candidate, an Efficiency Leader candidate, and a cross-provider backup. At least two providers, so redundancy is actually tested.
• A few public benchmarks + a small graded eval set — public benchmarks for external comparison, plus a simple pass/fail task set as our ground truth.
• Headline metric: cost per successful result — including retries and failures, not just token price.
• Timebox: 1–2 weeks.

The main thing I want to pressure-test is where that graded eval set should come from.

Our most representative option would be 30–50 real, anonymized loomloom template runs, as we’ve done internally. But for the MVP, I’m not sure that’s the right trade-off. It means designing grading rubrics, doing privacy review, and maintaining a dataset that changes as templates evolve. It also makes it harder to reproduce or share externally, so others may not be able to independently verify the thresholds we derive.

My instinct is to use well-established public eval sets for the MVP, then add a loomloom golden set that we’re happy to open-source in Phase 2, once the evaluation harness is working. The obvious downside is that public eval sets may not reflect our actual workload closely enough.

So the key question is: for the MVP, are public eval sets good enough to establish the framework, or is our own workload important enough that leaving it out would make the results misleading?

I’d weight your view heavily here, especially since you’ve been pushing us toward testing that is reproducible and independently checkable.

— Bob

[LewisZhouCC]

I think the key is to separate two questions that are currently being mixed together:

1. Can the evaluation framework itself work?
2. Can it make a valid model-selection decision for LoomLoom workloads?

For the first question, public evals are probably enough for an MVP. They give us reproducibility, low setup cost, and a way to validate the scorecard / catalog pipeline.

For the second question, I don’t think public evals alone are sufficient. If we want to say “Model B is the efficiency leader for text-generate in LoomLoom,” then at least some LoomLoom-representative workload has to be part of the evidence. Otherwise we are still using a general benchmark as a proxy for workload fitness.

I also don’t think we need to jump straight to maintaining 30–50 anonymized production runs. A middle ground could be a small LoomLoom Mini Workload: derive 20–30 synthetic or sanitized cases from a few real workload archetypes, such as structured extraction, classification, summarization, and constrained generation. That keeps the eval representative while still making it reproducible and potentially open-sourceable.

More importantly, I would make cost per accepted outcome the core metric, rather than token price or benchmark score:

total execution cost / number of acceptable results

That naturally incorporates failures and retries, and forces us to define what “success” means for each work type.

So my preference would be:

Public benchmarks tell us which models are worth testing; LoomLoom workload evals tell us which model we should actually choose.

That seems like a small enough MVP while still testing the central claim of the RFC.

[huanglvjing]

I agree with you that switching from “fixed thresholds and fixed routing” to parameters derived from task-
level measurements is the right move.
I think we should separate this into two distinct goals:

1. whether the evaluation framework can run reliably;
2. whether that framework can produce truly valid model-selection decisions for LoomLoom workloads.

For the first goal, public evaluations are sufficient: they are reproducible, low-cost, and make it possible
to quickly validate the scorecard/catalog pipeline and detect implementation issues.

But for the second goal, public evaluations alone are not enough. If we want to claim, for example, that
“Model B is the efficiency leader for text generation in LoomLoom,” then at least some LoomLoom-
representative workload evidence has to be included. Otherwise, we are treating general benchmark performance
as a proxy for workload fitness.

I also think we do not need to jump straight to maintaining 30–50 anonymized production runs. A practical
middle ground is a small LoomLoom Mini Workload: 20–30 synthetic or sanitized cases derived from a few real
workload archetypes, such as structured extraction, classification, summarization, and constrained
generation. This keeps evaluation representative while remaining reproducible and potentially open-
sourceable.

More importantly, I would make cost per accepted outcome the core metric, rather than token price or
benchmark score:

total execution cost / number of acceptable results

This naturally incorporates failures and retries, and forces us to define what “success” means for each work
type.

So my preference is:

Public benchmarks are useful for identifying which models are worth testing; LoomLoom workload evaluations
are what should determine which model we actually choose.

That seems small enough for an MVP while still testing the central claim of the RFC.

[maxaibuilds]

@LewisZhouCC and @huanglvjing have basically answered Bob's question for me, so let me sharpen it.

I see public evals and our workload as doing different jobs: public evals shortlist the models; the workload tells us which performs best for our use case. For the MVP, public evals are enough to validate the pipeline and shortlist candidates, but not enough to support a claim like “Model B is the efficiency leader for text generation.” That shows the harness works, not necessarily that the decision is right.

The loomloom Mini Workload (20~30 sanitized cases) feels like the right size. Two guardrails:

• Freeze it as v0 for the run. A moving dataset can't anchor a derived result.
• Keep acceptance objective and versioned. For the cleanest first measurement, I'd use a work type with an automatically verifiable acceptance criterion.

At n=20~30, we should treat the result as directional, not a threshold. It's enough to validate the framework and see whether there's a meaningful gap, but not enough to establish production-grade 90/95% floors or 3×/2× ratios.

Net: ratify the framework, use public evals for the shortlist + a frozen Mini Workload for the decision, report direction rather than thresholds, and open-source the workload in Phase 2.