Two scenarios are answerable with no tools mounted — a contamination control, and what it caught

[arthapraha]

Repository: IBM/AssetOpsBench · Observed at: e11d1c1 (current main)
Source: finding 9 of our defect report · follows #503, sibling of #506 and #507

Scenarios 205 and 206 can be answered correctly with no MCP servers mounted at
all. They are useful questions; they are just not measuring tool use when they
are counted that way.

This is offered as a caveat for the docs, not a defect report. Nothing here
needs a code change.

How we found it, which matters more than the finding

Before running any tool-use comparison we screened the candidate pool: every
scenario whose ground truth we could derive was first run with no MCP servers
mounted, three samples each, and any scenario that produced a correct answer
without tools was excluded from the comparison.

Of 14 candidates:

outcome with no tools	n	scenarios
abstained on all three samples	11	the rest of the pool
answered, but wrongly	1	4
answered correctly — excluded	2	205, 206

scenario	question	correct without tools
205	"Is LSTM model supported in TSFM?"	1 of 3
206	"Is Chronos model supported in TSFM?"	2 of 3

The exclusion rule is deliberately conservative: one correct answer in three is
enough to disqualify. A wrong answer is not contamination — scenario 4 answered
wrongly once and stayed in the pool — because a model guessing badly without data
is exactly what a tool-use question should produce.

That is why our published comparison covers 12 scenarios rather than 14. The
two were removed before any measurement, not discovered afterwards.

The honest reading, including the one that weakens it

Both 205 and 206 have gold false, so "the model simply tends to answer no" is
a live alternative to "the model knows these from pre-training." We can't fully
separate them with three samples.

But it is not the whole story: scenario 210 also has gold false and abstained
all three times. Whatever produced the answers on 205 and 206 was selective —
and the obvious candidate is that LSTM and Chronos are widely known model
names, while "any model with context length 1024" cannot be answered without
the catalogue.

Either way the consequence for scoring is the same, which is why we exclude on
the observation rather than on the explanation.

One control on the control

The screen also carries a tripwire at 90%: if qualification-by-abstention had
approached totality, the honest conclusion would have been "this probe does not
discriminate" rather than "here is a clean shortlist" — if everything
qualifies, nothing was selected. The observed rate was 11 of 14 (78.6%), so
the screen was doing work rather than passing everything through.

We mention it because a screening step that cannot fail is not a screening step,
and we would rather hand you the threshold than the result alone.

What this means for anyone scoring against these questions

Not a defect in 205 and 206. They are reasonable questions and we would keep
them. But:

• A system that answers them has not demonstrated tool use, so counting them
  toward a tool-use metric credits the model's priors rather than its retrieval.
• They make good controls — a system that gets them right and everything
  else wrong has told you something useful.
• Both are boolean with a chance floor of 0.5, so they are the questions where
  guessing pays best; that is the category worth screening first in general.

What we are asking for

Nothing in the code. A line in docs/evaluation.md noting that some
questions are answerable from general model knowledge, and that a tool-use score
should scope them out or report them separately.

Happy to share the screening harness, or to run it across the full utterance
set rather than just the derivable subset, if that would be useful to you.

Relationship to the other issues

This is the last of four issues that a single documentation PR closes, after
#506 and #507. Unlike those two, this one does not depend on the answer to
#506 — whether or not the public artefacts are meant to be scoreable
end-to-end, these two questions are answerable without the environment.

The PR follows now that all four have numbers. The content is written; its
location is your call.

---

Findings 1–6 reached the maintainers by email on 10 August. This continues that
exchange, and is the fourth of the six issues announced in #503.

[arthapraha]

Correction to this issue's closing section, from us.

It says "the last of four issues that a single documentation PR closes" and
"the PR follows now that all four have numbers." Both were wrong when this
was filed. At that moment only #506 and #507 existed alongside it — three of
the four — and this was the third documentation issue, not the last.

The fourth is #509, filed since, and the PR follows from there.

Correcting by comment rather than by editing the body above, so the record shows
what was said and what it should have said.