Skip to content

v0.7.0-beta.2 — the benchmark, re-measured

Pre-release
Pre-release

Choose a tag to compare

@raj-devnath raj-devnath released this 05 Nov 02:21
· 47 commits to main since this release
b41f3ea

The benchmark table is re-measured, and one headline number went down

docs/benchmark.md claimed to describe 0.6.0b1 and said so at the top,
because 0.7.0b1 shipped without re-recording the answers that had stopped
replaying. That debt is paid: 190 answers re-recorded against the pinned
models, every published row named explicitly rather than through the
Haiku-only default command, and the replay step in release.yml is blocking
again -- which was always the condition for having made it advisory.

messy_mart with a Haiku-class answerer: semantic 0.89 -> 0.82. Three
questions, two in complex and one in simple. That is beyond the ~1
question of run-to-run variance the page documents, so it is reported as a
real movement. The linter, which previously added a question on that fixture,
now adds none there. Against that: the Sonnet row improved (0.87 -> 0.89, and
0.89 -> 0.92 with lint), and the ontology column rose 0.79 -> 0.82 to sit
exactly level with the flat layer rather than below it.

The claim that the layer "equalises models" is weakened and has been
rewritten rather than quietly kept. Haiku and Sonnet are now three questions
apart with the layer where they were one; on raw schema they are five apart.
The defensible statement is that the layer moves the cheap model from well
behind to close behind.

Every schema_only cell is unchanged on both models, and fan_trap and
tpcds_clean replayed without re-recording at all, so the raw-schema
baseline is the same measurement it was. The derived figures were recomputed
rather than carried over: the untouched-30 arithmetic is 0.47 -> 0.80 (+70%
relative), and the eight reworded questions pass 7/8 under the layer, not 8/8.

The repository moved to github.com/semlayer-dev/semlayer

Same code, same maintainer, new home: the project now lives under an
organisation named for the product rather than for the company that
currently holds it. GitHub redirects the old path, so an existing clone,
remote or bookmark keeps working and nothing needs to be re-cloned.

What changed in this repository: project.urls, the CLA bot's document
link, the telemetry docs link the CLI prints, the clone instruction
semlayer benchmark gives when run from a wheel, and the beta-feedback
link. Nightly's live job is gated on github.repository_owner, so that
gate moved too -- left alone it would have stopped matching and the job
would have gone quiet without ever failing.

The security contact in SECURITY.md is now hello@semlayer.dev, on the
domain the project actually publishes under. The package's authors
field is now semlayer; the FSL copyright in LICENSE.md and the CLA's
grant are unchanged, because those name the legal licensor and not the
product.

A mcp-name: dev.semlayer/mcp marker is now in this README. The official
MCP Registry reads it off the published PyPI description to confirm that
whoever claims the dev.semlayer namespace also controls the package.

route_intent no longer tells an agent to use a summary table it may not use

Every aggregate the engine finds arrives as mapping_source: heuristic with
routing.status: advisory, and SPEC 2.7 says such a table MUST NOT be
routed to automatically. route_intent listed it under use anyway, next
to the fact table, and the ROUTING line in context_for_question said
"use ord_hdr, dly_sls_agg, mth_cust_agg". On any layer with a reconciled
aggregate, every agent was handed the instruction the spec forbids.

The document was never wrong: repo_knowledge.routing[].use is the
producer's pointer list, and docs/metrics.md said a consumer must read the
aggregate's grain and status before following it. The MCP tool is that
consumer, so the rule now lives there. An aggregate stays in use only when
its routing status is verified; otherwise it moves to aggregates, with
its grain, mapping source, consistency and a one-sentence caveat. A
deprecated or orphaned aggregate appears in neither. The ROUTING line
renders the same split, and the offered table is still disclosed in full so
the agent can see the grain it has to cover. The layer file is not modified
by the read, and semlayer review rejecting a mapping still removes it from
the producer's list as before.

The agent context is part of the benchmark prompt, so the messy_mart answer
cassettes miss after this change. They already missed before it (the
release-gate note at 0.7.0-beta.1 records the re-record as owed), so no
test that replayed yesterday skips today.

Whether the answer step worked is now a measurement

The session event recorded which MCP tools were called and could not say
whether any of them produced an answer. tools says run_sql was called;
it cannot say whether rows came back, whether the statement was refused,
whether check_sql stopped it, or whether the warehouse rejected it. A
session refused four times and a session that answered four questions were
the same row.

run_sql on mcp events is four integer counts — ran, refused,
lint_blocked, warehouse_error — all four always present, each capped at
999. Never the SQL, never the error text, never a table name, never a
per-call timestamp
, and a test asserts that over the whole serialized
event rather than over the one field.

Counted where each outcome happens rather than classified afterwards,
because one of them cannot be recognised after the fact: the refusal while
an inference holds the connection returns the same shape as a warehouse
rejection, so anything reading the result would file "we made you wait"
under "your warehouse is failing".

This is the measurement the keyless default was argued on — whether a layer
built with no model key actually answers questions — and it did not exist
when that decision was made. schema is 5.

A field can no longer ship undocumented

host, event_id, tools, via and requested_connector all reached
users with docs/telemetry.md updated by hand and the internal field spec
untouched, for two weeks, with every check green. The spec claimed a unit
test asserted the whitelist was complete; that test only proved the
serializer drops unknown keys, which is a different property, and nothing
tied the shipped field list to any document.

Every key the serializer can write must now appear in docs/telemetry.md
or the suite fails naming it. The public page is the promise that has to be
right, so that is what the test reads.