v0.10.0
Added
-
AEC-Bench (
bench/) — an agentic benchmark for Adobe Experience Cloud.
Nobody publishes one, and every MCP server in this space is currently
described by its tool count, a number that measures surface area rather than
competence: fifty tools that 404 score higher than ten that work.Three properties define it. Assertions are made against Adobe, not against
tool output — a task that says "create a segment" is scored by a GET that
finds it, never by the create call's own success flag, because a write
reporting on itself is not evidence. Cleanup is scored: completing a task
while leaving an orphan is not a pass, since a benchmark that dirties the
tenant can only be run once honestly. Tier 1 is read-only and safe against
any tenant including production, because a benchmark nobody dares execute
measures nothing.Tier 1 (5 tasks) and tier 2 (2 tasks) both score 100% against the live
sandbox, with zero residue independently confirmed. Tier 3 (irreversible) is
defined but ships empty on purpose — the tasks that belong there are
non-cancellable and can take 30 days to settle, and a benchmark is not a good
reason to run one.Run with
npm run bench(read-only) ornpm run bench:write. Tier 2+ fails
closed: it refuses unless the expected org and sandbox match the credential
and Adobe classifies the sandbox asdevelopment.Its own first run scored 4/5 — and the failing task was a bug in the task,
not the tool. That distinction is now a comment in the file, because a
benchmark that cannot tell its own bugs from the product's is not one anyone
should trust. -
npm run validate:tools— the tool-level live sweep, promoted to a script. -
Adobe Journey Optimizer tools —
ajo_list_campaignsand
ajo_get_campaign. AJO was already attached to the credential, so its surface
was probed live rather than inferred from documentation, and only what
answered is wrapped: campaigns is the sole reachable AJO surface on this
tenant. Journeys, messages, channel surfaces, content templates, fragments,
offers and decisions all return an HTML 404 — the gateway has no such route —
and are deliberately not implemented. That is the same test the datastream
tools failed for five releases.They use the
ajo_prefix rather thanaep_, because Journey Optimizer is a
separate Adobe product with separate licensing and a name that hides that
makes entitlement failures harder to read.ajo_get_campaigntreats Adobe'sCJMCMP-2044-404("no acceptable version")
as a distinct state rather than a generic 404: the campaign row exists but has
never been published, and flattening the two sends someone hunting for a
campaign that is right there as a draft.Campaign writes are not implemented. The routes exist, but this sandbox
holds no campaign to exercise them against, and shipping an unvalidated write
path into a product that sends messages to real people is not a trade worth
making. -
aep_delete_segment— segments could be created but never deleted, so
every segment an agent made was permanent. That asymmetry left orphans in any
sandbox an agent touched.dryRundefaults true, the confirmation is bound to
the segment id, and deletion is verified by a follow-up GET rather than by
trusting the200that DELETE returns with an empty body. Live-validated
create → delete → verified-gone. Tool count 48 → 49. -
scripts/fixture-run.mjs— creates the minimum throwaway fixtures needed
to exercise read-only tools that require an id, then removes them. It ledgers
each object before creating it, so a crash leaves a record rather than an
orphan. Zero orphans across runs; the sandbox held 0 segments before and after.
Fixed
-
An empty privacy job list was reported as an error. Adobe Privacy Service
answers a query that matched nothing with HTTP 404 and
"detail":"Not able to find job data.".aep_list_privacy_jobssurfaced that
asAEP_404, so "you have no privacy jobs" — the normal state of most
tenants — looked like a broken tool and pushed agents into retrying. It now
returns an empty list. Narrowly scoped: a 404 reading as not authorized or
not provisioned is still an error, because that one needs a human. -
Adobe's nested error detail was being thrown away entirely. The error-body
sanitizer whitelisted top-level keys only, and Adobe nests the useful part
of many errors undererrors:{"errors":{"errorCode":404,"title":"Resource not found","detail":"Not able to find job data."}}errorswas not on the whitelist, so the whole envelope was dropped and
clients received{}. Every error of that shape arrived with no explanation,
which is how a 404 meaning "nothing matched" became indistinguishable from a
404 meaning "no such route". The sanitizer now recurses one level into
errors, applying the same whitelist, so the detail survives while unlisted
fields inside it are still dropped. Onlyerrors, only one level — the
whitelist stays authoritative.
Added
-
scripts/validate-tools-readonly.mjs— live validation that invokes the
registered tool handlers rather than raw HTTP paths, so it reports what an
agent would actually get. It refuses to call any tool not annotated
readOnlyHint: trueand force-clearsAEP_ALLOW_MUTATIONSbefore building
the client, so it cannot mutate. 18 of 18 runnable read-only tools pass; the
other 9 need an id this sandbox cannot supply and are reported as
no-fixturerather than guessed at.It exists because a path-level probe and the tools it was meant to validate
had silently disagreed once before, and neither found the other's bug. Both
fixes above came out of its first run.