Skip to content

v0.10.0

Choose a tag to compare

@github-actions github-actions released this 18 Aug 18:28
· 4 commits to main since this release

Added

  • AEC-Bench (bench/) — an agentic benchmark for Adobe Experience Cloud.
    Nobody publishes one, and every MCP server in this space is currently
    described by its tool count, a number that measures surface area rather than
    competence: fifty tools that 404 score higher than ten that work.

    Three properties define it. Assertions are made against Adobe, not against
    tool output
    — a task that says "create a segment" is scored by a GET that
    finds it, never by the create call's own success flag, because a write
    reporting on itself is not evidence. Cleanup is scored: completing a task
    while leaving an orphan is not a pass, since a benchmark that dirties the
    tenant can only be run once honestly. Tier 1 is read-only and safe against
    any tenant including production, because a benchmark nobody dares execute
    measures nothing.

    Tier 1 (5 tasks) and tier 2 (2 tasks) both score 100% against the live
    sandbox, with zero residue independently confirmed. Tier 3 (irreversible) is
    defined but ships empty on purpose — the tasks that belong there are
    non-cancellable and can take 30 days to settle, and a benchmark is not a good
    reason to run one.

    Run with npm run bench (read-only) or npm run bench:write. Tier 2+ fails
    closed: it refuses unless the expected org and sandbox match the credential
    and Adobe classifies the sandbox as development.

    Its own first run scored 4/5 — and the failing task was a bug in the task,
    not the tool. That distinction is now a comment in the file, because a
    benchmark that cannot tell its own bugs from the product's is not one anyone
    should trust.

  • npm run validate:tools — the tool-level live sweep, promoted to a script.

  • Adobe Journey Optimizer toolsajo_list_campaigns and
    ajo_get_campaign. AJO was already attached to the credential, so its surface
    was probed live rather than inferred from documentation, and only what
    answered is wrapped: campaigns is the sole reachable AJO surface on this
    tenant. Journeys, messages, channel surfaces, content templates, fragments,
    offers and decisions all return an HTML 404 — the gateway has no such route —
    and are deliberately not implemented. That is the same test the datastream
    tools failed for five releases.

    They use the ajo_ prefix rather than aep_, because Journey Optimizer is a
    separate Adobe product with separate licensing and a name that hides that
    makes entitlement failures harder to read.

    ajo_get_campaign treats Adobe's CJMCMP-2044-404 ("no acceptable version")
    as a distinct state rather than a generic 404: the campaign row exists but has
    never been published, and flattening the two sends someone hunting for a
    campaign that is right there as a draft.

    Campaign writes are not implemented. The routes exist, but this sandbox
    holds no campaign to exercise them against, and shipping an unvalidated write
    path into a product that sends messages to real people is not a trade worth
    making.

  • aep_delete_segment — segments could be created but never deleted, so
    every segment an agent made was permanent. That asymmetry left orphans in any
    sandbox an agent touched. dryRun defaults true, the confirmation is bound to
    the segment id, and deletion is verified by a follow-up GET rather than by
    trusting the 200 that DELETE returns with an empty body. Live-validated
    create → delete → verified-gone. Tool count 48 → 49.

  • scripts/fixture-run.mjs — creates the minimum throwaway fixtures needed
    to exercise read-only tools that require an id, then removes them. It ledgers
    each object before creating it, so a crash leaves a record rather than an
    orphan. Zero orphans across runs; the sandbox held 0 segments before and after.

Fixed

  • An empty privacy job list was reported as an error. Adobe Privacy Service
    answers a query that matched nothing with HTTP 404 and
    "detail":"Not able to find job data.". aep_list_privacy_jobs surfaced that
    as AEP_404, so "you have no privacy jobs" — the normal state of most
    tenants — looked like a broken tool and pushed agents into retrying. It now
    returns an empty list. Narrowly scoped: a 404 reading as not authorized or
    not provisioned is still an error, because that one needs a human.

  • Adobe's nested error detail was being thrown away entirely. The error-body
    sanitizer whitelisted top-level keys only, and Adobe nests the useful part
    of many errors under errors:

    {"errors":{"errorCode":404,"title":"Resource not found","detail":"Not able to find job data."}}

    errors was not on the whitelist, so the whole envelope was dropped and
    clients received {}. Every error of that shape arrived with no explanation,
    which is how a 404 meaning "nothing matched" became indistinguishable from a
    404 meaning "no such route". The sanitizer now recurses one level into
    errors, applying the same whitelist, so the detail survives while unlisted
    fields inside it are still dropped. Only errors, only one level — the
    whitelist stays authoritative.

Added

  • scripts/validate-tools-readonly.mjs — live validation that invokes the
    registered tool handlers rather than raw HTTP paths, so it reports what an
    agent would actually get. It refuses to call any tool not annotated
    readOnlyHint: true and force-clears AEP_ALLOW_MUTATIONS before building
    the client, so it cannot mutate. 18 of 18 runnable read-only tools pass; the
    other 9 need an id this sandbox cannot supply and are reported as
    no-fixture rather than guessed at.

    It exists because a path-level probe and the tools it was meant to validate
    had silently disagreed once before, and neither found the other's bug. Both
    fixes above came out of its first run.