A2A Conformance Test Findings

[kapil8811]

I've been working with A2A for a while now and recently tried to get agents built with different SDKs to actually talk to each other. Ran into some interesting friction that I think is worth sharing.
I ended up writing a small compliance testing tool ([a2a-overture] (https://github.com/kapil8811/a2a-overture)) to automate the checks and tested it against 7 agents from a2a-samples — 4 Python, 3.NET. Here's what came up.

What I found

1. supportedInterfaces - missing everywhere

Every single agent I tested failed this one. The v1.0 spec lists it as required on the Agent Card, but neither the Python SDK (v0.3.x) nor the .NET SDK (v0.2.x) populates it. Feels like this either needs to land in the SDKs or the spec should clarify whether it's truly required vs. recommended.

2. Agent Card URL mismatch

The .NET SDK serves the card at /.well-known/agent.json, while the spec says /.well-known/agent-card.json. I had to add a fallback in my client to try both paths. Minor thing, but it does break discovery if you only look at one.

3. Part kind discriminator ordering (.NET)

This was the trickiest one. The .NET System.Text.Json deserializer needs kind to be the first JSON property in a Part object for polymorphic deserialization to work. So {"text": "hello", "kind": "text"} gets rejected, but {"kind": "text", "text": "hello"} works fine.

The spec doesn't say anything about property ordering (and JSON technically doesn't guarantee order), but in practice it matters if you want to talk to .NET agents. Might be worth a note in the interop guidance, or maybe the .NET SDK could be more lenient here?

4. Method names & role values

This one's expected since the SDKs are on older versions — message/send vs SendMessage, user/agent vs ROLE_USER/ROLE_AGENT. Just documenting it here since it's something any cross-version client needs to handle.

Test results by agent

For context, here's how each agent did against a 23-test suite covering Agent Card, messaging, task lifecycle, streaming, push notifications, and auth:

The 2 failures that hit every agent are the supportedInterfaces ones (schema validation + required fields). The .NET agents also fail on ListTasks which isn't implemented in their SDK yet.

The tool

In case it's useful to anyone else — the thing I built to run these tests is on npm: https://www.npmjs.com/package/a2a-overture

npx a2a-overture certify http://localhost:8001

It auto-detects the protocol version, switches binding to JSONRPC for older agents, handles the method name and role value differences, and adds the kind discriminator with the right property order for .NET. Zero config needed.

Also has a GitHub Action if anyone wants to run it in CI, and can generate badges + HTML reports.

• npm: a2a-overture
• Source: github.com/kapil8811/a2a-overture (Apache-2.0)

Questions for the community

A few things I'm curious about:

1. Is supportedInterfaces actually required? Every SDK I tested omits it. Should this be filed as SDK issues, or should the spec relax this to optional?
2. Agent Card URL — is there a plan to align the .NET SDK to use agent-card.json? Or should clients be expected to try both?
3. JSON property ordering — the spec says nothing about it, but .NET needs kind first. Should the spec add an interop note about this, or is this something for the .NET SDK to fix on the deserialization side?
4. Would a shared conformance suite be useful? Something like what gRPC has for interop testing. I'm happy to contribute what I've built in whatever form makes sense — whether that's a PR to a2a-samples, a reference in the docs, or just keeping it as a community tool. Curious what others think.

Related: the Agent Card versioning discussion in #741 touches on some of the same cross-version compatibility issues. @brasseld's point about protocolVersion in the card is exactly what made auto-detection possible here.

Would love to hear if others have run into similar interop issues, or if there are tests, I should be adding. Happy to share more details on any of the findings.

[msaleme]

Really appreciate you publishing these findings. The supportedInterfaces gap and the streaming inconsistencies are exactly the kind of interoperability failures that become security vulnerabilities in production - an agent that silently falls back from streaming to polling can miss real-time alerts.

We've been building a security-focused A2A test harness that's complementary to your conformance tool. Where a2a-overture tests whether agents implement the spec correctly, ours tests whether they hold up under adversarial conditions - spoofed Agent Cards, injected task messages, push notification hijacking, cross-session data leakage.

Some of your findings map directly to security concerns:

1. Missing supportedInterfaces - If a client can't discover what an agent actually supports, it may send data through an unsupported channel. An attacker who knows the agent falls back silently can exploit the fallback path.

2. Non-standard JSON-RPC responses - We test for this in A2A-011 (protocol abuse via malformed requests). Agents that return 200 with error bodies instead of proper JSON-RPC errors are harder to build reliable error handling for, which means security failures get swallowed.

3. The SDK divergence problem - If Python and .NET SDKs behave differently on the same spec, you can't write a single security test that covers both. We've been dealing with this by testing at the wire protocol level (raw HTTP/JSON-RPC) rather than through any SDK.

Would be interesting to combine a2a-overture (conformance) with our harness (adversarial). Conformance first, then security. If an agent fails basic conformance, adversarial testing results are unreliable anyway.

Repo: https://github.com/msaleme/red-team-blue-team-agent-fabric (A2A harness: 12 tests)

[msaleme]

Thanks for writing this up. We keep hitting the same three friction points when we run the A2A harness (agent-security-harness module) against SDK-based agents:

1. supportedInterfaces is required by the spec. A2A-001 in the harness treats it as a hard failure because section 3.2 explicitly lists it under MUST fields. We ended up adding the following to both the Python and .NET templates so it is auto-populated:

"supportedInterfaces": [
  {
    "protocol": "json-rpc",
    "version": "1.0",
    "transport": "http"
  }
]

Until the SDKs emit that block, the compliance story will keep looking worse than it really is.

2. Agent Card path mismatch. Your observation about /.well-known/agent.json vs agent-card.json is exactly what A2A-001 flags for us. Our workaround has been to probe both paths in clients, but it would be better for the SDKs to converge on the spec default to keep discovery deterministic.

3. kind discriminator ordering. The .NET serializer quirk is real. For cross-interop tests we normalize outgoing parts so kind is emitted first before handing them to .NET clients. Documenting that requirement (or fixing the deserializer) would save everyone else the time we burned rediscovering it.

The rest of your findings line up with the later tests (A2A-004 unauthorized task access, A2A-008 skill injection). If you want another data point, pip install agent-security-harness and run agent-security test a2a --url http://localhost:8001. Happy to compare outputs if you are collecting more examples.

[kapil8811]

HI @msaleme, Thank you for the inputs.

I completely agree on the supported interfaces point, that's why I have added it as a community question.
For the second point - At least for now the dual probe is needed as a workaround till there is SDK convergence.
Regarding Kind it would be great to have it as interop-guidance within spec or a deserializer fix within .NET is what we have also recommended.

[kapil8811]

@muscariello, @mindpower. I hope you two are doing well. I want to take your input on the compliance findings stated above. The findings above come from running a 23-test compliance suite against 7 agents from a2a-samples. I hope we can go through the conformance test findings above and if see that the findings do hold true.
I'm happy to file individual issues on the SDK repos if that's the right process or contribute to a shared conformance suite if there's interest. Would appreciate your take on whether these are SDK bugs to fix or spec items to clarify.