[follow-up] Create SDK tiering system

[dsp-ant]

During the NYC maintainer meetup, we discussed the need for SDK harmonization. We want SDKs to be clear about what features they support. @nickcoai had the idea to create an SDK tiering system, which the core maintainers liked. The goal here is to figure out a set of requirements for each tier that an SDK must support. For example, we could think of Tier 1 as SDKs that support the full MCP specification for both client and servers within 4 weeks of the release of a new MCP version, Tier 2 of SDKs that support the full MCP specification for both client and servers within 6 months, and Tier 3 that make no guarantees of what they support, but at least support basic primitives, notifications and all transports for servers.

This is just an example. The task is to solicit feedback from SDK maintainers and build such an tiering system and, write a quick SEP for core maintainers.

[pja-ant]

Some thoughts:

• I like that we're thinking about how to bring clarity to SDK implementation completeness.
• A risk a see with this is that "support the full spec" is a bit of a grey area. Do bugs count as lack of support? If an SDK throws together a bug-ridden but works-on-the-happy-path implementation of a new spec feature, is that enough to say that it has "support"?
• Couple the above with the harsh penalty for lack of support (your SDK gets dropped from Tier 1 to Tier 2, or Tier 2 to Tier 3) and I'd imagine this could lead to some difficult conversations ("sorry, but we're dropping your SDK from tier 1 to tier 2 because it doesn't handle rare edge-case outlined in spec clause 3.11.5(c)")
• How do we actually evaluate this?

Maybe it is sufficient to just open the SEP-implementation tasks for each SDK, and then core maintainers review the implementation, and decide if those tasks get closed. Task closed = support. Then it basically amounts to "close the tasks in time". Ignore bugs as long as core maintainers are happy that the impl is a good faith effort to add support. Could have some edge cases, but probably good enough.

Alternative idea: rather than tiers, why not just list the latest version(s) of the spec that each SDK has implemented? That's probably what users care about more.

[LucaButBoring]

How do we actually evaluate this?

Alternative idea: rather than tiers, why not just list the latest version(s) of the spec that each SDK has implemented? That's probably what users care about more.

Related: #1309, #1400

[cliffhall]

I don't know how tiers help developers choose an SDK. Seems more like a shame factor to motivate SDK maintainers, which feels a little icky, TBH.

Alternative idea: rather than tiers, why not just list the latest version(s) of the spec that each SDK has implemented? That's probably what users care about more.

But if an SDK doesn't fully support a version yet, it's a little confusing as to how much it supports and whether it would be adequate for the developer's needs. e.g, if a server developer isn't using elicitation, they wouldn't care that the SDK doesn't support elicitation, but all they could intuit from a "latest supported version" is that it doesn't fully support a particular version.

Another alternative: Feature matrix. SDKs are columns, features are rows. Sorted so the SDKs with the most boxes checked are on the left. Separate feature matrix page for each spec version. Some set of core features (to be amended with successive each version) are required to make the official list.

[PederHP]

I don't think tiers are inherently bad as long as they're tied to a specific property. The word 'tier' is perhaps the problem - it sounds like a normative statement about quality (ie 'tier lists' from popular culture), where it is actually a property related to feature support. I am not sure what the better word is - I am not a native English speaker.

I also think it is worth considering quality in terms of stability, scalability, developer experience, etc. But for that I'd suggest that SDK maintainers propose a rating based on their own perceptions, which is then considered and approved or not. It is valuable for developers to know what the ambition level is, and I have faith that SDK maintainers can be honest about this, so there won't be much need for overruling their proposed rating.

I've seen developers ask if the C# SDK is in a worse state than Python or TypeScript ones because of being marked as prerelease. I really don't think it is. It's a really solid SDK with excellent developer experience and an LLM-library integration (via MEAI) that I think no other SDK has. But due to .NET conventions it is a prerelease, which makes some developers think it is less ready for production use than the other SDKs.

So I am greatly in favor of an official stamp of approval. But I would consider it more in terms of properties than tiers - "enterprise ready", "feature complete", etc. and then mark each individual property.

When a new version is released that's a new property in the matrix, and once it's supported, then it's marked. Maybe with an allowance for SDK maintainers to put in a target date if they're not ready.

[chr-hertel]

I like the idea of having a matrix, which makes it super easy to check what's in what's not. also from the point of what people need for their implementation. having time based commitments feels a bit tricky, but maybe that's only my experience of software dev 🙈 😂 if the intention is to create some kind of pressure, that would work - but I'd favor having the matrix - that's sufficient as motivation for me at least - and creates transparency.

another idea could be to have a spec compliance test suite - that would also make it easier to actually tick a box.

[scottslewis]

Question: Has there been any discussion around a 'release cadence' for spec and sdks? For example: specification project produces updates every 6/x months, with downstream sdks expected/required major or minor releases following spec, with some number of > 0 of bug fix releases in between...perhaps only for some tiers?

Why? Consumers (mcp server and client devs)...benefit from predictable release schedules, and following semantic versioning conventions (major, minor, micro) for those releases. This makes it much easier for devs to predict how sdk changes are going to affect their downstream projects or products...and helps to build trust with the dev community.

[chr-hertel]

What do you think of some kind of conformance testsuite, ending up with some kind of score per SDK like "90% spec compliant" (as badge?).
Could be extended centrally with spec changes with a fast feedback loop on all SDKs.