Project Proposal: OpenTelemetry Support Maturity Model

[kaspernissen]

This PR contains a project proposal for the OpenTelemetry Support Maturity Model - a descriptive framework for evaluating how cloud native projects support OpenTelemetry across multiple dimensions.

Draft model can be seen here: https://docs.google.com/document/d/1KvRtYqdSR1ii-SLV2wEv0MH-j9Mh5xA5u7f51kO6xpw/edit?usp=sharing

Note: similar to #3000, this proposal will be used to socialize the effort and identify additional collaborators. Several contributors expressed interest on #3247; specific roles, TC sponsor, and GC liaison will be confirmed during review.

Why now

• OpenTelemetry just reached CNCF Graduated status — "supports OpenTelemetry" is now a claim users and downstream projects increasingly take at face value, and a shared vocabulary is easier to establish now than to retrofit later.
• The draft framework has already been applied to real projects (Kubernetes ingress controllers: Traefik, Istio Gateway, Contour, Emissary, kgateway), and the exercise fed concrete refinements back into the draft.
• Conversations with maintainers (Dapr, kgateway, and others) have already led to upstream changes as a direct result of evaluations.
• The OpenTelemetry Governance Committee has asked that this go through the formal project process.

Future considerations (out of scope for this proposal)

Contributors on #3247 raised the idea of a follow-up conformance program in the spirit of Prometheus conformance. This proposal explicitly does not include conformance work - the descriptive model is the necessary first step, and a separate proposal would be the right place to take it further.

Related

• Draft proposal: OpenTelemetry Support Maturity Model for CNCF projects (and others) #3247 (originating issue)
• Draft maturity model document
• CNCF Platform Engineering Maturity Model(structural inspiration)
• Instrumentation Score and OpenTelemetry Ecosystem Explorer(complementary efforts)

[mhausenblas]

LGTM, what’s the next step? I’m happy to not only participate but also serve as co-lead if that’s desirable @kaspernissen

[kaspernissen]

Awesome, thanks @mhausenblas - really glad to have you on this. Just pushed the update with you as co-lead.

From here:

• Confirm TC sponsor and GC liaison (both still open, pointers welcome).
• Keep collecting feedback on dimensions and scope from the others listed in the proposal.
• Once this lands, set up the GitHub project board and SIG meeting cadence.

Glad to have a group forming around this.

[reyang]

Sort these alphabetically

[tedsuo]

Hi @kaspernissen. The thought behind this project means well, but I don't believe that it's appropriate for OpenTelemetry as a project to be scoring other projects in the CNCF based on their support for us. I discussed this with @caniszczyk and he agrees – the CNCF actively tries to avoid CNCF projects judging each other, as it's too easy for projects and maintainers to take offense at the grade we are giving them, even if our intentions are good.

But, just to be clear, I think the spirit of what you are trying to do is great! I have some suggestions. It feels like the goals of this project are to improve OpenTelemetry support in OSS and to allow projects to promote themselves based on their support. If we remove the scoring aspect, what would that look like?

One thing that would be very helpful would be implementation/integration guides on the website, based on the type of OSS project.

• If your project is a library, how integrate native instrumentation. We're still working on our tooling to make this easier, but this will become a hot topic in the next year as that becomes available.
• If you project is a database or service, how to write federated semantic conventions and use declarative config to emit OTLP. We would love to see more projects do this and guidance on best practices would help those projects quite a bit.
• If your project is an observability analysis tool, how best to consume OTLP and semantic conventions.

Down the road, it's possible that we could work with the CNCF to create certification programs for some of these classes of support. Our highest priority on that front is a certification program for the OpenTelemetry Collector; we'll definitely be doing that. But I could see the value in badges or certifications that cover other forms of OpenTelemetry support, like native instrumentation.

Even before we have badges to hand out, we could give space for projects to promote their OpenTelemetry support on the website and in places like the registry (and its successor the ecosystem-explorer).

Again, apologies, because I can see how excited you are for this. But I hope that feedback makes sense. Let me know what you think about these proposals for how to re-implement the goals you are after. Or, if I've misunderstood the goals, let me know what other goals you are trying to achieve.

[tigrannajaryan]

I like the intent of this project, but I have concerns about the specifics:

• The evaluation criteria seems to be subjective. I think we should aim for objective, automatic evaluations where possible. For example if we are evaluating conformance of an OTLP sender there must be a tool that can receive OTLP payloads and assess its conformance with OTLP spec and Otel semconv. This needs to be an automated and objective process.
• We are suggesting to evaluate and label other projects and vendors. I think this can get messy if whoever we are labeling disagrees with our opinion.

I suggest that we make the following changes to this proposal:

• Ensure evaluations and labeling is automated as much as possible.
• Limit the scope to evaluating our own Otel implementations only, in collaboration with Otel maintainers. We can publish conformance of Otel SDKs and Otel instrumentation. Projects outside Otel may choose to voluntarily use the same evaluation tools. We can collaborate and help them do it, but we shouldn't make these evaluations ourselves.
• Make sure you secure a TC sponsorship at the guiding or leading level. This is a cross-functional project impacting many SIGs in Otel and it will benefit from such sponsorship.

We don't want to be in the business of subjective, opinionated labeling of third parties. No matter how well intended we are there is good chance it will end up creating arguments and conflict. I suggest that we start small, within Otel and move carefully. If it works well within Otel and is proven to be a good, objective measure, we can consider extending the scope.

[tigrannajaryan]

Looks like @tedsuo and I cross-posted independently, but we mostly agree about concerning areas.

[salaboy]

@tedsuo @tigrannajaryan I am in the same page with you folks, but having talked with @kaspernissen I would love to reiterate this is not about scoring, it is about evaluating support and helping CNCF projects to grow. And in no way this is about vendors.

I've been working a bit on the automation for this (as @tigrannajaryan mentioned "automatic evaluations where possible"), because having a framework to define what these evaluations are so projects can run them themselves is a must for an initiative like this. These evaluations then can be used for conformance too.

As pointed out, libraries, Kubernetes controllers, sidecars, databases, message brokers all require different approaches for performing the evaluations, so what I would love to see this group do is to refine what those evaluations would be.

The dimensions included in this proposal allows us to categorize the evaluation results so if a project is interested in improving one of its dimensions it can drill down to find specific suggestions.

I hope this makes sense..

[tedsuo]

@tedsuo @tigrannajaryan I am in the same page with you folks, but having talked with @kaspernissen I would love to reiterate this is not about scoring, it is about evaluating support and helping CNCF projects to grow. And in no way this is about vendors.

C'mon, man. This proposal literally states that OpenTelemetry should assign maturity levels to other CNCF projects. That is the main thrust of this proposal. It's true that this document does start by saying that it is not an evaluation, but then it immediately proposes that we grade projects on a scale of 0-3 based on their support for OpenTelemetry, with the main deliverable being a visualization for comparison purposes. How is that not an industry analyst report?

As pointed out, libraries, Kubernetes controllers, sidecars, databases, message brokers all require different approaches for performing the evaluations, so what I would love to see this group do is to refine what those evaluations would be.

I completely agree with your point that all of these types of projects require different approaches. But precisely to that point, I don't see how a generic, high-level analysis framework would be very helpful to engineers that want to actually improve their telemetry.

These kinds of frameworks and reports can definitely be helpful to CTOs and other decision makers at end user companies that are trying to decide which technologies to adopt. But that is also the primary reason why OpenTelemetry as a project shouldn't be issuing them. Saying "please don't use our comparative analysis to make judgements" wouldn't change their utility, and other projects will see through that.

If the CNCF wants to develop reports on the state of observability within the CNCF, and use a framework like this, I don't think that anyone from this project would have any objections. But as a CNCF project, we've been directly asked by CNCF leadership to not get into this business, as part of a general guideline that CNCF projects should not evaluate each other.

The dimensions included in this proposal allows us to categorize the evaluation results so if a project is interested in improving one of its dimensions it can drill down to find specific suggestions.

If the ultimate value in this project lies in making these suggestions and providing better tools to implement them, how about we start there? A high-level comparative framework is not a required step for publishing implementation guides, and there's definitely a lot of excitement for both improved documentation and better tooling to support these efforts. We would absolutely appreciate any efforts on these fronts.

• Guides on how to use instrumentation score.
• Guides on how to add native instrumentation to libraries. There's a lot of nuance here.
• Guides on how to instrument services. Best practices for exposing configuration, writing semantic conventions.
• Better tests, SemConv tooling and AI skills that support these activities. There's already great work being done here but it could use more hands.

What do you think about organizing an effort around those pieces? Are there any specific parts that you are particularly interested in? I know I know, "the maturity model" is the specific part you're interest in! But help us understand something more concrete. Is there a particular kind of project you are most interested in helping? Libraries? Kubernetes? Message brokers? Can you present a run through on a specific project you're trying to engage with, where making progress is difficult?

[kaspernissen]

@tedsuo fair pushback. Let me clarify where I actually am on this, because I think we're closer than the thread suggests.

The proposal is a draft, and was put forward to start exactly this kind of discussion. The dimensions and levels were one way to talk about maturity on a range. You're right that the doc as written specifies 0-3 levels and a comparative visualization. I'm fully open to dropping both the levels and the comparative visualization. They were a starting point for the conversation, not the point of the proposal.

The point is the underlying observation: "supports OpenTelemetry" is becoming a binary claim that hides enormous variance across projects, and end users absorb that cost when they migrate their tooling. As OTel becomes the foundation for observability in many organizations, that variance is going to matter more, not less. I think we agree on this. The question is what to do about it.

On your concrete suggestions, I'm in. Implementation guides per project type, better tooling, support for instrumentation score, native instrumentation guidance for libraries, semconv guidance for services. Those are the kind of deliverables that would actually help projects mature their OTel support, and they're concrete in a way a comparative framework isn't.

The framing I'd add: applied guides. We work with maintainers to figure out where the gaps are, then write guides grounded in those gaps rather than in the abstract. The dimensions from the original proposal become internal scaffolding for what the guides need to cover, not a public deliverable. That's effectively the pattern the five gateway evaluations followed: structured look at where the gaps are, maintainer conversations, upstream changes, and then a writeup that captures what good looks like for that project type.

There's also room for a deterministic evaluation tool alongside the guides. Something maintainers can run against their project's actual telemetry output and get back a factual report of what's emitted, what semconv it follows, and where the gaps are. No score, just diagnostic information they can act on. That's the direction @salaboy's automation work is already heading, and it complements the instrumentation score effort. It also gives the applied-guides pattern a way to scale past evaluations done by hand.

On your specific question around project interest, I'm interested in helping where I can. I started with gateways, since that was a place where I saw visible variance in the support provided. That's really what sparked the idea for this draft proposal.

On the CNCF leadership constraint, understood. If the framework dimensions turn out to be useful as input to a CNCF-level effort someday, that's a separate conversation. The OTel-side scope should be the implementation work.

If that's roughly what you had in mind, I'd be glad to redraft the proposal along those lines and start the discussion in the group on which project types to prioritize after gateways.

[tedsuo]

Thanks @kaspernissen for being so flexible!

Regarding the "supports OpenTelemetry" concern, I completely agree with you. Unless we start providing more guidance, people will start making up their own definitions. But even beyond the definitions, there's nuance to OpenTelemetry support that people will miss without guidance, leading to sub-optimal implementations. It feels like we're all in alignment with solving those problems, and it's a good time to start thinking about them.

I can think of three general classes of support that all require different guidance. Let me know if you see others.

• OpenTelemetry Collector Distributions
• Native instrumentation for libraries
• Native instrumentation for data services

In the first case, the OpenTelemetry Collector, we've seen some genuine confusion, with some companies concluding that "OpenTelemetry Collector" is a generic term. We added a definition of a Collector Distribution here to make it more clear that this is a trademarked term meant to refer to a specific codebase, with some wiggle room for forking and repackaging (but not infinite room). There's been interest in taking this another step now that we have graduated, and providing a Collector Certification. The CNCF only provides that kind of support for graduated projects, which is why we've been waiting.

For the other two cases – library and service instrumentation – I like your idea of "applied guides" as well as providing tooling that can help make it easier, not to mention verify correctness. In the roadmap we are working on, the SemConv Tooling SIG is owning this workstream, with a focus on library instrumentation. But even without tooling, guides that describe common pitfalls and best practices would be helpful. I can see how your framework was aiming to address that, especially for services. If we turn this into guides I think we have agreement on a way to move forwards. Writing the guides will also show us the gaps in our toolchain for supporting this.

All that said... I'm not sure we need a new SIG for this? The Comms SIG would be a great place to start the work on guides. @svrnm what do you think? The SemConv Tooling SIG like I mentioned is handling the tooling. And the Collector SIG along with the GC/TC can handle the Collector certification once the CNCF is able to make those resources available to us.

[graz-dev]

Hi everyone, thanks @kaspernissen for kicking off this initiative, and a special thanks to @salaboy for pointing me to it.

I’ve been reading through the thread, and I strongly agree with the concerns raised by @tigrannajaryan and @tedsuo regarding the risks of OpenTelemetry acting as a "rating agency" for third-party projects. Providing a top-down score could indeed lead to friction.

However, I think the core need remains: CNCF projects and end-users need guidance on what "OTel support" actually means and how to improve it. To bridge this gap, I’d love to propose an alternative approach and offer my availability to contribute to it.

Instead of a centralized scoring/reporting system, we could reframe the Maturity Model as an Opt-in Self-Assessment Framework.

Here is what I’m proposing:

• The model shouldn't be a report card we publish about others. It should be a framework that projects can voluntarily use to test themselves against OTel standards. They get a score/report, and they decide whether to publish it (e.g., via a badge in their README).
• Instead of using the CNCF landscape mapping by functional areas, we could categorize projects by technical footprint (e.g., Library, Middleware, Database, Infrastructure Agent). The maturity levels (0 to 3) would then have specific, actionable criteria for each technical category.
• We could eventually build a tool that projects can use to evaluate their initial state and track progress. Rather than just spitting out a score, the main goal of this tool would be to generate a concrete "Missing Points & Next Steps" report to actively guide developers.

This approach keeps OpenTelemetry in the role of an enabler rather than a judge, while still providing the clarity the ecosystem needs and that this initiative aims to provide. It could also be worth considering involving one of the CNCF TAGs in this initiative to help us collaborate and reach other projects.

I'd be more than happy to contribute to this initiative. Let me know what you think!

[svrnm]

I'm not sure we need a new SIG for this? The Comms SIG would be a great place to start the work on guides. @svrnm what do you think?

Would like to have other @open-telemetry/docs-maintainers speak to this as well, here are my thoughts: we had the idea of growing the instructions (aka applied guides) for building components (for collector, but also instrumentation libararies and ESPECIALLY native instrumantation for libraries and applications) into the documentation for a long long time. But like many other things in docs it never grew beyond a certain base level because nobody owned that part of the docs. Our collector docs are right now a prime example for healthy docs, since we have a well functioning co-ownership between the SIG maintainers and @tiffany76 from our end. A similar success story is the ecosystem explorer which is part of Comms that is owned by @jaydeluca. Both projects are also so successful because they grow organically, so instead of doing one big splash we started small and build things incrementally. If we have 2-3 people who are really enthusiastic about making this happen, I am optimistic about it working out, but again deferring to other Comms maintainers.

[chalin]

I support and agree with what @svrnm has shared.

[cijothomas]

➕ 💯 on the underlying problem — "supports OpenTelemetry" has become a binary claim that can be used to mean anything!, and a more formal vocabulary is better. I'd love to see things like "this library instruments via OTel APIs and uses OTel semconv" or "this app honors OTEL_EXPORTER_OTLP_ENDPOINT and can ship to any OTLP backend."

One observation about greenfield bias. Most of the seven dimensions (integration surface, semconv currency, OTEL_* config, multi-signal) nautrally favor projects that had OTLP and current otel conventions as day-1 design choices. Established projects that didn't have OTel when they started will naturally score lower.

Also, if we can consider "observable behavior" - like does OTLP come out? is semconv followed? are OTEL_* working?, rather than implementation choices (using OTel API internally or other one) that'd be helpful too— it stops penalizing incumbents who reach the same outcome via a different path.

Hopefully this can be addressed before the proposal is accepted