Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Incorporate an OTel-wide maturity matrix #290

Merged
merged 6 commits into from
Feb 11, 2020
Merged
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
161 changes: 161 additions & 0 deletions maturity-matrix.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,161 @@
# OpenTelemetry is a complex project with many moving parts. Thankfully, we
# strive to keep those parts loosely coupled where we can (this has been a goal
# since the very announcement of OpenTelemetry, per this blog post:
# https://medium.com/opentracing/merging-opentracing-and-opencensus-f0fe9c7ca6f0)
#
# This file describes the structure of the project and its sub-components,
# definitions of API and stability maturity levels, and finally records the
# current maturity of each component. We maintain this file as YAML in order to
# make it easier to automate scripts (e.g., for the website) that present the
# data to OpenTelemetry end-users.


###############################################################################
# MATURITY LEVELS
#
# OpenTelemetry components advertise their own maturity along two axes:
# - PRODUCTION MATURITY: How reliable is this component from a
# production-readiness standpoint?
# - API MATURITY: How stable is the component's API from a backwards- and
# forwards-compatibility standpoint)?
#
# In some cases, for instance, OpenTelemetry components may be immature from an
# API maturity standpoint, yet be nearly 100% safe from a production standpoint
# – and vice versa. Similarly, end-users may have more tolerance for lost
# development time or software stability depending on their situation, so we
# strive to separate self-reported maturity along these two axes.
#
# Needless to say, representations here are best-effort, and there is no
# substitute for a mature release process. I.e., don't blindly deploy "mature"
# OpenTelemetry components (or any software, for that matter!).
productionMaturityLevels:
# "unknown" maturity is just what it purports to be. It's reasonable to
# assume "unstable", but we separate "unknown" and "unstable" to distinguish
# between a lack of evidence about production maturity and actual evidence of
# immaturity.
- unknown
# "unstable" components are not recommended for production workloads. They
# may crash the process, introduce performance artifacts, or have known
# security issues.
- unstable
# "beta" components have been used successfully in production workloads at
# scale, but either due to the uncertainty introduced by active development
# or a small number of production deployments, they should still be used
# cautiously in new deployments. They do not have known security issues.
- beta
lizthegrey marked this conversation as resolved.
Show resolved Hide resolved
# "stable" components are, to the best of the author's knowledge, safe for
# typical production use cases.
- stable

# All OpenTelemetry APIs follow semver conventions (i.e., after v1.x,
lizthegrey marked this conversation as resolved.
Show resolved Hide resolved
# backwards-incompatible changes should bump the major version number).
apiMaturityLevels:
# "unimplemented" APIs do not exist yet for the component/language.
lizthegrey marked this conversation as resolved.
Show resolved Hide resolved
- unimplemented
lizthegrey marked this conversation as resolved.
Show resolved Hide resolved
# "alpha" maturity APIs can change in incompatible ways at any time.
- alpha
# "beta" maturity APIs should not introduce backwards-incompatible changes
# more than once every three months; and when those changes are introduced, the
# authors will make a best effort to provide compatibility bridges.
- beta
# "stable" maturity APIs should not introduce backwards-incompatible changes
# more than once every twelve months, and will make every effort to provide
# compatibility bridges if at all possible.
- stable


###############################################################################
# OPENTELEMETRY COMPONENTS
#
# Some OpenTelemetry components run within end-user applications and thus have
# a footprint across many languages. Naturally, the production and API maturity
# of these components varies from language to language. Other components run as
# infrastructure in their own containers or VMs and thus have a single maturity
# level regardless of the end-user's programming language.
#
# The maturity – or even the existence – of the "perLanguageComponents" varies
# from language to language. These are components that integrate into the
# actual end-user application runtime in production. See "languages" for the
# actual maturity levels.
perLanguageComponents:
- traceAPI
- metricsAPI
- contextAPI
- traceSDK
- metricsSDK
- contextSDK
- automaticInstrumentation
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should the integrations and exporters be also tracked? Zipking and Jaeger exporters, integrations like SpringBoot, etc.?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO we should focus solely on SDK/API for trace/metric/context here and allow exporters and integrations to simply reference their stability or maturity against the lower level components.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some integrations and exporters are the baseline for the stability as you cannot use API/SDK without ability to export it. Similar for integrations. I'm not sure how automaticInstrumentation here is different than integrations?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, that's a good point, although I don't know how to square it. Automatic instrumentation does seem to require integrations.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, I think that end-users expect at least some of these (OT exporter, Jaeger exporter, Prometheus exporter, basic HTTP integrations) to be tracked

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not want to suggest that we try to maintain a full compatibility matrix with framework-level things like Spring. The "registry" seems like the right place to track that sort of thing.

By "automaticInstrumentation", I meant specifically the idea of "no-code" (literally) installation of the OpenTelemetry instrumentation. This is an important requirement for many brownfield services that are no longer actively maintained but nevertheless participate in transactions. As such, I would not like to extend the definition to "1-line code changes" that nevertheless require recompilation.

@SergeyKanzhelev re your comment:

some integrations and exporters are the baseline for the stability

I disagree – they may be the baseline for "usefulness", but not for stability.

Given the possibly large diversity of exporters in the future, I would personally prefer that we:

  1. Adopt the API and production maturity levels for exporters, and
  2. Track exporter maturity via the registry

Remember that we're not debating how this information is displayed on a website at the moment, we're just debating what should be considered "internal" to OpenTelemetry rather than a plugin/etc. The website can absolutely bring in the maturity levels for popular exporters in designated languages.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to avoid situation when OT is only usable with the specific vendor exporter. This is why I think we should declare some baseline exporters here. It may be minimum functionality and not scalable, but must be reliable and stable.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder about the 'diversity of exporters' -- Yuri could probably weigh in here, but I suspect that many "vendors", both OSS and commercial, would like to simply accept OTel data format natively and thus not require an exporter at all.

With that in mind, this feels like a transient issue; Yes, there'll be maturity levels for exporters but over time those exporters simply won't matter since your backend will accept the data with no translation.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suspect that many "vendors", both OSS and commercial, would like to simply accept OTel data format natively and thus not require an exporter at all.

Not immediately. It may take a while

With that in mind, this feels like a transient issue; Yes, there'll be maturity levels for exporters but over time those exporters simply won't matter since your backend will accept the data with no translation.

Yes, but if we intent to track progress towards 1.0 I'd suggest to include tracking of this transient issue

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@SergeyKanzhelev I'm going to continue to push back on this... if I want to use Jaeger's exporter and it's the only thing that's mature for my given language, IMO that's fine. Now, whether the SIGs would feel comfortable marking their APIs as "stable" (or even "beta") with zero or one exporter implemented is a different question.


# The maturity of the "standaloneComponents" is independent of any particular
# end-user programming language.
standaloneComponents:
collector:
api: beta
SergeyKanzhelev marked this conversation as resolved.
Show resolved Hide resolved
production: beta
operator:
api: alpha
production: beta

# For each language, we either specify the perLanguageComponents maturity
# inline, or we specify a single "url" that points to another YAML file with a
# top-level perLanguageComponents key (presumably in the OpenTelemetry language
# repository). E.g., something like this:
#
# languages:
# logo:
# url: "https://raw.githubusercontent.com/open-telemetry/opentelemetry-logo/master/maturity.yaml"
languages:
java:
traceAPI:
api: alpha
production: beta
metricsAPI:
api: alpha
production: beta
contextAPI:
api: alpha
production: beta
traceSDK:
api: alpha
production: unstable
metricsSDK:
api: alpha
production: unstable
contextSDK:
api: alpha
production: unstable
automaticInstrumentation:
api: beta
production: unstable

# TODO once we've converged on the schema for all of this
js: ""
dotnet: ""
ruby: ""
erlang: ""
python: ""
golang:
traceAPI:
api: beta
production: beta
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd suggest to include the link to the issues query on GitHub showing what's left for this area. It may be an optional attribute

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a little hesitant to do this since in the early stages, that list will have lots of errors of omission and could lead to a false sense of security...

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we declare it as optional so I can use it for .NET?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO it'd be good to have a URL field in the schema that points to the repo.

metricsAPI:
api: alpha
production: beta
contextAPI:
api: unimplemented
production: unknown
traceSDK:
api: beta
production: beta
metricsSDK:
api: alpha
production: unstable
contextSDK:
api: unimplemented
production: unknown
automaticInstrumentation:
api: unimplemented
SergeyKanzhelev marked this conversation as resolved.
Show resolved Hide resolved
production: unknown
php: ""
cpp: ""
rust: ""