Skip to content

Add 2021-05-03-introducing-prometheus-conformance-program.md #1941

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
bwplotka merged 2 commits into from
May 3, 2021
Merged

Add 2021-05-03-introducing-prometheus-conformance-program.md #1941

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
a3a7c96
Add 2021-05-03-introducing-prometheus-conformance-program.md
RichiH Apr 30, 2021
2bdbf41
Julius feedback
RichiH May 2, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions content/blog/2021-05-03-introducing-prometheus-conformance-program.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
---
title: Introducing the Prometheus Conformance Program
created_at: 2021-05-03
kind: article
author_name: Richard "RichiH" Hartmann
---

Prometheus is the standard for metric monitoring in the cloud native space and beyond. To ensure interoperability, to protect users from suprises, and to enable more parallel innovation, the Prometheus project is introducing the [Prometheus Conformance Program](https://github.com/cncf/prometheus-conformance) with the help of [CNCF](https://www.cncf.io/) to certify component compliance and Prometheus compatibility.

The CNCF Governing Board is expected to formally review and approve the program during their next meeting. We invite the wider community to help improve our tests in this ramp-up phase.

With the help of our [extensive and expanding test suite](https://github.com/prometheus/compliance), projects and vendors can determine the compliance to our specifications and compatibility within the Prometheus ecosystem.

At launch, we are offering compliance tests for three components:
* PromQL (needs manual interpretation, somewhat complete)
* Remote Read-Write (fully automated, WIP)
* OpenMetrics (partially automatic, somewhat complete, will need questionnaire)

We plan to add more components. Tests for Prometheus Remote Read or our data storage/TSDB are likely as next additions. We explicitly invite everyone to extend and improve existing tests, and to submit new ones.

The Prometheus Conformance Program works as follows:

For every component, there will be a mark "foo YYYY-MM compliant", e.g. "OpenMetrics 2021-05 compliant", "PromQL 2021-05 compliant", and "Prometheus Remote Write 2021-05 compliant". Any project or vendor can submit their compliance documentation. Upon reaching 100%, the mark will be granted.

For any complete software, there will be a mark "Prometheus x.y compatible", e.g. "Prometheus 2.26 compatible". Relevant component compliance scores are multiplied. Upon reaching 100%, the mark will be granted.
Copy link
Member

@juliusv juliusv May 1, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Upon reaching 100%, the mark will be granted."

Note that query tweaks in my PromQL tests (https://promlabs.com/promql-compliance-tests) have been treated separately from the % score so far, and the current 100% score wouldn't always be a true 100% when we factor those query tweaks in (which we have to, to be fully compliant). E.g. Grafana Cloud would be failing then due to their timestamp alignment, but this is probably also how it should be unless Prometheus Team wants to grant certain exemptions.

Copy link
Member Author

@RichiH RichiH May 2, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Our thinking in OM is to follow the system of MUST, SHOULD, MAY and we chose the same language in the Remote Write spec. This maps 1:1 to if tests are fail, warning, or info level. If timestamp alignment MUST NOT happen, then Grafana Cloud does not pass until it stops aligning timestamps, agreed.

This is why I like normative language: It makes gaps and disagreements visible.


As an example, the Prometheus Agent supports both OpenMetrics and Prometheus Remote Write, but not PromQL. As such, only compliance scores for OpenMetrics and Prometheus Remote Write are multiplied.

Both compliant and compatible marks are valid for 2 minor releases or 12 weeks, whichever is longer.