Add ddev test option to verify support of new metrics

[mgarabed]

What does this PR do?

Adds a new option to ddev test called --check-metrics.

When the flag is present, only tests marked by @pytest.mark.check_metrics will run.

Motivation

The original motivation for these tests were to ensure that we kept ahead of new developments for the integrations. However, it's turned out that development on these areas progresses a little more frequently than expected, and running these tests as part of our usual CI runs can make things noisy and require fixes more quickly than really needed.

Adding this option will move the checks out of regular CI execution, and a separate PR will then be setup to schedule them for periodic execution so we still maintain the insight when new metrics get added and can schedule a fix accordingly.

Review checklist (to be filled by reviewers)

• Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
• PR title must be written as a CHANGELOG entry (see why)
• Files changes must correspond to the primary purpose of the PR as described in the title (small unrelated changes should have their own PR)
• PR must have changelog/ and integration/ labels attached

[codecov]

Codecov Report

Merging #6141 into master will decrease coverage by 2.68%.
The diff coverage is 100.00%.

Impacted Files	Coverage Δ
airflow/tests/test_check_metrics_up_to_date.py	41.17% <100.00%> (-58.83%)	⬇️
clickhouse/tests/test_unit.py	88.23% <100.00%> (ø)
datadog_checks_dev/tests/tooling/test_utils.py	100.00% <0.00%> (ø)
...ts/tooling/configuration/consumers/test_example.py	100.00% <0.00%> (ø)
kubelet/datadog_checks/kubelet/kubelet.py
nagios/tests/conftest.py
druid/datadog_checks/druid/__about__.py
cassandra/datadog_checks/cassandra/__about__.py
ibm_db2/tests/test_integration_e2e.py
... and 689 more

[ChristineTChen]

🥇

[ofek]

Nice! WDYTA calling the marker metric_validation?

[ofek]

Can we explain what this does in comments?

[florimondmanca]

Great idea!

I agree the naming should be a little more precise — eg the word "latest" should be present. (IMO metrics_check and metric_validation make me think of eg "this test verifies check yields metrics in line with metadata.csv", not that it tries to see if we're up to date with the latest available metrics.)

WDYT of latest_metrics?

[pytest]
marks =
    latest_metrics: Mark test as checking for latest available metrics

@pytest.mark.latest_metrics
def check_metrics_are_up_to_date():
    ...

$ ddev test --latest-metrics

[mgarabed]

Agreed on the naming, I struggled with that one and wasn't totally happy with the one I picked either.

I like latest-metrics - we have a bunch of validation checks already, and while this performs a similar "validation" to the ddev sub-commands if we used the same name for the tests it could easy to confuse what's actually happening.

[mgarabed]

Also, pro-tip which was helpful here, you can get explicit skip reasons by passing -rs to pytest, so for our ddev command:

ddev test -k test_latest_metrics_supported -pa -rs clickhouse:py38-latest