-
Notifications
You must be signed in to change notification settings - Fork 1.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add ddev test
option to verify support of new metrics
#6141
Conversation
Codecov Report
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🥇
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice! WDYTA calling the marker metric_validation
?
def pytest_collection_modifyitems(config, items): | ||
if config.getoption("--run-check-metrics"): | ||
# --run-check-metrics given in cli: do not skip slow tests | ||
return | ||
skip_check_metrics = pytest.mark.skip(reason="need --run-check-metrics option to run") | ||
for item in items: | ||
if "check_metrics" in item.keywords: | ||
item.add_marker(skip_check_metrics) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we explain what this does in comments?
Great idea! I agree the naming should be a little more precise — eg the word "latest" should be present. (IMO WDYT of [pytest]
marks =
latest_metrics: Mark test as checking for latest available metrics @pytest.mark.latest_metrics
def check_metrics_are_up_to_date():
... $ ddev test --latest-metrics |
Agreed on the naming, I struggled with that one and wasn't totally happy with the one I picked either. I like |
Also, pro-tip which was helpful here, you can get explicit skip reasons by passing
|
ddev test
option to verify support of new metrics
Co-Authored-By: Ofek Lev <ofekmeister@gmail.com>
What does this PR do?
Adds a new option to
ddev test
called--check-metrics
.When the flag is present, only tests marked by
@pytest.mark.check_metrics
will run.Motivation
The original motivation for these tests were to ensure that we kept ahead of new developments for the integrations. However, it's turned out that development on these areas progresses a little more frequently than expected, and running these tests as part of our usual CI runs can make things noisy and require fixes more quickly than really needed.
Adding this option will move the checks out of regular CI execution, and a separate PR will then be setup to schedule them for periodic execution so we still maintain the insight when new metrics get added and can schedule a fix accordingly.
Review checklist (to be filled by reviewers)
changelog/
andintegration/
labels attached