-
Notifications
You must be signed in to change notification settings - Fork 3.3k
Performance regression testing using K6. Perfkit tool #9553
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## develop #9553 +/- ##
===========================================
+ Coverage 73.57% 82.32% +8.75%
===========================================
Files 409 464 +55
Lines 44882 47851 +2969
Branches 4056 4056
===========================================
+ Hits 33022 39394 +6372
+ Misses 11860 8457 -3403
🚀 New features to boost your workflow:
|
@orazvaliev you are invited to fix the remaining linting errors locally |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR introduces performance regression testing infrastructure using K6 and a custom Python CLI tool called "perfkit" for managing and comparing K6 performance baselines. The implementation includes Docker-based test execution, metrics monitoring, and baseline comparison capabilities.
Key changes:
- Adds perfkit Python package with CLI commands for running golden baseline tests and regression comparisons
- Implements K6 test scripts for performance testing with warmup and task regression scenarios
- Provides Docker Compose configuration for K6, Prometheus, and Grafana integration
Reviewed Changes
Copilot reviewed 23 out of 26 changed files in this pull request and generated 11 comments.
Show a summary per file
File | Description |
---|---|
tests/perf/setup.py | Package setup for perfkit CLI tool |
tests/perf/perfkit/*.py | Core perfkit modules for test execution, metrics handling, and baseline management |
tests/perf/scripts/ | K6 test scripts and API libraries for performance testing |
tests/perf/docker-compose*.yml | Docker configuration for performance testing stack |
tests/perf/README.md | Documentation for machine preparation and tool usage |
tests/package.json | Added K6 types dependency and linting configuration |
if metric_name in ALLOWED_DELTAS and stat_name in ALLOWED_DELTAS[metric_name]: | ||
allowed_delta = ALLOWED_DELTAS[metric_name][stat_name] | ||
else: | ||
# add_report_row(metric_stat, baseline_value, stat_value) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Commented-out code should be removed to improve code cleanliness.
# add_report_row(metric_stat, baseline_value, stat_value) |
Copilot uses AI. Check for mistakes.
@orazvaliev Food for thought: wouldn't it be nicer to just let the exceptions propagate to the entrypoint instead of |
|
@@ -0,0 +1,41 @@ | |||
// Copyright (C) CVAT.ai Corporation | |||
// | |||
// SPDX-License-Identifier: MIT |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
// SPDX-License-Identifier: MIT | |
// SPDX-License-Identifier: MIT | |
|
Motivation and context
This PR introduces performance regression testing infrastructure using K6 and a custom Python CLI tool called "perfkit" for managing and comparing K6 performance baselines. The implementation includes Docker-based test execution, metrics monitoring, and baseline comparison capabilities.
Key changes:
Adds perfkit Python package with CLI commands for running golden baseline tests and regression comparisons
Implements K6 test scripts for performance testing with warmup and task regression scenarios
Provides Docker Compose configuration for K6, Prometheus, and Grafana integration
How has this been tested?
Checklist
develop
branchLicense
Feel free to contact the maintainers if that's a concern.