Skip to content

Performance regression testing using K6. Perfkit tool #9553

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 50 commits into from
Aug 15, 2025

Conversation

orazvaliev
Copy link
Contributor

@orazvaliev orazvaliev commented Jun 20, 2025

Motivation and context

This PR introduces performance regression testing infrastructure using K6 and a custom Python CLI tool called "perfkit" for managing and comparing K6 performance baselines. The implementation includes Docker-based test execution, metrics monitoring, and baseline comparison capabilities.

Key changes:

Adds perfkit Python package with CLI commands for running golden baseline tests and regression comparisons
Implements K6 test scripts for performance testing with warmup and task regression scenarios
Provides Docker Compose configuration for K6, Prometheus, and Grafana integration

How has this been tested?

Checklist

  • I submit my changes into the develop branch
  • I have created a changelog fragment
  • I have updated the documentation accordingly
  • I have added tests to cover my changes
  • I have linked related issues (see GitHub docs)

License

  • I submit my code changes under the same MIT License that covers the project.
    Feel free to contact the maintainers if that's a concern.

@codecov-commenter
Copy link

codecov-commenter commented Jun 20, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.32%. Comparing base (0c3b8b6) to head (aeba08d).

Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #9553      +/-   ##
===========================================
+ Coverage    73.57%   82.32%   +8.75%     
===========================================
  Files          409      464      +55     
  Lines        44882    47851    +2969     
  Branches      4056     4056              
===========================================
+ Hits         33022    39394    +6372     
+ Misses       11860     8457    -3403     
Components Coverage Δ
cvat-ui 77.29% <ø> (+<0.01%) ⬆️
cvat-server 86.20% <ø> (+15.85%) ⬆️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@archibald1418
Copy link
Contributor

@orazvaliev you are invited to fix the remaining linting errors locally

@nmanovic nmanovic requested a review from Copilot August 6, 2025 20:59
Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR introduces performance regression testing infrastructure using K6 and a custom Python CLI tool called "perfkit" for managing and comparing K6 performance baselines. The implementation includes Docker-based test execution, metrics monitoring, and baseline comparison capabilities.

Key changes:

  • Adds perfkit Python package with CLI commands for running golden baseline tests and regression comparisons
  • Implements K6 test scripts for performance testing with warmup and task regression scenarios
  • Provides Docker Compose configuration for K6, Prometheus, and Grafana integration

Reviewed Changes

Copilot reviewed 23 out of 26 changed files in this pull request and generated 11 comments.

Show a summary per file
File Description
tests/perf/setup.py Package setup for perfkit CLI tool
tests/perf/perfkit/*.py Core perfkit modules for test execution, metrics handling, and baseline management
tests/perf/scripts/ K6 test scripts and API libraries for performance testing
tests/perf/docker-compose*.yml Docker configuration for performance testing stack
tests/perf/README.md Documentation for machine preparation and tool usage
tests/package.json Added K6 types dependency and linting configuration

if metric_name in ALLOWED_DELTAS and stat_name in ALLOWED_DELTAS[metric_name]:
allowed_delta = ALLOWED_DELTAS[metric_name][stat_name]
else:
# add_report_row(metric_stat, baseline_value, stat_value)
Copy link
Preview

Copilot AI Aug 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Commented-out code should be removed to improve code cleanliness.

Suggested change
# add_report_row(metric_stat, baseline_value, stat_value)

Copilot uses AI. Check for mistakes.

@archibald1418
Copy link
Contributor

archibald1418 commented Aug 8, 2025

@orazvaliev Food for thought: wouldn't it be nicer to just let the exceptions propagate to the entrypoint instead of exit()-ing ? What is your take on this?

@orazvaliev
Copy link
Contributor Author

@orazvaliev Food for thought: wouldn't it be nicer to just let the exceptions propagate to the entrypoint instead of exit()-ing ? What is your take on this?

exit() of Typer basically is a raise Exception but with fancy formatting.

@@ -0,0 +1,41 @@
// Copyright (C) CVAT.ai Corporation
//
// SPDX-License-Identifier: MIT
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// SPDX-License-Identifier: MIT
// SPDX-License-Identifier: MIT

Copy link

Quality Gate Failed Quality Gate failed

Failed conditions
1 Security Hotspot
5.4% Duplication on New Code (required ≤ 3%)
C Reliability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

@archibald1418 archibald1418 self-requested a review August 15, 2025 09:45
@orazvaliev orazvaliev merged commit aa994c2 into develop Aug 15, 2025
37 of 38 checks passed
@orazvaliev orazvaliev deleted the ro/regression-performance-tests branch August 15, 2025 09:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants