Skip to content

Conversation

@vkarak
Copy link
Contributor

@vkarak vkarak commented Sep 2, 2025

This PR adds two significant performance improvements for DB queries involving session information.

  1. It adds an index on the session's start time in the sessions table. This significantly improves the session queries as well as test case queries that use session filtering criteria.
  2. The --describe-stored-sessions option performance is considerably improved as the unnecessary decoding and encoding of the results is removed. Since the result is already going to be returned as JSON, there is no need to pay the cost of decoding and re-encoding it, so it is returned raw from the database.

Another minor improvement was also done to further optimize the JSON decoding inside the backend.

More specifically in a 15GB database and the query spanning roughly 3K sessions was ~30x faster using the index:

reframe --list-stored-testcases='now-7d:now?ci_pipeline_id=="123456789"'/mean:/+presult

where ci_pipeline_id is custom session property stored in the DB (see --session-extras).

The --describe-stored-session with a similar span of records was 3x faster using the improvements from this PR.

Finally, the index addition is seamless. The first time a query to the DB will be made with the new version the necessary DB indexes will be created if they do not exist. Note this very first query make take some time to execute depending on the size of the DB.

@vkarak vkarak added this to the ReFrame 4.9 milestone Sep 2, 2025
@vkarak vkarak self-assigned this Sep 2, 2025
@vkarak vkarak added the reporting Issues related to reporting and processing the test results label Sep 2, 2025
@codecov
Copy link

codecov bot commented Sep 2, 2025

Codecov Report

❌ Patch coverage is 96.77419% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 91.21%. Comparing base (f9c3c56) to head (590db46).
⚠️ Report is 5 commits behind head on develop.

Files with missing lines Patch % Lines
reframe/frontend/reporting/utility.py 0.00% 1 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff            @@
##           develop    #3546   +/-   ##
========================================
  Coverage    91.21%   91.21%           
========================================
  Files           62       62           
  Lines        13451    13462   +11     
========================================
+ Hits         12269    12280   +11     
  Misses        1182     1182           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@vkarak vkarak force-pushed the enhancement/db-perf-improvements branch from d62a69b to 0a4cb9a Compare September 2, 2025 14:18
@github-project-automation github-project-automation bot moved this from Todo to In Progress in ReFrame Backlog Sep 10, 2025
@vkarak vkarak force-pushed the enhancement/db-perf-improvements branch from 52fd884 to 590db46 Compare September 10, 2025 15:06
@vkarak vkarak enabled auto-merge September 10, 2025 22:25
@vkarak vkarak merged commit d2ecc2a into reframe-hpc:develop Sep 10, 2025
67 of 108 checks passed
@github-project-automation github-project-automation bot moved this from In Progress to Done in ReFrame Backlog Sep 10, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement prio: normal reporting Issues related to reporting and processing the test results

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants