feat: make comparison session agnostic by only comparing the totals #410
Conversation
| @@ -486,7 +486,17 @@ def merge_report(cumulative_report, obj): | |||
| incremental_report = obj["report"] | |||
| parallel_idx = obj["parallel_idx"] | |||
|
|
|||
| assert len(incremental_report.sessions) == 1 | |||
There was a problem hiding this comment.
from the logs, i noticed sometimes an incremental report has 0 sessions, which is weird
Codecov ReportAttention: Patch coverage is ✅ All tests successful. No failed tests found @@ Coverage Diff @@
## main #410 +/- ##
==========================================
- Coverage 97.52% 97.48% -0.05%
==========================================
Files 395 395
Lines 33355 33369 +14
==========================================
Hits 32530 32530
- Misses 825 839 +14
Flags with carried forward coverage won't be shown. Click here to find out more.
|
Codecov ReportAttention: Patch coverage is
✅ All tests successful. No failed tests found @@ Coverage Diff @@
## main #410 +/- ##
==========================================
- Coverage 97.52% 97.48% -0.05%
==========================================
Files 395 395
Lines 33355 33369 +14
==========================================
Hits 32530 32530
- Misses 825 839 +14
Flags with carried forward coverage won't be shown. Click here to find out more.
|
Codecov ReportAttention: Patch coverage is
✅ All tests successful. No failed tests found Additional details and impacted files@@ Coverage Diff @@
## main #410 +/- ##
==========================================
- Coverage 97.52% 97.48% -0.05%
==========================================
Files 395 395
Lines 33355 33369 +14
==========================================
Hits 32530 32530
- Misses 825 839 +14
Flags with carried forward coverage won't be shown. Click here to find out more.
|
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #410 +/- ##
==========================================
- Coverage 97.55% 97.51% -0.05%
==========================================
Files 426 426
Lines 34046 34060 +14
==========================================
Hits 33212 33212
- Misses 834 848 +14
Flags with carried forward coverage won't be shown. Click here to find out more.
|
tasks/parallel_verification.py
Outdated
| # file level totals comparison | ||
| for filename, file_summary in parallel_report._files.items(): | ||
| parallel_file_level_totals = file_summary.file_totals | ||
| serial_file_level_totals = serial_report._files[filename].file_totals |
There was a problem hiding this comment.
I know we're assuming that parallel report and serial report should contain the same files but in the case that parallel report contains a file that serial report does not this will crash, just in case that's a possibility we should handle this
There was a problem hiding this comment.
good point. I was originally relying on the check here: https://github.com/codecov/worker/pull/410/files#diff-49dba3d15152574626bb8cc6bb4d690c86c34d1d5d9be202e012f70f27be9b5dR110
but that wouldn't run if this throws an error. Thanks
daniel-codecov
force-pushed
the
daniel/use-totals-for-parallel-verification
branch
from
cc40771
to
7032820
Compare
For the parallel upload processing verification experiment, I am changing how the outputs of the parallel vs serial flow are compared.
Instead of comparing exactly the report json and chunks.txt of our reports, I compare the top level totals of coverage, and the file level totals for each file instead. This way, even if the way we assign session ids to uploads in the parallel flow differs from the serial flow (which would've caused a diff in the old comparison method), the coverage totals should hopefully still be the same indicating that the processing was successful.
Legal Boilerplate
Look, I get it. The entity doing business as "Sentry" was incorporated in the State of Delaware in 2015 as Functional Software, Inc. In 2022 this entity acquired Codecov and as result Sentry is going to need some rights from me in order to utilize my contributions in this PR. So here's the deal: I retain all rights, title and interest in and to my contributions, and by keeping this boilerplate intact I confirm that Sentry can use, modify, copy, and redistribute my contributions, under Sentry's choice of terms.