{Codecov upload issue debugging} - Ensure Codecov consistently waits for ALL reports to successfully process before reporting on coverage on UI and PR comment

[rohan-at-sentry]

CONTEXT In issue #2400, we confirmed inconsistencies in the reporting experience across the app and PR comments. Specifically:

• The app’s commit page displays a fully processed report.
• The pulls page and PR comments only show partial data.

This investigation was conducted by @nora-codecov and @RulaKhaled

User Experience Impact

Users frequently see partial data reports, leading to the perception of incorrect or incomplete information, which undermines trust in our reporting accuracy. Improving user trust and data clarity is a key focus area.

One solution under consideration involves providing feedback as processing occurs (see #2522). However, implementing this depends on resolving blockers related to #2400, particularly around the feasibility of showing full reports and identifying intermediary processing states.

Aside, but related, @spalmurray-codecov has explored a potential solution, which may address some of these blockers (see findings here).

Next Steps

1. Determine Reporting Capabilities

   • Assess if and how we can show full report data (start, stop).
   • Investigate options for displaying an intermediary processing state.
     • A. Applications: @suejung-sentry to lead exploration 1.A Applications: Determine Reporting Capabilities #2918.
     • B. Platform: @adrian-codecov to lead exploration 1.B Platform: Determine Reporting Capabilities #2919.

2. Evaluate Findings

   • Based on investigation results, the Eng/Design/PM team will assess next steps, such as showing an intermediary state, adjusting configuration defaults, or other potential improvements.
   • Consider a more standardized approach to report processing, potentially deprecating after_n_builds or setting default behavior without configuration. This will depend heavily on investigation outcomes.
     • cc @Adal3n3 @codecovdesign @rohan-at-sentry @trent-codecov @katia-sentry
     • 2. Evaluate Findings #2920

3. Parallel Investigations

   • A. @spalmurray-codecov to continue his related investigation (see findings here).
     • issue: 3.A Investigation: Estimating the expected number of uploads for a commit #2921
   • B. TBD (platform team @trent-codecov) revisit open PRs 3.B Investigation #2922:
     • feat: notify_error default=True for repos created after a certain date (shared#320)
     • feat: make notify_error default to true in config worker ([worker#601](Discovery: 1) GH codecov coverage action, and 2) appless codecov #2067

kickoff details / rough notes / other

Today we have 2 ways to let customers instruct customers on when to kick off the notification process..

1. After_n_builds - https://docs.codecov.com/docs/pull-request-comments#after_n_builds
2. manual_trigger and send-notifications - https://github.com/codecov/codecov-cli#send-notifications

Problem 1 - Not every customer knows to turn this feature on. Turning this feature on allows Codecov to "know" what is the max # of reports to expect per commit, which allows us to better surface that information to users in-case of dropped / missing uploads.

Problem 2 - This applies a "delay" to notifications sent to the PR comment. The problem is that the UI is effectively decoupled from this leading to inconsistencies in how coverage is reported for uploads that fail on the UI and PR Comment.

Path Forward

Solve problems 1 & 2

• For customers that do not have this turned on, Codecov effectively cannot know if all reports have been processed. This means that project coverage reporting is inherently inaccurate. We should surface messaging on PR comment and UI (on commits tab and PR tab) inviting users to turn this on

  • Copy "Codecov isn't confident that it has received all coverage reports for this commit. See how to fix this (link to docs) "

• Ensure manual_trigger and after_n_builds impacts both PR comment and Codecov UI

  • Move after_n_builds earlier in the report processing cycle. #1245

For Q4 - the goal would be to wrap up discovery on this topic

---

Related problem to solve

Previously in #2400 we found that:

• in app the commits page awaits full data reporting
• in app pulls data is (xx)
• pr comment:
  • pr comment does not notify user that there are failed uploads
  • pr comment displays patch and project coverage
  • there is no indication anywhere on the pr that there was an issue with uploads
  • similarly, in codecov.io there's very little indication of upload failure
  • it's unclear that 1 failed upload invalidates all past and any future uploads on the commit

The issue is particularly with the PR comment experience is the data is showing partially as uploads come in and therefore can sometimes give the user a false understanding.

Solution exploration

We'd like to explore 1) how we can make the reporting experience consistent in awaiting to see the full report and 2) have an intermediary state to clarify to the user processing is occurring.

• Align experiences to signal processing is occurring and show full report once complete
• Related to the PR comment, this discovery to look at how we can do this. (related open/former work feat: notify_error default=True for repos created after certain date shared#320 and feat: make notify_error default to true in config worker#601)

related ongoing iteration to show processing: #2522

[rohan-at-sentry]

Things to talk about during kickoff

• Can manual_trigger and after_n_builds be replaced by a single "feature"

  • What makes both things required

• If either manual_trigger or after_n_builds were the default, what are the performance implications (or other platform considerations) that would give us pause

[codecovdesign]

sync with @adrian-codecov @Adal3n3 @rohan-at-sentry @suejung-sentry @spalmurray-codecov

• issue 1: We'd like to explore 1) how we can make the reporting experience consistent in awaiting to see the full report and 2) have an intermediary state to clarify to the user processing is occurring
  • issue a @suejung-sentry to investigate applications (what is start/end, and processing state? and can these be determined)
  • issue b @adrian-codecov to investigate (what is start/end, and processing state? and can these be determined)
• issue 2: spencer's review
  • pending: Implement the Upload processing status UI component #2522 (we'll need results of issue 1/2 to determine if/what/how we'll approach this)
• issue 3: revisit feat: notify_error default=True for repos created after certain date shared#320 and feat: make notify_error default to true in config worker#601)

✅ @codecovdesign to create issues and one more sync to debrief: #2400 and ping katia and trent for capacity planning (adrian: good candidate for code freeze time)
rohan: also, let's keep async updates high lever here in this issue