Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: optimize codecov config #13329

Merged
merged 2 commits into from Feb 25, 2021
Merged

test: optimize codecov config #13329

merged 2 commits into from Feb 25, 2021

Conversation

ktmud
Copy link
Member

@ktmud ktmud commented Feb 25, 2021

SUMMARY

Optimize CodeCov config to

  1. Always let GitHub CodeCov checks pass because we don't always report all test coverage (because of conditional CI jobs)
  2. Add more detailed flags for coverage uploads
  3. Run tests also on workflow file changes

It seems the CodeCov App has recently been enabled in this repo (#9930), which is good, but it starts to block some PRs on status check due to false alarms of decreased test coverage.

blocked

This is because:

  1. The target coverage setting is set to "auto", which will report any decrease in test coverage
  2. We conditionally skip Python tests for pure frontend changes (and vice versa), hence certain PRs will never have the full test coverage reported.
  3. Sometimes frontend and backend workflows are delayed in queue and run in very different time. After wait for a while, CodeCov may come to believe the full CI has finished, therefore reporting only partial coverage from the already finished jobs.

The after_n_builds config is designed to address this issue, but since we don't know which jobs will run for each PR, we can't really set a meaningful value for this config.

I'm setting it to 4 because for Python tests there are 5 uploads (mysql, postgres, sqlite, hive, presto), and for frontend, there are 4 (unit tests + Cypress * 3).

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

N/A

TEST PLAN

  • CI Passes and flags correctly marked on CodeCov
    commit-flags
  • GH checks are not blocked by CodeCov

ADDITIONAL INFORMATION

  • Has associated issue:
  • Changes UI
  • Requires DB Migration.
  • Confirm DB Migration upgrade and downgrade tested.
  • Introduces new feature or API
  • Removes existing feature or API

@pull-request-size pull-request-size bot added size/M and removed size/S labels Feb 25, 2021
@codecov
Copy link

codecov bot commented Feb 25, 2021

Codecov Report

Merging #13329 (c694ee9) into master (8ab45c9) will decrease coverage by 0.06%.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff             @@
##           master   #13329      +/-   ##
==========================================
- Coverage   77.08%   77.01%   -0.07%     
==========================================
  Files         876      876              
  Lines       45439    45439              
  Branches     5440     5440              
==========================================
- Hits        35026    34997      -29     
- Misses      10290    10319      +29     
  Partials      123      123              
Flag Coverage Δ
cypress 58.16% <ø> (ø)
hive 79.97% <ø> (?)
javascript 62.27% <ø> (ø)
mysql 80.30% <ø> (?)
postgres 80.34% <ø> (?)
python 80.73% <ø> (-0.12%) ⬇️
sqlite 79.96% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
superset/db_engine_specs/presto.py 82.47% <0.00%> (-5.99%) ⬇️
superset/models/core.py 88.55% <0.00%> (-0.28%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8ab45c9...5ccc773. Read the comment docs.

@ktmud ktmud requested a review from villebro February 25, 2021 08:14
Copy link
Member

@villebro villebro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Member

@willbarrett willbarrett left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you Jesse for the optimization! Apologies for the unintended consequences.

@willbarrett willbarrett merged commit d39bc14 into apache:master Feb 25, 2021
@ktmud ktmud deleted the codecov-config branch February 25, 2021 18:36
allanco91 pushed a commit to allanco91/superset that referenced this pull request May 21, 2021
* test: optimize codecov config to let GitHub checks always pass

* Run tests on workflow changes, too
@mistercrunch mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 1.2.0 labels Mar 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels size/M 🚢 1.2.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants