BaseTools: reformat HostBasedUnitTestRunner coverage results by inf #616

Javagedes · 2023-11-07T22:55:39Z

Description

Coverage results that are generated from HostBasedUnitTestRunner are inconsistent in terms of how the cobertura file is formatted. On Windows, cobertura results are grouped by test executable while on Linux, cobertura results are grouped by INF. This commit reformats coverage results to always be by INF and to additionally be package-path relative.

An additional change this commit makes is to rename the package coverage file to include the package name. This is to make easily identify which coverage results come from which host test DSC and additionally prevent name conflicts when coverage results are uploaded during a PR gate. Overall coverage is still generated and at the root of the output directory (typically $(ws)/Build/)

An additional change is made to how the input_coverage parameter is provided to OpenCppCoverage for msvc builds. input_coverage is now provided via a config file due to the command line argument limit being reached as the number of tests being run increases. The config file is written, the command is run, then the file is deleted.

Impacts functionality?
- Functionality - Does the change ultimately impact how firmware functions?
- Examples: Add a new library, publish a new PPI, update an algorithm, ...
Impacts security?
- Security - Does the change have a direct security impact on an application,
  flow, or firmware?
- Examples: Crypto algorithm change, buffer overflow fix, parameter
  validation improvement, ...
Breaking change?
- Breaking change - Will anyone consuming this change experience a break
  in build or boot behavior?
- Examples: Add a new library class, move a module to a different repo, call
  a function in a new library class in a pre-existing module, ...
Includes tests?
- Tests - Does the change include any explicit test code?
- Examples: Unit tests, integration tests, robot tests, ...
Includes documentation?
- Documentation - Does the change contain explicit documentation additions
  outside direct code modifications (and comments)?
- Examples: Update readme file, add feature readme file, link to documentation
  on an a separate Web page, ...

How This Was Tested

N/A

Integration Instructions

Any repository consuming MU_BASECORE for this commit and later must use edk2-pytool-extensions >= 0.26.0.

Coverage results that are generated from HostBasedUnitTestRunner are inconsistent in terms of how the cobertura file is fomratted. On Windows, cobertura results are grouped by test executable while on Linux, cobertura results are grouped by INF. This commit reformat reformats coverage results to always be by INF and to additionally be package-path relative. An additional change this commit makes is to rename the package coverage file to include the platform name. This is to make easily identify which coverage results come from which host test DSC and additionally prevent name conflicts when coverage results are uploaded during a PR gate.

codecov-commenter · 2023-11-08T21:44:40Z

Codecov Report

All modified and coverable lines are covered by tests ✅

❗ No coverage uploaded for pull request base (release/202302@a4a9353). Click here to learn what that means.

Additional details and impacted files

@@                Coverage Diff                @@
##             release/202302     #616   +/-   ##
=================================================
  Coverage                  ?   24.85%           
=================================================
  Files                     ?      128           
  Lines                     ?    21559           
  Branches                  ?     5756           
=================================================
  Hits                      ?     5359           
  Misses                    ?    16150           
  Partials                  ?       50

Flag	Coverage Δ
CryptoPkg	`29.15% <0.00%> (?)`
MdeModulePkg	`11.71% <0.00%> (?)`
MdePkg	`39.37% <0.00%> (?)`
PolicyServicePkg	`65.99% <0.00%> (?)`
UefiCpuPkg	`87.05% <0.00%> (?)`
UnitTestFrameworkPkg	`∅ <0.00%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

…616) Coverage results that are generated from HostBasedUnitTestRunner are inconsistent in terms of how the cobertura file is formatted. On Windows, cobertura results are grouped by test executable while on Linux, cobertura results are grouped by INF. This commit reformats coverage results to always be by INF and to additionally be package-path relative. An additional change this commit makes is to rename the package coverage file to include the package name. This is to make easily identify which coverage results come from which host test DSC and additionally prevent name conflicts when coverage results are uploaded during a PR gate. Overall coverage is still generated and at the root of the output directory (typically `$(ws)/Build/`) An additional change is made to how the `input_coverage` parameter is provided to `OpenCppCoverage` for msvc builds. `input_coverage` is now provided via a config file due to the command line argument limit being reached as the number of tests being run increases. The config file is written, the command is run, then the file is deleted. - [ ] Impacts functionality? - **Functionality** - Does the change ultimately impact how firmware functions? - Examples: Add a new library, publish a new PPI, update an algorithm, ... - [ ] Impacts security? - **Security** - Does the change have a direct security impact on an application, flow, or firmware? - Examples: Crypto algorithm change, buffer overflow fix, parameter validation improvement, ... - [x] Breaking change? - **Breaking change** - Will anyone consuming this change experience a break in build or boot behavior? - Examples: Add a new library class, move a module to a different repo, call a function in a new library class in a pre-existing module, ... - [ ] Includes tests? - **Tests** - Does the change include any explicit test code? - Examples: Unit tests, integration tests, robot tests, ... - [ ] Includes documentation? - **Documentation** - Does the change contain explicit documentation additions outside direct code modifications (and comments)? - Examples: Update readme file, add feature readme file, link to documentation on an a separate Web page, ... N/A Any repository consuming MU_BASECORE for this commit and later must use edk2-pytool-extensions >= 0.26.2.

…icrosoft#616) Coverage results that are generated from HostBasedUnitTestRunner are inconsistent in terms of how the cobertura file is formatted. On Windows, cobertura results are grouped by test executable while on Linux, cobertura results are grouped by INF. This commit reformats coverage results to always be by INF and to additionally be package-path relative. An additional change this commit makes is to rename the package coverage file to include the package name. This is to make easily identify which coverage results come from which host test DSC and additionally prevent name conflicts when coverage results are uploaded during a PR gate. Overall coverage is still generated and at the root of the output directory (typically `$(ws)/Build/`) An additional change is made to how the `input_coverage` parameter is provided to `OpenCppCoverage` for msvc builds. `input_coverage` is now provided via a config file due to the command line argument limit being reached as the number of tests being run increases. The config file is written, the command is run, then the file is deleted. - [ ] Impacts functionality? - **Functionality** - Does the change ultimately impact how firmware functions? - Examples: Add a new library, publish a new PPI, update an algorithm, ... - [ ] Impacts security? - **Security** - Does the change have a direct security impact on an application, flow, or firmware? - Examples: Crypto algorithm change, buffer overflow fix, parameter validation improvement, ... - [x] Breaking change? - **Breaking change** - Will anyone consuming this change experience a break in build or boot behavior? - Examples: Add a new library class, move a module to a different repo, call a function in a new library class in a pre-existing module, ... - [ ] Includes tests? - **Tests** - Does the change include any explicit test code? - Examples: Unit tests, integration tests, robot tests, ... - [ ] Includes documentation? - **Documentation** - Does the change contain explicit documentation additions outside direct code modifications (and comments)? - Examples: Update readme file, add feature readme file, link to documentation on an a separate Web page, ... N/A Any repository consuming MU_BASECORE for this commit and later must use edk2-pytool-extensions >= 0.26.2.

…icrosoft#616) Coverage results that are generated from HostBasedUnitTestRunner are inconsistent in terms of how the cobertura file is formatted. On Windows, cobertura results are grouped by test executable while on Linux, cobertura results are grouped by INF. This commit reformats coverage results to always be by INF and to additionally be package-path relative. An additional change this commit makes is to rename the package coverage file to include the package name. This is to make easily identify which coverage results come from which host test DSC and additionally prevent name conflicts when coverage results are uploaded during a PR gate. Overall coverage is still generated and at the root of the output directory (typically `$(ws)/Build/`) An additional change is made to how the `input_coverage` parameter is provided to `OpenCppCoverage` for msvc builds. `input_coverage` is now provided via a config file due to the command line argument limit being reached as the number of tests being run increases. The config file is written, the command is run, then the file is deleted. cherry-picked from 991a64e, pulled from 43ae607 in 202311 - [ ] Impacts functionality? - **Functionality** - Does the change ultimately impact how firmware functions? - Examples: Add a new library, publish a new PPI, update an algorithm, ... - [ ] Impacts security? - **Security** - Does the change have a direct security impact on an application, flow, or firmware? - Examples: Crypto algorithm change, buffer overflow fix, parameter validation improvement, ... - [x] Breaking change? - **Breaking change** - Will anyone consuming this change experience a break in build or boot behavior? - Examples: Add a new library class, move a module to a different repo, call a function in a new library class in a pre-existing module, ... - [ ] Includes tests? - **Tests** - Does the change include any explicit test code? - Examples: Unit tests, integration tests, robot tests, ... - [ ] Includes documentation? - **Documentation** - Does the change contain explicit documentation additions outside direct code modifications (and comments)? - Examples: Update readme file, add feature readme file, link to documentation on an a separate Web page, ... N/A Any repository consuming MU_BASECORE for this commit and later must use edk2-pytool-extensions >= 0.26.2.

…icrosoft#616) (microsoft#688) Coverage results that are generated from HostBasedUnitTestRunner are inconsistent in terms of how the cobertura file is formatted. On Windows, cobertura results are grouped by test executable while on Linux, cobertura results are grouped by INF. This commit reformats coverage results to always be by INF and to additionally be package-path relative. An additional change this commit makes is to rename the package coverage file to include the package name. This is to make easily identify which coverage results come from which host test DSC and additionally prevent name conflicts when coverage results are uploaded during a PR gate. Overall coverage is still generated and at the root of the output directory (typically `$(ws)/Build/`) An additional change is made to how the `input_coverage` parameter is provided to `OpenCppCoverage` for msvc builds. `input_coverage` is now provided via a config file due to the command line argument limit being reached as the number of tests being run increases. The config file is written, the command is run, then the file is deleted. Integrates edk2-pytool-extensions 0.27.0 and edk2-pytool-library 0.20.0, which overhauls the database functionality to use an ORM for managing the database schema and access to the database. Updates the only plugin in MU_BASECORE that uses the database functionality, HostBasedUnitTestRunner. cherry-picked from 991a64e, pulled from 43ae607 in 202311 cherry-picked from 785fe3d - [ ] Impacts functionality? - [ ] Impacts security? - [x] Breaking change? - [ ] Includes tests? - [ ] Includes documentation?.

…616) (#688) Coverage results that are generated from HostBasedUnitTestRunner are inconsistent in terms of how the cobertura file is formatted. On Windows, cobertura results are grouped by test executable while on Linux, cobertura results are grouped by INF. This commit reformats coverage results to always be by INF and to additionally be package-path relative. An additional change this commit makes is to rename the package coverage file to include the package name. This is to make easily identify which coverage results come from which host test DSC and additionally prevent name conflicts when coverage results are uploaded during a PR gate. Overall coverage is still generated and at the root of the output directory (typically `$(ws)/Build/`) An additional change is made to how the `input_coverage` parameter is provided to `OpenCppCoverage` for msvc builds. `input_coverage` is now provided via a config file due to the command line argument limit being reached as the number of tests being run increases. The config file is written, the command is run, then the file is deleted. Integrates edk2-pytool-extensions 0.27.0 and edk2-pytool-library 0.20.0, which overhauls the database functionality to use an ORM for managing the database schema and access to the database. Updates the only plugin in MU_BASECORE that uses the database functionality, HostBasedUnitTestRunner. cherry-picked from 991a64e, pulled from 43ae607 in 202311 cherry-picked from 785fe3d - [ ] Impacts functionality? - [ ] Impacts security? - [x] Breaking change? - [ ] Includes tests? - [ ] Includes documentation?.

…icrosoft#616) (microsoft#688) Coverage results that are generated from HostBasedUnitTestRunner are inconsistent in terms of how the cobertura file is formatted. On Windows, cobertura results are grouped by test executable while on Linux, cobertura results are grouped by INF. This commit reformats coverage results to always be by INF and to additionally be package-path relative. An additional change this commit makes is to rename the package coverage file to include the package name. This is to make easily identify which coverage results come from which host test DSC and additionally prevent name conflicts when coverage results are uploaded during a PR gate. Overall coverage is still generated and at the root of the output directory (typically `$(ws)/Build/`) An additional change is made to how the `input_coverage` parameter is provided to `OpenCppCoverage` for msvc builds. `input_coverage` is now provided via a config file due to the command line argument limit being reached as the number of tests being run increases. The config file is written, the command is run, then the file is deleted. Integrates edk2-pytool-extensions 0.27.0 and edk2-pytool-library 0.20.0, which overhauls the database functionality to use an ORM for managing the database schema and access to the database. Updates the only plugin in MU_BASECORE that uses the database functionality, HostBasedUnitTestRunner. cherry-picked from 991a64e, pulled from 43ae607 in 202311 cherry-picked from 785fe3d - [ ] Impacts functionality? - [ ] Impacts security? - [x] Breaking change? - [ ] Includes tests? - [ ] Includes documentation?.

github-actions bot added language:python Pull requests that update Python code impact:breaking-change Requires integration attention impact:non-functional Does not have a functional impact labels Nov 7, 2023

Javagedes force-pushed the unittestrunner-update branch from 53d5b20 to bc5f740 Compare November 7, 2023 22:56

Javagedes requested review from cfernald, makubacki and apop5 November 7, 2023 23:44

apop5 approved these changes Nov 8, 2023

View reviewed changes

Javagedes added 4 commits November 8, 2023 11:43

Use CI_PACKAGE_NAME

ffdfe21

Filter report on Package

7b51988

Formatting

b50c490

Javagedes force-pushed the unittestrunner-update branch from 95df55b to b50c490 Compare November 8, 2023 16:43

Javagedes marked this pull request as draft November 8, 2023 21:24

Javagedes force-pushed the unittestrunner-update branch 5 times, most recently from 714af1e to ce5b9f4 Compare November 8, 2023 21:33

Javagedes force-pushed the unittestrunner-update branch 10 times, most recently from 505d546 to e2808b0 Compare November 9, 2023 17:20

Javagedes force-pushed the unittestrunner-update branch from 9fc596b to ffe37e7 Compare November 9, 2023 19:09

Javagedes and others added 3 commits November 13, 2023 13:45

updatre

28fe936

Merge branch 'release/202302' into unittestrunner-update

0f0e51e

remove dots.

eb0754b

Javagedes marked this pull request as ready for review November 13, 2023 23:58

Javagedes added 9 commits November 13, 2023 16:09

Provide input_coverage in a config file

a39e07f

Add codecov.yml file

0f60eb4

update

35fe683

move codecov.yml file

1ca0646

update

1f43490

update cspell / markdownlint

b274b30

update

4f7a5cf

udate

fb03756

Update

78101e0

cfernald approved these changes Nov 14, 2023

View reviewed changes

Merge branch 'release/202302' into unittestrunner-update

44e49c6

makubacki approved these changes Nov 14, 2023

View reviewed changes

Javagedes merged commit 991a64e into microsoft:release/202302 Nov 14, 2023
38 checks passed

ProjectMuBot mentioned this pull request Nov 16, 2023

Bump MU_BASECORE from 2023020007.4.0 to 2023020008.0.0 microsoft/mu_tiano_platforms#773

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BaseTools: reformat HostBasedUnitTestRunner coverage results by inf #616

BaseTools: reformat HostBasedUnitTestRunner coverage results by inf #616

Javagedes commented Nov 7, 2023 •

edited

Loading

codecov-commenter commented Nov 8, 2023 •

edited

Loading

BaseTools: reformat HostBasedUnitTestRunner coverage results by inf #616

BaseTools: reformat HostBasedUnitTestRunner coverage results by inf #616

Conversation

Javagedes commented Nov 7, 2023 • edited Loading

Description

How This Was Tested

Integration Instructions

codecov-commenter commented Nov 8, 2023 • edited Loading

Codecov Report

Javagedes commented Nov 7, 2023 •

edited

Loading

codecov-commenter commented Nov 8, 2023 •

edited

Loading