BaseTools: reformat HostBasedUnitTestRunner coverage results by inf (m…

…icrosoft#616)

## Description

Coverage results that are generated from HostBasedUnitTestRunner are
inconsistent in terms of how the cobertura file is formatted. On
Windows, cobertura results are grouped by test executable while on
Linux, cobertura results are grouped by INF. This commit reformats
coverage results to always be by INF and to additionally be package-path
relative.

An additional change this commit makes is to rename the package coverage
file to include the package name. This is to make easily identify which
coverage results come from which host test DSC and additionally prevent
name conflicts when coverage results are uploaded during a PR gate.
Overall coverage is still generated and at the root of the output
directory (typically `$(ws)/Build/`)

An additional change is made to how the `input_coverage` parameter is
provided to `OpenCppCoverage` for msvc builds. `input_coverage` is now
provided via a config file due to the command line argument limit being
reached as the number of tests being run increases. The config file is
written, the command is run, then the file is deleted.

- [ ] Impacts functionality?
- **Functionality** - Does the change ultimately impact how firmware
functions?
- Examples: Add a new library, publish a new PPI, update an algorithm,
...
- [ ] Impacts security?
- **Security** - Does the change have a direct security impact on an
application,
    flow, or firmware?
  - Examples: Crypto algorithm change, buffer overflow fix, parameter
    validation improvement, ...
- [x] Breaking change?
- **Breaking change** - Will anyone consuming this change experience a
break
    in build or boot behavior?
- Examples: Add a new library class, move a module to a different repo,
call
    a function in a new library class in a pre-existing module, ...
- [ ] Includes tests?
  - **Tests** - Does the change include any explicit test code?
  - Examples: Unit tests, integration tests, robot tests, ...
- [ ] Includes documentation?
- **Documentation** - Does the change contain explicit documentation
additions
    outside direct code modifications (and comments)?
- Examples: Update readme file, add feature readme file, link to
documentation
    on an a separate Web page, ...

## How This Was Tested

N/A

## Integration Instructions

Any repository consuming MU_BASECORE for this commit and later must use
edk2-pytool-extensions >= 0.26.2.

.azurepipelines/Ubuntu-GCC5.yml

@@ -27,6 +27,7 @@ extends:
     do_non_ci_setup: true
     do_pr_eval: true
     calculate_code_coverage: true
+    coverage_publish_target: 'codecov'
     container_build: true
     os_type: Linux
     build_matrix:

.azurepipelines/Windows-VS.yml

@@ -27,6 +27,7 @@ extends:
     do_non_ci_setup: true
     do_pr_eval: true
     calculate_code_coverage: true
+    coverage_publish_target: 'codecov'
     os_type: Windows_NT
     build_matrix:
       TARGET_MDE_CPU:

.github/codecov.yml

@@ -0,0 +1,30 @@
+## @file
+# codecov upload configuration file to carryforward coverage results of
+# packages that do not upload coverage results for a given pull request.
+##
+# Copyright (c) Microsoft Corporation.
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+flags:
+  CryptoPkg:
+    carryforward: true
+  MdeModulePkg:
+    carryforward: true
+  MdePkg:
+    carryforward: true
+  NetworkPkg:
+    carryforward: true
+  PcAtChipsetPkg:
+    carryforward: true
+  PerformancePkg:
+    carryforward: true
+  PolicyServicePkg:
+    carryforward: true
+  ShellPkg:
+    carryforward: true
+  StandaloneMmPkg:
+    carryforward: true
+  UefiCpuPkg:
+    carryforward: true
+  UnitTestFrameworkPkg:
+    carryforward: true

.pytool/Plugin/HostUnitTestCompilerPlugin/HostUnitTestCompilerPlugin.py

@@ -87,6 +87,7 @@ def __GetHostUnitTestArch(self, environment):
     def RunBuildPlugin(self, packagename, Edk2pathObj, pkgconfig, environment, PLM, PLMHelper, tc, output_stream=None):
         self._env = environment
         environment.SetValue("CI_BUILD_TYPE", "host_unit_test", "Set in HostUnitTestCompilerPlugin")
+        environment.SetValue("CI_PACKAGE_NAME", packagename, "Set in HostUnitTestCompilerPlugin")
 
         # Parse the config for required DscPath element
         if "DscPath" not in pkgconfig:

BaseTools/Plugin/HostBasedUnitTestRunner/HostBasedUnitTestRunner.py

@@ -16,6 +16,8 @@
 from edk2toolext.environment import shell_environment
 from edk2toollib.utility_functions import RunCmd
 from edk2toollib.utility_functions import GetHostInfo
+from edk2toollib.database import Edk2DB  # MU_CHANGE - reformat coverage data
+from edk2toollib.database.tables import SourceTable, PackageTable, InfTable  # MU_CHANGE - reformat coverage data
 from textwrap import dedent
 
 
@@ -135,6 +137,15 @@ def do_post_build(self, thebuilder):
                     self.gen_code_coverage_msvc(thebuilder)
                 else:
                     logging.info("Skipping code coverage. Currently, support GCC and MSVC compiler.")
+                    return failure_count # MU_CHANGE - reformat coverage data
+
+                # MU_CHANGE begin - reformat coverage data
+                if thebuilder.env.GetValue("CC_REORGANIZE", "TRUE") == "TRUE":
+                    ret = self.organize_coverage(thebuilder)
+                    if ret != 0:
+                        logging.error("Failed to reorganize coverage data by INF.")
+                        return -1
+                # MU_CHANGE end - reformat coverage data
 
         return failure_count
 
@@ -162,14 +173,11 @@ def gen_code_coverage_gcc(self, thebuilder):
             logging.error("UnitTest Coverage: Failed to aggregate coverage data.")
             return 1
 
-        # Generate coverage XML
-        ret = RunCmd("lcov_cobertura",f"{buildOutputBase}/total-coverage.info -o {buildOutputBase}/compare.xml")
-        if ret != 0:
-            logging.error("UnitTest Coverage: Failed to generate coverage XML.")
-            return 1
-
         # Filter out auto-generated and test code
-        ret = RunCmd("lcov_cobertura",f"{buildOutputBase}/total-coverage.info --excludes ^.*UnitTest\|^.*MU\|^.*Mock\|^.*DEBUG -o {buildOutputBase}/coverage.xml")
+        # MU_CHANGE begin - reformat coverage data
+        file_out = thebuilder.env.GetValue("CI_PACKAGE_NAME", "") + "_coverage.xml"
+        ret = RunCmd("lcov_cobertura",f"{buildOutputBase}/total-coverage.info --excludes ^.*UnitTest\|^.*MU\|^.*Mock\|^.*DEBUG -o {buildOutputBase}/{file_out}")
+        # MU_CHANGE end - reformat coverage data
         if ret != 0:
             logging.error("UnitTest Coverage: Failed generate filtered coverage XML.")
             return 1
@@ -205,29 +213,73 @@ def gen_code_coverage_msvc(self, thebuilder):
         workspace = thebuilder.env.GetValue("WORKSPACE")
         workspace = (workspace + os.sep) if workspace[-1] != os.sep else workspace
         # Generate coverage file
-        coverageFile = ""
-        for testFile in testList:
-            ret = RunCmd("OpenCppCoverage", f"--source {workspace} --export_type binary:{testFile}.cov -- {testFile}", workingdir=f"{workspace}Build/")
-            coverageFile += " --input_coverage=" + testFile + ".cov"
-            if ret != 0:
-                logging.error("UnitTest Coverage: Failed to collect coverage data.")
-                return 1
+        # MU_CHANGE begin - reformat coverage data
+        pkg_cfg_file = os.path.join(buildOutputBase, "pkg-opencppcoverage.cfg")
+        if os.path.isfile(pkg_cfg_file):
+            os.remove(pkg_cfg_file)
+
+        with open(pkg_cfg_file, "w") as f:
+            for testFile in testList:
+                ret = RunCmd("OpenCppCoverage", f"--source {workspace} --export_type binary:{testFile}.cov -- {testFile}", workingdir=f"{workspace}Build/")
+                f.write(f"input_coverage={testFile}.cov\n")
+                if ret != 0:
+                    logging.error("UnitTest Coverage: Failed to collect coverage data.")
+                    return 1
 
         # Generate and XML file if requested.by each package
-        ret = RunCmd("OpenCppCoverage", f"--export_type cobertura:{os.path.join(buildOutputBase, 'coverage.xml')} {coverageFile}", workingdir=f"{workspace}Build/")
+
+        file_out = thebuilder.env.GetValue("CI_PACKAGE_NAME", "") + "_coverage.xml"
+        ret = RunCmd("OpenCppCoverage", f"--export_type cobertura:{os.path.join(buildOutputBase, file_out)} --config_file={pkg_cfg_file}", workingdir=f"{workspace}Build/")
+        os.remove(pkg_cfg_file)
+
         if ret != 0:
             logging.error("UnitTest Coverage: Failed to generate cobertura format xml in single package.")
             return 1
 
         # Generate total report XML file for all package
         testCoverageList = glob.glob(os.path.join(workspace, "Build", "**","*Test*.exe.cov"), recursive=True)
-        coverageFile = ""
-        for testCoverage in testCoverageList:
-            coverageFile += " --input_coverage=" + testCoverage
+        total_cfg_file = os.path.join(buildOutputBase, "total-opencppcoverage.cfg")
+        if os.path.isfile(total_cfg_file):
+            os.remove(total_cfg_file)
+
+        with open(total_cfg_file, "w") as f:
+            for testCoverage in testCoverageList:
+                f.write(f"input_coverage={testCoverage}\n")
+
+        ret = RunCmd("OpenCppCoverage", f"--export_type cobertura:{workspace}Build/coverage.xml --config_file={total_cfg_file}", workingdir=f"{workspace}Build/")
+        os.remove(total_cfg_file)
 
-        ret = RunCmd("OpenCppCoverage", f"--export_type cobertura:{workspace}Build/coverage.xml {coverageFile}", workingdir=f"{workspace}Build/")
         if ret != 0:
             logging.error("UnitTest Coverage: Failed to generate cobertura format xml.")
             return 1
 
         return 0
+
+    def organize_coverage(self, thebuilder) -> int:
+        """Organize the generated coverage file by INF."""
+        db_path = self.parse_workspace(thebuilder)
+
+        workspace = thebuilder.env.GetValue("WORKSPACE")
+        buildOutputBase = thebuilder.env.GetValue("BUILD_OUTPUT_BASE")
+        package = thebuilder.env.GetValue("CI_PACKAGE_NAME", "")
+        file_out = package + "_coverage.xml"
+        cov_file = os.path.join(buildOutputBase, file_out)
+
+        params = f"--database {db_path} coverage {cov_file} -o {cov_file} --by-package -ws {workspace}"
+
+        params += f" -p {package}" * int(package != "")
+        params += " --full" * int(thebuilder.env.GetValue("CC_FULL", "FALSE") == "TRUE")
+        params += " --flatten" * int(thebuilder.env.GetValue("CC_FLATTEN", "FALSE") == "TRUE")
+
+        return RunCmd("stuart_report", params)
+
+    def parse_workspace(self, thebuilder) -> str:
+        """Parses the workspace with Edk2DB with the tables necessarty to run stuart_report."""
+        db_path = os.path.join(thebuilder.env.GetValue("BUILD_OUTPUT_BASE"), "DATABASE.db")
+        with Edk2DB(db_path, thebuilder.edk2path) as db:
+            db.register(SourceTable(), PackageTable(), InfTable())
+            env_dict = thebuilder.env.GetAllBuildKeyValues() | thebuilder.env.GetAllNonBuildKeyValues()
+            db.parse(env_dict)
+
+        return db_path
+    # MU_CHANGE end - reformat coverage data

UnitTestFrameworkPkg/ReadMe.md

@@ -854,46 +854,61 @@ This mode is used by the test running plugin to aggregate the results for CI tes
 
 ### Code Coverage
 
-Host based Unit Tests will automatically enable coverage data.
+Code coverage can be enabled for Host based Unit Tests with `CODE_COVERAGE=TRUE`, which generates a cobertura report
+per package tested, and combined cobertura report for all packages tested. The per-package cobertura report will be
+present at `Build/<Pkg>/HostTest/<Target_Toolchain>/<Pkg>_coverage.xml`. The overall cobertura report will be present
+at `Build/coverage.xml`
+
+Code coverage generation has two config knobs:
+
+1. `CC_FULL`: If set to `TRUE`, will generate zero'd out coverage data for untested source files in the package.
+2. `CC_FLATTEN`: If Set to `TRUE`, will group all source files together, rather than by INF.
+
+**TIP: `CC_FLATTEN=TRUE/FALSE` will produce different coverage percentage results as `TRUE` de-duplicates source files
+that are consumed by multiple INFs.
 
 For Windows, this is primarily leverage for pipeline builds, but this can be leveraged locally using the
 OpenCppCoverage windows tool to parse coverage data to cobertura xml format.
 
+#### Prerequisites
+
+In addition to required prerequisites to build and test, there are additional requirements for calculating code
+coverage files as noted below.
+
 * Windows Prerequisite
 
-  ```text
-  Download and install https://github.com/OpenCppCoverage/OpenCppCoverage/releases
-  python -m pip install --upgrade -r ./pip-requirements.txt
-  stuart_ci_build -c .pytool/CISettings.py  -t NOOPT TOOL_CHAIN_TAG=VS2019 -p MdeModulePkg
-  Open Build/coverage.xml
-  ```
+  1. OpenCppCoverage: Download and install <https://github.com/OpenCppCoverage/OpenCppCoverage/releases>
 
-  * How to see code coverage data on IDE Visual Studio
+* Linux Prerequisite
 
-    ```text
-    Open Visual Studio VS2019 or above version
-    Click "Tools" -> "OpenCppCoverage Settings"
-    Fill your execute file into "Program to run:"
-    Click "Tools" -> "Run OpenCppCoverage"
-    ```
+  1. lcov: sudo apt-get install -y lcov
 
-For Linux, this is primarily leveraged for pipeline builds, but this can be leveraged locally using the
-lcov linux tool, and parsed using the lcov_cobertura python tool to parse it to cobertura xml format. pycobertura
-is used to covert this coverage data to a human readable HTML file. These tools must be installed to parse code
-coverage.
+#### Examples
 
 ```bash
-sudo apt-get install -y lcov
-pip install lcov_cobertura
-pip install pycobertura
+stuart_ci_build -c .pytool/CISettings.py -t NOOPT TOOL_CHAIN_TAG=VS2019 -p MdeModulePkg CODE_COVERAGE=TRUE
+stuart_ci_build -c .pytool/CISettings.py -t NOOPT TOOL_CHAIN_TAG=VS2019 CODE_COVERAGE=TRUE CC_FLATTEN=TRUE CC_FULL=FALSE
 ```
 
-During CI builds, use the  ```CODE_COVERAGE=TRUE``` flag to generate the code coverage XML files,
-and additionally use the ```CC_HTML=TRUE``` flag to generate the HTML file. This will be generated
-in Build/coverage.html.
+How to see code coverage data on IDE Visual Studio
+
+```text
+Open Visual Studio VS2019 or above version
+Click "Tools" -> "OpenCppCoverage Settings"
+Fill your execute file into "Program to run:"
+Click "Tools" -> "Run OpenCppCoverage"
+```
+
+#### Additional Tools
+
+There are a plethora of open source tools for generating reports from a Cobertura file, which is why it was selected as
+the output file format. Tools such as pycobertura (`pip install pycobertura`) and [reportgenerator](https://www.nuget.org/packages/dotnet-reportgenerator-globaltool)
+can be utilized to generate different report types, such as local html reports. VSCode Extensions such as [Coverage Gutters](https://marketplace.visualstudio.com/items?itemName=ryanluker.vscode-coverage-gutters)
+can highlight coverage results directly in the file, and cloud tools such as [CodeCov](https://about.codecov.io/) can
+consume cobertura files to provide PR checks and general code coverage statistics for the repository.
 
-There is currently no official guidance or support for code coverage when compiling
-in Visual Studio at this time.
+*** REMINDER: During CI builds, use the ``CODE_COVERAGE=TRUE` flag to generate the code coverage XML files,
+and additionally use the `CC_FLATTEN=TRUE` or `CC_FULL=TRUE` flags to customize coverage results.
 
 ### Important Note
 

pip-requirements.txt

@@ -13,9 +13,10 @@
 ##
 
 edk2-pytool-library~=0.19.4 # MU_CHANGE
-edk2-pytool-extensions~=0.26.0 # MU_CHANGE
+edk2-pytool-extensions~=0.26.2 # MU_CHANGE
 edk2-basetools==0.1.49
 antlr4-python3-runtime==4.13.1
 regex
 lcov-cobertura==2.0.2
+pygount==1.6.1 # MU_CHANGE
 toml==0.10.2 # MU_CHANGE