Invalid code coverage report using sfdx force:source:deploy and force:mdapi:deploy #1568

bzacharski · 2022-06-12T21:54:56Z

Summary

Tested on version 7.155.0 (salesforce/salesforcedx:7.155.0-full), 154 and 153
I wanted to use code coverage from deploy commands to feed sonarqube (json format) and gitlab (cobertura format) with code coverage information, but it is failing. I've analyzed reports from deploy commands and compared them to reports from force:apex:test:run command.

For json format it is different between deploy and test:run commands
It looks like reports from deploy commands are invalid. For example I have a class x on destination org and by running it using test:run command I received report that class has 13 total lines and 11 is covered. Which is correct, but both json and cobertura from deploy commands have entries for 15 lines of which 2 are not covered (0). I've checked for sure that I am deploying the same class. It is reporting lines that are empty, have only comments, etc. And it looks that data from deploy commands are invalid for all files. Class form example (I replaced x, y, z, v, w, s, t):

/**************************************************************************************
 *  @Class            : x
 *  @Author           : y
 *  @Version History  : 1.0
 *  @Date             : 
 *  @Description      : Handler class for  z.
****************************************************************************************/
public with sharing class x extends TriggerHandler {
    
    private static final string RE_OBJ_API = 'v';
    
    /* Class constructor which invokes parent constructor using standard super function */
    public x(){
        super(RE_OBJ_API );     
    } 
    /* To get the API name of the executing trigger context */
    public override String getName(){
        return RE_OBJ_API ;
    }
    public override  void beforeInsert(List<SObject> newItems){}
    
    /* To invoke before update trigger logic */
    public override void beforeUpdate(List<SObject> newItems, Map<Id, SObject> newItemsMap, Map<Id, SObject> oldItemsMap) {}
    
    /* To invoke before delete trigger logic */
    public override void beforeDelete(Map<Id, SObject> oldItemsMap) {
     w.s(oldItemsMap);
    }
    
    /* To invoke after update trigger logic */
    public override void afterUpdate(List<SObject> newItems, Map<Id, SObject> newItemsMap, Map<Id, SObject> oldItemsMap) {}
    
    /* To invoke after delete trigger logic */
    public override void afterDelete(Map<Id, SObject> oldItemsMap) {
        w.t(oldItemsMap);
    }
    
    /* To invoke after undelete trigger logic */
    public  override void afterUndelete(Map<Id, SObject> oldItemsMap) {}
    
    /* To invoke after insert trigger logic */
    public override void afterInsert(List<SObject> newItems, Map<Id, SObject> newItemsMap) {}

}

test:run entry for given class:

{
    "id": "01p3N000009V0efQAC",
    "name": "x",
    "totalLines": 13,
    "lines": {
      "13": 1,
      "14": 1,
      "17": 0,
      "18": 0,
      "20": 1,
      "23": 1,
      "26": 1,
      "27": 1,
      "31": 1,
      "34": 1,
      "35": 1,
      "39": 1,
      "42": 1
    },
    "totalCovered": 11,
    "coveredPercent": 85
  }

deploy json entry for class:

"src/x": {
        "fnMap": {},
        "branchMap": {},
        "path": "src/x",
        "f": {},
        "b": {},
        "s": {
            "17": 0,
            "18": 0,
            "19": 1,
            "20": 1,
            "21": 1,
            "22": 1,
            "23": 1,
            "24": 1,
            "25": 1,
            "26": 1,
            "27": 1,
            "28": 1,
            "29": 1,
            "30": 1,
            "31": 1
        },
        "statementMap": {
            "17": {
                "start": {
                    "line": 17,
                    "column": 0
                },
                "end": {
                    "line": 17,
                    "column": 0
                }
            },
            "18": {
                "start": {
                    "line": 18,
                    "column": 0
                },
                "end": {
                    "line": 18,
                    "column": 0
                }
            },
            "19": {
                "start": {
                    "line": 19,
                    "column": 0
                },
                "end": {
                    "line": 19,
                    "column": 0
                }
            },
            "20": {
                "start": {
                    "line": 20,
                    "column": 0
                },
                "end": {
                    "line": 20,
                    "column": 0
                }
            },
            "21": {
                "start": {
                    "line": 21,
                    "column": 0
                },
                "end": {
                    "line": 21,
                    "column": 0
                }
            },
            "22": {
                "start": {
                    "line": 22,
                    "column": 0
                },
                "end": {
                    "line": 22,
                    "column": 0
                }
            },
            "23": {
                "start": {
                    "line": 23,
                    "column": 0
                },
                "end": {
                    "line": 23,
                    "column": 0
                }
            },
            "24": {
                "start": {
                    "line": 24,
                    "column": 0
                },
                "end": {
                    "line": 24,
                    "column": 0
                }
            },
            "25": {
                "start": {
                    "line": 25,
                    "column": 0
                },
                "end": {
                    "line": 25,
                    "column": 0
                }
            },
            "26": {
                "start": {
                    "line": 26,
                    "column": 0
                },
                "end": {
                    "line": 26,
                    "column": 0
                }
            },
            "27": {
                "start": {
                    "line": 27,
                    "column": 0
                },
                "end": {
                    "line": 27,
                    "column": 0
                }
            },
            "28": {
                "start": {
                    "line": 28,
                    "column": 0
                },
                "end": {
                    "line": 28,
                    "column": 0
                }
            },
            "29": {
                "start": {
                    "line": 29,
                    "column": 0
                },
                "end": {
                    "line": 29,
                    "column": 0
                }
            },
            "30": {
                "start": {
                    "line": 30,
                    "column": 0
                },
                "end": {
                    "line": 30,
                    "column": 0
                }
            },
            "31": {
                "start": {
                    "line": 31,
                    "column": 0
                },
                "end": {
                    "line": 31,
                    "column": 0
                }
            }
        }
    }

deploy cobertura entry for class:

<class name="x" filename="src/x" line-rate="0.8665999999999999" branch-rate="1">
          <methods>
          </methods>
          <lines>
            <line number="17" hits="0" branch="false"/>
            <line number="18" hits="0" branch="false"/>
            <line number="19" hits="1" branch="false"/>
            <line number="20" hits="1" branch="false"/>
            <line number="21" hits="1" branch="false"/>
            <line number="22" hits="1" branch="false"/>
            <line number="23" hits="1" branch="false"/>
            <line number="24" hits="1" branch="false"/>
            <line number="25" hits="1" branch="false"/>
            <line number="26" hits="1" branch="false"/>
            <line number="27" hits="1" branch="false"/>
            <line number="28" hits="1" branch="false"/>
            <line number="29" hits="1" branch="false"/>
            <line number="30" hits="1" branch="false"/>
            <line number="31" hits="1" branch="false"/>
          </lines>
        </class>

Steps To Reproduce:

Create class similar to example, deploy to org, run test:run and gather json coverage report, deploy the same class using deploy command (mdapi or source) and gather json and/or cobertura coverage reports

Expected result

Deploy commands produce valid reports. Json reports for tesr:run and deploy commands have similar/identical structure.

Actual result

Reports from deploy commands are invalid. Json reports have different structure.

System Information

{
	"cliVersion": "sfdx-cli/7.155.0",
	"architecture": "linux-x[64](https://code.roche.com/dws/DWS/-/jobs/9233716#L64)",
	"nodeVersion": "node-v16.13.2",
	"pluginVersions": [
		"@oclif/plugin-autocomplete 0.3.0 (core)",
		"@oclif/plugin-commands 1.3.0 (core)",
		"@oclif/plugin-help 3.3.1 (core)",
		"@oclif/plugin-not-found 1.2.6 (core)",
		"@oclif/plugin-plugins 1.10.11 (core)",
		"@oclif/plugin-update 1.5.0 (core)",
		"@oclif/plugin-warn-if-update-available 1.7.3 (core)",
		"@oclif/plugin-which 1.0.4 (core)",
		"@salesforce/sfdx-plugin-lwc-test 0.1.7 (core)",
		"alias 2.0.1 (core)",
		"apex 0.13.0 (core)",
		"auth 2.1.0 (core)",
		"community 2.0.0 (core)",
		"config 1.4.12 (core)",
		"custom-metadata 2.0.0 (core)",
		"data 2.0.3 (core)",
		"generator 2.0.1 (core)",
		"info 2.0.1 (core)",
		"limits 2.0.1 (core)",
		"org 1.13.2 (core)",
		"salesforce-alm 54.5.0 (core)",
		"schema 2.1.1 (core)",
		"sfdx-cli 7.155.0 (core)",
		"signups 1.1.2 (core)",
		"source 1.10.2 (core)",
		"telemetry 2.0.0 (core)",
		"templates 54.8.0 (core)",
		"trust 2.0.1 (core)",
		"user 2.0.2 (core)"
	],
	"osVersion": "Linux 5.4.0-109-generic"
}

Additional information

The text was updated successfully, but these errors were encountered:

github-actions · 2022-06-12T21:55:11Z

Thank you for filing this issue. We appreciate your feedback and will review the issue as soon as possible. Remember, however, that GitHub isn't a mechanism for receiving support under any agreement or SLA. If you require immediate assistance, contact Salesforce Customer Support.

bzacharski · 2022-06-12T22:03:23Z

Test level was always RunLocalTests

WillieRuemmele · 2022-06-13T15:36:31Z

Hi @bzacharski could you please provide the exact commands and arguments you running?

bzacharski · 2022-06-13T16:26:05Z

@WillieRuemmele
for deployment it will be for example:
sfdx force:source:deploy --json --loglevel ERROR --targetusername myAlias --checkonly --wait 1440 --testlevel RunLocalTests --verbose --sourcepath source --resultsdir tests/apex/ --coverageformatters cobertura,json --junit
for apex test:
sfdx force:apex:test:run -c -r json,cobertura -l RunLocalTests -w 60 -d ./testoutput -u myAlias

bzacharski · 2022-06-24T09:12:07Z

I've just checked it on 7.156.1 and the issue still exist. @WillieRuemmele were you able to check / reproduce it?

mshanemc · 2022-07-14T13:57:02Z

I dug into this with the person who added all the fancy new coverage stuff to deploy commands and I'm attempting to summarize:

deploy and test actually do slightly different things when they hit the server, and report back different information, which is why they won't agree
deploy results give us back only the uncovered lines and the number of lines...so we are 100% cheating and saying "that which is not uncovered must be covered" which, like you saw is different from apex:test:run results with respect to how blank lines and comments are treated.
We have a longer term plan to do this the right way (changing how the server-side stuff works) but those types of changes

have to be done by somebody else
have to be done as part of the major release cycle
are pretty complicated

I know it sucks, but I'm going to close this and call it "working as designed" with full sympathy that the design isn't great.

nvuillam · 2023-02-15T10:48:08Z

Meanwhile, here is a workaround for those who need to check code coverage while calling force:source:deploy :)

https://nicolas.vuillamy.fr/test-apex-tests-org-wide-coverage-while-simulating-salesforce-deployments-bb738130480f

mcarvin8 · 2024-03-06T20:16:42Z

I dug into this with the person who added all the fancy new coverage stuff to deploy commands and I'm attempting to summarize:

deploy and test actually do slightly different things when they hit the server, and report back different information, which is why they won't agree

deploy results give us back only the uncovered lines and the number of lines...so we are 100% cheating and saying "that which is not uncovered must be covered" which, like you saw is different from apex:test:run results with respect to how blank lines and comments are treated.

We have a longer term plan to do this the right way (changing how the server-side stuff works) but those types of changes

have to be done by somebody else

have to be done as part of the major release cycle

are pretty complicated

I know it sucks, but I'm going to close this and call it "working as designed" with full sympathy that the design isn't great.

@mshanemc - Is there any updates to the above in regards to fixing how the deploy commands (sf project deploy at this point) report back code coverage, specifically "covered" lines? @xL3o and I have exhausted workarounds with "covered" lines and fed that to SonarQube for coverage. The "uncovered" lines are correctly reported by the deploy command as stated previously and I'm currently using that alone to feed SonarQube via my plugin. However, SonarQube requires "covered" and "uncovered" lines to calculate Coverage %. So my plugin can correctly set "uncovered" lines in SonarQube but cannot set "covered" lines to calculate coverage %.

My plugin just converts the sf project deploy code coverage JSON output into an XML that SonarQube will accept. We have figured out the CLI's JSON output will print extra lines as "covered" (ex: line 100 as "covered" if the apex class is only 95 lines of code) and lines. So there's no reliable way to get the "covered" line count accurate for the code coverage %, even after I basically mapped out-of-range "covered" lines to unused lines in-range.

Basically, I can't update my plugin to report "covered" lines until the Salesforce CLI is updated to correctly return "covered" lines in the coverage JSON output.

Ref: mcarvin8/apex-code-coverage-transformer#2

bzacharski added the investigating We're actively investigating this issue label Jun 12, 2022

mshanemc closed this as completed Jul 14, 2022

mcarvin8 mentioned this issue Mar 4, 2024

Code Coverage is falsely calculated in SonarQube mcarvin8/apex-code-coverage-transformer#2

Open

mcarvin8 mentioned this issue Mar 19, 2024

Code Coverage Reports in Deploy Commands Produce Unreliable Covered Lines forcedotcom/salesforcedx-vscode#5511

Open

mcarvin8 mentioned this issue May 9, 2024

The file name {n} was not found in any package directory. mcarvin8/apex-code-coverage-transformer#13

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Invalid code coverage report using sfdx force:source:deploy and force:mdapi:deploy #1568

Invalid code coverage report using sfdx force:source:deploy and force:mdapi:deploy #1568

bzacharski commented Jun 12, 2022 •

edited

github-actions bot commented Jun 12, 2022

bzacharski commented Jun 12, 2022

WillieRuemmele commented Jun 13, 2022

bzacharski commented Jun 13, 2022

bzacharski commented Jun 24, 2022

mshanemc commented Jul 14, 2022 •

edited

nvuillam commented Feb 15, 2023

mcarvin8 commented Mar 6, 2024 •

edited

Invalid code coverage report using sfdx force:source:deploy and force:mdapi:deploy #1568

Invalid code coverage report using sfdx force:source:deploy and force:mdapi:deploy #1568

Comments

bzacharski commented Jun 12, 2022 • edited

Summary

Steps To Reproduce:

Expected result

Actual result

System Information

Additional information

github-actions bot commented Jun 12, 2022

bzacharski commented Jun 12, 2022

WillieRuemmele commented Jun 13, 2022

bzacharski commented Jun 13, 2022

bzacharski commented Jun 24, 2022

mshanemc commented Jul 14, 2022 • edited

nvuillam commented Feb 15, 2023

mcarvin8 commented Mar 6, 2024 • edited

bzacharski commented Jun 12, 2022 •

edited

mshanemc commented Jul 14, 2022 •

edited

mcarvin8 commented Mar 6, 2024 •

edited