Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

VSTest task saves test results to wrong TestResults directory #2745

Closed
harmpauw opened this issue Oct 3, 2016 · 8 comments
Assignees
Labels

Comments

@harmpauw
Copy link

@harmpauw harmpauw commented Oct 3, 2016

Hi all,

I've noticed that since a couple of weeks, Unit Tests results and Code Coverage files are no longer picked up by the SonarQube Scanner when running a build on VSTS using Hosted Agents. When I looked in the log, I saw that SonarQube can't find any results files:

2016-10-03T09:58:06.3589980Z SonarQube Scanner for MSBuild End Step 2.1 2016-10-03T09:58:06.3589980Z 09:58:03.798 Fetching code coverage report information from TFS... 2016-10-03T09:58:06.3589980Z 09:58:03.799 Attempting to locate a test results (.trx) file... 2016-10-03T09:58:06.3599982Z 09:58:03.799 No test results files found

I found out that SonarQube is looking in the $(Common.TestResultsDirectory) for files (e.g. c:\a\1\TestResults), while the results and coverage files are written to $(build.SourcesDirectory)\TestResults (e.g. c:\a\1\s\TestResults). I expected them to be written to the Common.TestResultsDirectory.

It seems like the change in the VSTest task responsible for this was introduced by this commit. Line 159, which reads Common.TestResultsDirectory was deleted and replaced by this line.

    $testResultsDirectory = Get-TaskVariable -Context $distributedTaskContext -Name "Common.TestResultsDirectory"

    if([string]::IsNullOrEmpty($testResultsDirectory))
    {
        $testResultsDirectory = $workingDirectory + [System.IO.Path]::DirectorySeparatorChar + "TestResults"
    }`

Could you please look into this? Thanks!

@allendm-msft

This comment has been minimized.

Copy link
Member

@allendm-msft allendm-msft commented Oct 4, 2016

@harmpauw I would simply pick the test run and download the trx instead of scanning it locally since the task isnt promising to retain it in place either. If you still want to do it locally you can either a) pick the directory from code (fragile) or b) do a file scan (non optimal)

In general I would recommend against taking dependencies on the internal implementations of the task.

@harmpauw

This comment has been minimized.

Copy link
Author

@harmpauw harmpauw commented Oct 4, 2016

@allendm-msft Thanks for you reply. After some more digging. I found the problem: The SonarQube Scanner for MSBuild had a hard dependency to the location of the TestResults directory (see this pull request). They've fixed it in master a couple of days ago but it's not released yet.

For others that have the same problems and read this thread: What I've done for now a a temporary hack is I added a Copy Files task to copy the test results to the directory that the SonarQube Scanner for MSBuild expects by copying from $(build.SourcesDirectory)/TestResults to $(Agent.BuildDirectory)/TestResults (for now the same as $(Common.TestResultsDirectory)). I inserted this task between the Test Assemblies task and the Finish analysis task of SonarQube.

That leaves me with one question: Shouldn't the VSTest task (or another one responsible for this) set the $(Common.TestResultsDirectory) variable to the right location and if not, for what purpose should we use it since this is a publicly documented variable?

@allendm-msft

This comment has been minimized.

Copy link
Member

@allendm-msft allendm-msft commented Oct 5, 2016

@bryanmacfarlane any recommendation on this variable? Should the test task be setting this?
We didnt want to update it with the path we use, assuming that there would be someone taking a dependency on the existing value already.

@allendm-msft

This comment has been minimized.

Copy link
Member

@allendm-msft allendm-msft commented Oct 10, 2016

This is a variable set by the Agent currently and not being updated by the test task. We don't intend to change this to avoid breaking others who have taken dependencies on it as-is. If we introduce a similar variable it will be posted in the docs.

@mack0196

This comment has been minimized.

Copy link

@mack0196 mack0196 commented Oct 12, 2016

Why doesn't vstest create the test results in $(Common.TestResultsDirectory)? I don't understand why the same breaking consideration wasn't considered when changing the TestResults directory.

@allendm-msft

This comment has been minimized.

Copy link
Member

@allendm-msft allendm-msft commented Oct 13, 2016

@mack0196 let me try to explain. The change to the location where the testresults are placed by the vstest task is internal to the task. This location was not published as a output/common variable to be consumed by external tasks. Since another task made the assumption that this location wont change they broke.
This is pretty much similar to using non documented apis.

@bobylevm

This comment has been minimized.

Copy link

@bobylevm bobylevm commented Jan 24, 2017

@allendm-msft $(Common.TestResultsDirectory) is published as output/common variable at visualstudio.com: https://www.visualstudio.com/en-us/docs/build/define/variables

@jerryjoseph-net

This comment has been minimized.

Copy link

@jerryjoseph-net jerryjoseph-net commented Apr 11, 2017

@allendm-msft since VSTest is a VSTS task it has to respect the VSTS variable $(Common.TestResultsDirectory) IMO

No need to update it with the value you use. You just need to use that value instead of ignoring it. This was the behavior until the recent change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
7 participants
You can’t perform that action at this time.