-
Notifications
You must be signed in to change notification settings - Fork 2.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
vstest output path changed recently #7975
Comments
@baywet That said, you can control where your coverage file gets generated by specifying the results directory in run settings file.https://blogs.msdn.microsoft.com/aseemb/2013/09/14/how-to-specify-results-directory-with-vstest-console/ |
+1 to what @prawalagarwal said. that's the location layout of our code. |
Thanks for the answer. If I'm not mistaking the runsettings file only accepts full paths and not relatives correct? (and the build working path is dynamic by nature) |
You can use variables to supply a full path that's not hardcoded. https://docs.microsoft.com/en-us/vsts/pipelines/build/variables?view=vsts The agent replaces those for you. For example $(System.DefaultWorkingDirectory) |
thanks for that answer @bryanmacfarlane |
Tagging @PBoraMSFT for the feedback on providing it as a task option. Prachi whats your take on this? |
@acesiddhu @baywet - Trying to add a setting in the task for every single config will simply bloat it. We can consider enhancing the override test run parameters as an easier means to override values in runsettings - but that would be an enhancement and not a bug fix. Marking this issue as such. |
thanks for the answer @PBoraMSFT ! Yes I think there might be other settings that are not currently exposed that people might want to provide to the task/process. A generic parameters override section would probably be better. |
* Add gitversion * Setup sonarcloud directory for vsts test output (due to bug microsoft/azure-pipelines-tasks#7975)
* Add gitversion * Setup sonarcloud directory for vsts test output (due to bug microsoft/azure-pipelines-tasks#7975) * Enable code coverage * Add variables to yml files for easier setup
Any progress on this? @prawalagarwal you say that the there is no documented , but the documentation clearly states that the variable "Common.TestResultsDirectory" should point to the "The local path on the agent where the test results are created." and the trx and coverage files are the result of the test task. I ve been trying to get the sonarcloud extension working again for the last week after you changed the behaviour without finding a solution I am satisfied with. i've been trying to look for both to look in the temp directory as well as copy the output back to the correct directory, but in the latter case the .trx file is not found. copy step can be found here but as you can se in the build log it does not find the trx file, do you delete it after publish? Can you provide an example of how to access the trx file? As long as you don't follow the expected outcome and place the test results in the test results directory we really need an easy way to get back the expected behavior. The best solution would clearly be to put the test results under the test result directory (even if you need separate subfolders). If you really cannot do that you should provide an easy to use setting to store test results in the test results directory ("redirect test result to agent temp folder" or similar). |
Hi,
|
Closing this now. Feel free to reopen if needed or open another one in case you hit into other issues. |
@bryanmacfarlane the agent doesn't seem to be replacing variables. I tried this in my runsettings file: <RunConfiguration>
<ResultsDirectory>$(System.DefaultWorkingDirectory)\TestResults</ResultsDirectory>
</RunConfiguration> But in the logs I see:
and this from the updated runsettings file: <ResultsDirectory>d:\a\_temp\$(System.DefaultWorkingDirectory)\TestResults\TestResults</ResultsDirectory> Is there some way to access the generated trx files? At present it seems like it's impossible to access them in a Publish Test Results task, because the trx files are always dumped in a temp folder that's cleared after every task! @kaadhina pointed out that there are APIs to access the attachments, but that's going to be a major PITA |
@cocowalla may I know why you need access to the test result files when the vstest task already uploads the test results for you?
This is not true by the way. The agent clears the temp folder at the end of the pipeline and not after each task. |
@ShreyasRmsft for 2 different reasons:
|
Fair enough. You can find the trx and the coverage files in the $(Agent.TempDirectory)/TestResults The reason we started putting them in temp directory is two fold:
This left us a choice between using Common.TestResultsDirectory or Agent.TempDirectory. The distributed flow and re-run flows of the task (non legacy code) have always been using the temp directory and it made sense for the task experience to be uniform across all these flows. Another reason we leaned towards Agent.TempDirectory is because these files we generate are intermediate and not a well documented contract (We did stop the task code from cleaning up the results when he heard that folks have taken a dependency on it though, the results should be present in the temp directory till the build is complete). The recommended way is to get these metrics via the well documented apis. This experience can simplified if the SonarQube task (and others if any) could get these attachments from the well-documented apis for you, thus ensuring this is a contract between the tasks that will be honored and customers will not have to bother about the internal implementation of where the task stores it's intermediate result files) |
We are in the process of initiating conversations with the SonarQube folks to see what we can do towards this end. |
Few alternative options I see:
Yes, but that is a major PITA - it's a lot of effort to just access some files that you'd expect to be easily accessible.
I'd politely request that you don't get carried away because I specifically mentioned SonarQube - this is a more general requirement, not specific to a single SAST tool or usage; for example, I also mentioned the ReportGenerator task. Ultimately, the best option for customers is to simply make the files easily accessible in the file system. |
Option 3 ie. exposing this input in the task is something we are considering and I should be able to get back to you on this very soon. As for not finding the files in the temp folder after the task: May I know what version of the VsTest task you are running? Also let me know the version of the azure-pipelines agent you are running the build on. Providing the full zip logs (obtained by clicking on download all logs in the build summary page) with system.debug = true should help me get to the bottom of this very quickly. |
Any further update on this @ShreyasRmsft ? I'm interested in having this available, i would like to post summary from trx test results to teams |
Environment
Server - VSTS or TFS on-premises?
If using TFS on-premises, provide the version:
If using VSTS, provide the account name, team project name, build definition name/build number: 2tolead, <team project name contains customer name, can provide privately>, build definition id 32, build id 3998 (working), 4191 (non working)
Agent - Hosted or Private: Hosted VS2017
Issue Description
Recently the output path of the vstest coverage file change from
D:\a\1\s\TestResults\63a93925-fc5e-4a09-9725-0a1850687297\VssAdministrator_factoryvm-az361 2018-07-18 13_20_02.coverage
toD:\a\_temp\TR_47834234-9496-460e-b9c3-e66d33e8c426\8d2bd737-fbf2-4323-b34c-26dd8f45cba4\VssAdministrator_factoryvm-az389 2018-08-08 16_01_27.coverage
.Not only this is not located within the build directory, it also broke subsequent scripts in our pipeline (calling codecoverage.exe to convert to xml and later import to sonarqube).
To my knowledge there's no way to specify the output path other than setting the
RunSettings.RunConfiguration.ResultsDirectory
node in the runsettings file. However setting that value would require extra scripts and tokenization.My ask: please add an output path configuration for the task relative to the build folder.
Error logs
not relevant, it's our subsequent PowerShell task failing because it doesn't find the file at the right spot.
The text was updated successfully, but these errors were encountered: