Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

VSTS build release agent unable to run Coded UI tests when using the Nuget VsTest platform #1353

Closed
mjethwa opened this issue Dec 29, 2017 · 4 comments
Closed

VSTS build release agent unable to run Coded UI tests when using the Nuget VsTest platform #1353

mjethwa opened this issue Dec 29, 2017 · 4 comments
Labels
enhancement question

Comments

@mjethwa
Copy link

mjethwa commented Dec 29, 2017

I'm having issues running Coded UI tests using the Nuget VsTest platform on a VSTS build agent, I don't think this is a licensing restriction as I have managed to get it working by independently installing a Visual Studio Test agent (https://docs.microsoft.com/en-us/visualstudio/test/lab-management/install-configure-test-agents) on the machine and then using the vstest.console.exe in it's path to run tests.

Steps to reproduce

In a VSTS agent running on Windows run the following steps on the agent phase:

  • Visual Studio Test Platform Installer (gets the nuget package so that a Visual Studio install is not required on the agent machine)
  • Visual Studio Test - target Coded UI test dlls created using VS 2017 Enterprise

Expected behavior

Should be able to run Coded UI tests

Actual behavior

Unable to find test framework dlls required to run the tests - The step also ignores the test steps and reports success

Diagnostic logs

Logs from the Visual Studio Test step:
2017-12-29T13:56:17.0872849Z ##[section]Starting: VsTest - testAssemblies
2017-12-29T13:56:17.0872849Z ==============================================================================
2017-12-29T13:56:17.0872849Z Task : Visual Studio Test
2017-12-29T13:56:17.0872849Z Description : Run tests with Visual Studio test runner
2017-12-29T13:56:17.0872849Z Version : 2.3.4
2017-12-29T13:56:17.0872849Z Author : Microsoft Corporation
2017-12-29T13:56:17.0872849Z Help : More Information
2017-12-29T13:56:17.0872849Z ==============================================================================
2017-12-29T13:56:21.5800849Z Run the tests locally using vstest.console.exe
2017-12-29T13:56:21.5800849Z ========================================================
2017-12-29T13:56:21.5800849Z Test selector : Test assemblies
2017-12-29T13:56:21.5800849Z Test assemblies : *test.dll,!*TestAdapter.dll,!*\obj**
2017-12-29T13:56:21.5800849Z Test filter criteria : null
2017-12-29T13:56:21.5800849Z Search folder : E:\Temp\VsTests\Acme
2017-12-29T13:56:21.5800849Z Run settings file : E:\VstsAgent_work\r1\a
2017-12-29T13:56:21.5800849Z Run in parallel : false
2017-12-29T13:56:21.5800849Z Run in isolation : false
2017-12-29T13:56:21.5800849Z Path to custom adapters : null
2017-12-29T13:56:21.5800849Z Other console options : null
2017-12-29T13:56:21.5800849Z Code coverage enabled : false
2017-12-29T13:56:21.5800849Z Rerun failed tests: false
2017-12-29T13:56:23.4208849Z VisualStudio version selected for test execution : toolsInstaller
2017-12-29T13:56:24.2632849Z ========================================================
2017-12-29T13:56:25.1212849Z [command]E:\VstsAgent_work_tool\VsTest\15.6.0-preview-20171211-02\x64\tools\net451\Common7\IDE\Extensions\TestPlatform\vstest.console.exe @e:\VstsAgent_work_temp\0cab39c1-eca0-11e7-9a65-c50660987fc7.txt
2017-12-29T13:56:25.1212849Z ##[warning]Windows Error Reporting DontShowUI not set, if the windows error dialog pops-up in the middle of UI test execution than the test will hang
2017-12-29T13:56:25.1212849Z Microsoft (R) Test Execution Command Line Tool Version 15.6.0-preview-20171211-02
2017-12-29T13:56:25.1212849Z Copyright (c) Microsoft Corporation. All rights reserved.
2017-12-29T13:56:25.1212849Z
2017-12-29T13:56:25.1992849Z vstest.console.exe
2017-12-29T13:56:25.1992849Z "E:\Temp\VsTests\Acme\CodedUITestProject1.dll"
2017-12-29T13:56:25.1992849Z "E:\Temp\VsTests\Acme\XyzPlc.Acme.Tests.AcmePhoneChatWebApp.UITests.dll"
2017-12-29T13:56:25.1992849Z /logger:"trx"
2017-12-29T13:56:30.1444849Z Starting test execution, please wait...
2017-12-29T13:56:34.4812849Z Unable to load types from the test source 'E:\Temp\VsTests\Acme\CodedUITestProject1.dll'. Some or all of the tests in this source may not be discovered. If you are running unit tests for Windows Store apps and referencing custom WinMD type then please visit http://go.microsoft.com/fwlink/?LinkId=238340 for more info. Error details: System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.VisualStudio.TestTools.UITesting, Version=15.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.
2017-12-29T13:56:34.4812849Z .
2017-12-29T13:56:34.4968849Z MSTestAdapter failed to discover tests in class 'CodedUITestProject1.CodedUITest1' of assembly 'E:\Temp\VsTests\Acme\CodedUITestProject1.dll'. Reason Could not load file or assembly 'Microsoft.VisualStudio.QualityTools.CodedUITestFramework, Version=15.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified..
2017-12-29T13:56:34.6216849Z Unable to load types from the test source 'E:\Temp\VsTests\Acme\XyzPlc.Acme.Tests.AcmePhoneChatWebApp.UITests.dll'. Some or all of the tests in this source may not be discovered. If you are running unit tests for Windows Store apps and referencing custom WinMD type then please visit http://go.microsoft.com/fwlink/?LinkId=238340 for more info. Error details: System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.VisualStudio.TestTools.UITesting, Version=15.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.
2017-12-29T13:56:34.6216849Z .
2017-12-29T13:56:34.6216849Z MSTestAdapter failed to discover tests in class 'XyzPlc.Acme.Tests.AcmePhoneChatWebApp.UITests.CodedUITest1' of assembly 'E:\Temp\VsTests\Acme\XyzPlc.Acme.Tests.AcmePhoneChatWebApp.UITests.dll'. Reason Could not load file or assembly 'Microsoft.VisualStudio.QualityTools.CodedUITestFramework, Version=15.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified..
2017-12-29T13:56:34.6216849Z MSTestAdapter failed to discover tests in class 'XyzPlc.Acme.Tests.AcmePhoneChatWebApp.UITests.Categories' of assembly 'E:\Temp\VsTests\Acme\XyzPlc.Acme.Tests.AcmePhoneChatWebApp.UITests.dll'. Reason Could not load file or assembly 'Microsoft.VisualStudio.QualityTools.CodedUITestFramework, Version=15.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified..
2017-12-29T13:56:34.6216849Z MSTestAdapter failed to discover tests in class 'XyzPlc.Acme.Tests.AcmePhoneChatWebApp.UITests.Account' of assembly 'E:\Temp\VsTests\Acme\XyzPlc.Acme.Tests.AcmePhoneChatWebApp.UITests.dll'. Reason Could not load file or assembly 'Microsoft.VisualStudio.QualityTools.CodedUITestFramework, Version=15.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified..
2017-12-29T13:56:34.6216849Z MSTestAdapter failed to discover tests in class 'XyzPlc.Acme.Tests.AcmePhoneChatWebApp.UITests.Login' of assembly 'E:\Temp\VsTests\Acme\XyzPlc.Acme.Tests.AcmePhoneChatWebApp.UITests.dll'. Reason Could not load file or assembly 'Microsoft.VisualStudio.QualityTools.CodedUITestFramework, Version=15.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified..
2017-12-29T13:56:34.6996849Z No test is available in E:\Temp\VsTests\Acme\CodedUITestProject1.dll E:\Temp\VsTests\Acme\XyzPlc.Acme.Tests.AcmePhoneChatWebApp.UITests.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
2017-12-29T13:56:34.7776849Z
2017-12-29T13:56:34.7776849Z Additionally, path to test adapters can be specified using /TestAdapterPath command. Example /TestAdapterPath:.
2017-12-29T13:56:34.8088849Z ##[warning]No results found to publish.
2017-12-29T13:56:35.9164849Z ##[section]Finishing: VsTest - testAssemblies

Environment

Windows Server 2008 R2 running VSTS agent version 2.126.0
Nuget VsTest 15.6.0-preview-20171211-02 x64
@smadala
Copy link
Contributor

smadala commented Dec 29, 2017 •
edited

@mjethwa Currently Coded UI and UWP tests not support with VSTest platform nuget package.
/cc @PBoraMSFT for blog/doc and timeline for support.

@ShreyasRmsft ShreyasRmsft reopened this Jan 2, 2018
@PBoraMSFT
Copy link
Member

We seem to have not called out the limitations in the package on nuget. Filing a bug to get the package description updated. Closing this issue out as adding Coded UI support is already being tracked as a story on our backlog.

@smadala smadala mentioned this issue Jan 23, 2018
Closed
@acesiddhu acesiddhu mentioned this issue Jan 29, 2018
Merged
@smadala
Copy link
Contributor

smadala commented Mar 8, 2018

Fixed in latest preview package https://www.nuget.org/packages/Microsoft.TestPlatform/15.7.0-preview-20180307-01

@MagnusJohansson
Copy link

I am still getting this error in my Release when using the latest preview of the Microsoft.TestPlatform Nuget package.

2018-05-13T14:09:36.8713257Z Microsoft.TestPlatform.Build 15.8.0-preview-20180510-03
 
--

Warnings (1)Windows Error Reporting DontShowUI not set, if the windows error dialog pops-up in the middle of UI test execution than the test will hang |   | Windows Error Reporting DontShowUI not set, if the windows error dialog pops-up in the middle of UI test execution than the test will hang
  | Windows Error Reporting DontShowUI not set, if the windows error dialog pops-up in the middle of UI test execution than the test will hang

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement question
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@MagnusJohansson @mjethwa @smadala @PBoraMSFT @ShreyasRmsft