Run Functional Tests: The test run was aborted, failing the task.

[yorhpokie]

Environment

• TFS 2017 Update 3 on-premise

• Private Agent:

  • Win10 Enterprise 1703 OS Build: 15063.909
  • Agent Version: 2.117.2

• Private Test Agents:

  • Win10 Enterprise 1709 OS Build 16299.248
  • Deploy Test Agent task version 1.0.42
  • Run Functional Tests task version 1.0.52

Issue Description

Run Functional Tests task fails periodically with "The test run was aborted, failing the task.
The DTAExecutionHost.exe.log shows this exception:

E, 4440, 16, 2018/03/19, 14:34:14.104, TV3-AGNT-0909\DTAExecutionHost, TestExecutionServiceCommandQueueListener.SendStopCommandToHandleException : An exception occured in queue listener. System.Threading.Tasks.TaskCanceledException: A task was canceled.
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
at Microsoft.VisualStudio.TestService.AgentExecutionHost.TestExecutionServiceCommandQueueListener.d__7.MoveNext() Sending Stop command...

Error logs

2018-03-19T21:34:00.5677513Z DistributedTests: Test run '4630725' is in 'InProgress' state.
2018-03-19T21:34:00.5677513Z DistributedTests: Total Tests : 586, Passed Tests : 513
2018-03-19T21:34:10.6808466Z DistributedTests: Test run '4630725' is in 'Aborted' state.
2018-03-19T21:34:10.6808466Z DistributedTests: Total Tests : 586, Passed Tests : 513
2018-03-19T21:34:20.6843066Z ##[warning]DistributedTests: Test run is aborted. Logging details of the run logs.
2018-03-19T21:34:20.6843066Z ##[debug]Processed: ##vso[task.logissue type=warning]DistributedTests: Test run is aborted. Logging details of the run logs.
2018-03-19T21:34:20.7155519Z ##[warning]DistributedTests: New test run created.
2018-03-19T21:34:20.7155519Z ##[debug]Processed: ##vso[task.logissue type=warning]DistributedTests: New test run created.
2018-03-19T21:34:20.7155519Z Test Run queued for Project Collection Build Service (TEAM FOUNDATION).
2018-03-19T21:34:20.7155519Z
2018-03-19T21:34:20.7155519Z ##[warning]DistributedTests: Test execution started. Test run id : 4630725
2018-03-19T21:34:20.7155519Z ##[debug]Processed: ##vso[task.logissue type=warning]DistributedTests: Test execution started. Test run id : 4630725
2018-03-19T21:34:20.7311857Z ##[warning]DistributedTests: Test run timed out. Test run id : 4630725
2018-03-19T21:34:20.7311857Z ##[debug]Processed: ##vso[task.logissue type=warning]DistributedTests: Test run timed out. Test run id : 4630725
2018-03-19T21:34:20.7311857Z ##[warning]DistributedTests: Test run aborted. Test run id: 4630725
2018-03-19T21:34:20.7311857Z ##[debug]Processed: ##vso[task.logissue type=warning]DistributedTests: Test run aborted. Test run id: 4630725
2018-03-19T21:34:20.7311857Z ##[warning]DistributedTests: Test run timed out. Test run id : 4630725
2018-03-19T21:34:20.7311857Z ##[debug]Processed: ##vso[task.logissue type=warning]DistributedTests: Test run timed out. Test run id : 4630725
2018-03-19T21:34:20.7311857Z ##[warning]DistributedTests: Test run timed out. Test run id : 4630725
2018-03-19T21:34:20.7311857Z ##[debug]Processed: ##vso[task.logissue type=warning]DistributedTests: Test run timed out. Test run id : 4630725
2018-03-19T21:34:20.7311857Z ##[warning]DistributedTests: Test run timed out. Test run id : 4630725
2018-03-19T21:34:20.7311857Z ##[debug]Processed: ##vso[task.logissue type=warning]DistributedTests: Test run timed out. Test run id : 4630725
2018-03-19T21:34:20.7311857Z ##[error]System.Exception: The test run was aborted, failing the task.
2018-03-19T21:34:20.7311857Z ##[debug]Processed: ##vso[task.logissue type=error;]System.Exception: The test run was aborted, failing the task.
2018-03-19T21:34:20.7780676Z ##[debug]DistributedTests: Deleted test settings with id : 1052771
2018-03-19T21:34:20.8718067Z ##[error]PowerShell script completed with 1 errors.
2018-03-19T21:34:20.8718067Z ##[section]Finishing: Run Tests

DTAExecutionHost.exe.log

[ShreyasRmsft]

@yorhpokie This looks like a known issue we had in DTA RFT. As of now in the latest Tfs (2018 Update 2) the DTA RFT task has been deprecated. We are asking folks to move to the VsTest v2 task that is capable of parallel execution both in release and build (But in Tfs 2017 update 3 only parallel execution releases is supported). Are you guys running tests in Build or Release?

[akzhn]

It always happens when RunTests task runs for 6 hours.

Setting up 'TestSessionTimeout' in .runsettings not working, while .testsetting (with 'runTimeout' param inside) is not supported by MsTestV2.

[ShreyasRmsft]

@nshr12 The timeout is not for MsTestV2 but for the RFT task.
You should be able to specify runTimeout with .testsettings and it should override the default 6 hour timeout of the RFT task.

Just override your preferred timeout value in the testsettings below

<?xml version="1.0" encoding="UTF-8"?>
<TestSettings name="ci" id="adc5287b-c979-4581-b076-044af81d138b" xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010">
  <Description>These are default test settings for a ci run.</Description>
  <Deployment enabled="false" />
  <Execution>
    <Timeouts runTimeout="36000000" testTimeout="0" />
    <TestTypeSpecific>
      <UnitTestRunConfig testTypeId="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b">
        <AssemblyResolution applicationBaseDirectory="c:\tests" />
      </UnitTestRunConfig>
    </TestTypeSpecific>
    <AgentRule name="LocalMachineDefaultRole">
    </AgentRule>
  </Execution>
  <Properties />
</TestSettings>

[akzhn]

@ShreyasRmsft , thanks, but how can I use both .runsettings and .testsettings at the same time?

As I sad above, .testsettings is not supported by mstestv2..

[yorhpokie]

Thank you for the reply, @ShreyasRmsft! We're running tests in Build. I guess we'll just retry the tests until we upgrade to TFS 2018.

[ShreyasRmsft]

@nshr12 Sadly there's no way to use both test settings and run settings together. In VSTS we're moving to replace testsettings with runsettings and everything that was supported with test settings will be supported by runsettings.

[ShreyasRmsft]

@yorhpokie Was your issue also with 6 hour timeouts or arbitrary duration timeouts? If they were 6 hour timeouts you can add more agents to reduce the overall execution time to less than 6 hours or divide the tests into 2 halves and run them via 2 instances of the tasks. So that each run happens within 6 hours as a workaround. @nshr12 you can do this too.

[yorhpokie]

@ShreyasRmsft: our tests were aborted after 30 or 40 minutes. It was definitely not 6 hours.

[ShreyasRmsft]

@yorhpokie the arbitrary timeout is known issue I was talking about. We fixed it in 2018 Update 1. The 6 hour timeout was configurable always just that runsettings support was added later.