Visual Studio Test task fails with System.IO.FileNotFoundException on temp file when run Impacted tests is enabled.

[snew3687]

Type: Bug

Task Name: Visual Studio Test

Environment

• Server - Azure Pipelines

• Agent - Hosted:

  • OS: Windows Server 2016 Standard

  • Visual Studio Version: 2019

  • Agent Version: 2.155.1

Issue Description

The task has started failing every time it runs. It started failing when the pipeline auto-updated the task from version 2.159.1 to version 2.160.2.

The log for this task shows it failing with a System.IO.FileNotFoundException:

Task logs

logs_15713.zip

Error logs

##[error]Error occured while executing System.IO.FileNotFoundException: Could not find file 'E:\agentG2\_work\_temp\m0umztbldww.tmp'. File name: 'E:\agentG2\_work\_temp\m0umztbldww.tmp' at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath) at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost) at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost) at System.IO.StreamReader..ctor(String path, Encoding encoding, Boolean detectEncodingFromByteOrderMarks, Int32 bufferSize, Boolean checkHost) at System.IO.StreamReader..ctor(String path, Encoding encoding) at System.IO.File.InternalReadAllLines(String path, Encoding encoding) at MS.VS.TestService.VstestConsoleAdapter.TestSelectorCommandInputs.GetTestsList(String listTestsFile) at MS.VS.TestService.VstestConsoleAdapter.TestSelectorCommandInputs.GetImpactedTestsCommandInput(VstestConsoleRunContext testRunContext) at MS.VS.TestService.VstestConsoleAdapter.ExecutionStateResolver.GetNextPhase(VstestConsoleRunContext consoleRunContext, ITestRunPhase currentPhase, ITestRunPhaseResults phaseResult) Completed TestExecution Model... ##[warning]Vstest failed with error. Check logs for failures. There might be failed tests. ##[error]Error: The process 'E:\agentG2\_work\_tasks\VSTest_ef087383-ee5e-42c7-9a53-ab56c98420f9\2.160.2\Modules\DTAExecutionHost.exe' failed with exit code 1 ##[error]Vstest failed with error. Check logs for failures. There might be failed tests. ##[section]Finishing: VsTest - testAssemblies x86

[garrettloy]

Running into the same error. I have 6 test tasks in a pipeline and this was happening to two of them - the ones that had "Run only impacted tests" checked. Unchecked that and now the test tasks don't error out.

[snew3687]

Running into the same error. I have 6 test tasks in a pipeline and this was happening to two of them - the ones that had "Run only impacted tests" checked. Unchecked that and now the test tasks don't error out.

I tried unchecking that box on my test task...and my pipeline runs OK. That is a good enough workaround for me. Thanks for providing the suggestion.

[Spikrit]

We have this problem too, only since today...
Task is 2.160.2 but i can't be sure about the previous version.

We never had any issue with the checked option "Run only impacted tests" before today.

The temporary workaround will do but a quick fix or rollback would be great!

[dkmiller]

We are encountering this problem as well.

[liliangc2]

We get the same error

[KevinCust]

FYI - I reached out to @AzureDevOps on twitter yesterday about this. Here is their most recent response:

Thanks for sharing this with us! We noticed that some other users are experiencing a similar problem and that you shared a considerable amount of details, including the error log. Our engineers will reach out rather swiftly to look further into this. We appreciate your help.

[Arjan321]

We are facing the same issue, since the auto-upgrade from 2.159.1 to 2.160.2.
We are getting System.IO.FileLoadException loading our CLI/C++ assemblies from C# test projects.

[goozdev]

We have faced the same issue, since the auto-upgrade from 2.159.1 to 2.160.2. We have found the reason: the generated .runsettings file by the task uses single '\' instead of '\\'. See attached files.

Using the settings file with '\\' solved the issue for us.

runsettings_2.159.1.runsettings.txt
runsettings_2.160.2.runsettings.txt

[ShreyasRmsft]

I believe we have a regression when "run only impacted tests" is selected. We will get the RCA at the soonest and hotfix the task in the next few days. Stay tuned here for updates.

[ShreyasRmsft]

As for

We have faced the same issue, since the auto-upgrade from 2.159.1 to 2.160.2. We have found the reason: the generated .runsettings file by the task uses single '' instead of '\'. See attached files.

We have already hotfixed this and the update should reach all accounts in the next 2-3 days.

[ShreyasRmsft]

I have identified the issue and have made the fix. Will post the hotfix deployment status here once i start the process

[KevinCust]

If something like this occurs in the future, is there a way we can force the pipeline to use a specific previous minor version of the task?

In this instance, making it use 2.159.1 would have been an easy workaround until the hotfix version was published.

[ShreyasRmsft]

@KevinCust this is something we've actually thought about before.

There are a few limitations though, a simple roll back is not possible. As of today if i wanted to rollback i would have take the old task package change the task version to something greater than what is present and deploy a hotfix the make the change. And this generally can be completed in 4-5 hours tops. And we have done this once or twice when the break was so complex that it would take us a day or two to figure out the root cause.

In this case the change required was quite literally 3 lines of code, (and of course a few reactionary items to go fix our telemetry so that we can auto detect a similar issue in the future) but we were able to make the fix in a very short amount of time and i am starting the deployment as i am typing this out and will update the status here.

[ShreyasRmsft]

This has been hotfixed for all accounts. You will find the fix in task version 2.160.4 or 2.161.2 and above. Please let me know if you are able to turn on test impact and get things going again. And thank for you bringing this to our notice.

Please note that by adding the "Area: Test" tag to the issue you can get the issue routed to us much sooner and reduce the delay for a response.

[Spikrit]

I gave it a go and can confirm v2.160.4 works as expected while looking for impacted tests again :

***************************** Section 'Get Impacted Tests' Starting *****************************
V, 2019/11/20, 15:06:05.737, PERF: CreateTestRun: took 153.1381 ms
V, 2019/11/20, 15:06:05.737, PERF WARNING: CreateTestRun: took 153.1381 ms
V, 2019/11/20, 15:06:05.737, Tests that are discovered.

...

V, 2019/11/20, 15:06:07.018, Published Test Run : 1013592
***************************** Section 'Update test results' Ended ********************************

[ShreyasRmsft]

Apologies again for the break and thank you everyone for reporting this and being patient while we patched it up.