Skip to content

Modify MRT Core UnpackagedTests clean-up, and reorganize test reporting#870

Merged
jefgen merged 16 commits intomainfrom
user/jefgen/mrtcore-unpackagedtest-changes
May 27, 2021
Merged

Modify MRT Core UnpackagedTests clean-up, and reorganize test reporting#870
jefgen merged 16 commits intomainfrom
user/jefgen/mrtcore-unpackagedtest-changes

Conversation

@jefgen
Copy link
Copy Markdown
Member

@jefgen jefgen commented May 26, 2021

This PR makes changes to the MRT Core UnpackagedTests to unblock running on my local PC, and also improves how the tests are reported in the Azure DevOps UI.

Unblocking the tests on my PC

This change adds what is effectively a work-around to get the test working on my PC. For some reason I have issues with the test clean-up when it attempts to delete the copied resources.pri file.

Hui wasn't able to reproduce this -- yet I consistently hit this issue every time I run the tests. (Thanks to Hui for trying to repro this).
FWIW, the error that I was seeing when trying to run the tests locally on my PC is similar to the following:

StartGroup: MrtCoreUnpackagedTests.TestClass.DefaultResourceManagerWithResourcePri
Verify: AreNotEqual(1269, 0)
Verify: AreEqual(Groove Music, Groove Music)
Error: System.UnauthorizedAccessException: Access to the path 'D:\code\ProjectReunion\BuildOutput\Release\x64\UnpackagedTests\resources.pri' is denied.
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.File.InternalDelete(String path, Boolean checkHost)
   at MrtCoreUnpackagedTests.TestClass.DefaultResourceManagerWithResourcePri()
EndGroup: MrtCoreUnpackagedTests.TestClass.DefaultResourceManagerWithResourcePri [Failed]
 
Summary of Non-passing Tests:
    MrtCoreUnpackagedTests.TestClass.DefaultResourceManagerWithResourcePri [Failed]

The access denied error message is somewhat misleading, as the real error is STATUS_CANNOT_DELETE from KERNELBASE!DeleteFileW.
(This gets converted to an access denied error). I'm not sure what causes the issue as MRT Core opens the file with FILE_SHARE_DELETE. The file isn't read-only, and the user should have permissions to delete the file.

I thought perhaps it was due to Windows Defender -- but adding exceptions to Windows Defender didn't help.
I tried excluding the folder (UnpackagedTests), the file (resources.pri), and the process (te.processhost.exe), but none of them had any effect. Also adding various sleep()s to the test didn't help either. (I wasn't sure if maybe there was a timing issue or not).

The other work-around that worked was to run each test method in isolation by adding: [TestProperty("IsolationLevel", "Method")] to each test method. However, this slows the test down somewhat.

This change moves the delete to a Clean up method, and calls GC.Collect() beforehand.

Azure DevOps Test Reporting

This change also reorganizes the MRT Core tests so that they show up with their own label in the Azure DevOps Test reporting UI. Before all the test results were all munged together -- which makes it hard to tell when a test fails where it is from.

The test reporting tab now looks like this:
image

And the CI pipeline has separate steps for each test:
image

Note: I can't seem to get the MRT Core UnpackagedTests to work in the CI pipeline yet. I'm still trying to figure out what is the root cause.

@ghost ghost added the needs-triage label May 26, 2021
@jefgen jefgen merged commit 51d5fe2 into main May 27, 2021
@jefgen jefgen deleted the user/jefgen/mrtcore-unpackagedtest-changes branch May 27, 2021 02:18
DrusTheAxe pushed a commit that referenced this pull request Jun 14, 2021
…ng (#870)

This change modifies the MRT Core UnpackagedTests to unblock running on my local PC.
It also improves how the tests are reported in the Azure DevOps UI.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants