TabTests.cpp test cases fail to run - TAEF harness fails to run with "[HRESULT: 0x80270254] Failed to create the test host process for out of process test execution"

[metathinker]

On my computer, the test case file src\cascadia\LocalTests_TerminalApp\TabTests.cpp fails to run test setup with the above error message. I'm not sure if this is a general problem or just in my setup, but these tests don't seem to run in Azure DevOps CI (I checked test run 31597) so I can't be sure.

Environment

Windows build number: Microsoft Windows [Version 10.0.18362.295]
Windows Terminal version: current master commit 734fc1d

Steps to reproduce

1. Checkout the Terminal repo.
2. Run tools\razzle.cmd & bz rel to get a successful build.
3. Run: "%TAEF%" bin\%PLATFORM%\Release\LocalTests_TerminalApp\TerminalApp.LocalTests.dll /name:"TerminalAppLocalTests::TabTests::*"

Expected behavior

All tests pass.

Actual behavior

The test file does not run at all, with error 0x80270254 == E_APPLICATION_NOT_REGISTERED (lines manually wrapped):

X:\> "%TAEF%"
  bin\x64\Release\LocalTests_TerminalApp\TerminalApp.LocalTests.dll
  /name:"TerminalAppLocalTests::TabTests::*"

Test Authoring and Execution Framework v10.38k for x64
TAEF: Deploying custom AppXManifest...
  Copying X:\bin\x64\Release\LocalTests_TerminalApp\TerminalApp.LocalTests.AppxManifest.xml
  to X:\bin\x64\Release\LocalTests_TerminalApp\AppXManifest.xml
Warning: TAEF: Using TE.ProcessHost.exe as the Application Executable for UAP tests
  is deprecated, please consider using TE.ProcessHost.UAP.exe in your custom manifest.
Error: TAEF: [HRESULT: 0x80270254] Failed to create the test host process for
  out of process test execution. (The IApplicationActivationManager::
  ActivateApplication call failed while using a default host. TAEF's ETW logs
  which are gathered with the /enableEtwLogging switch should contain
  events from relevant providers that may help to diagnose the failure.)

StartGroup: TerminalAppLocalTests::TabTests::TryInitXamlIslands
EndGroup: TerminalAppLocalTests::TabTests::TryInitXamlIslands [Blocked]

StartGroup: TerminalAppLocalTests::TabTests::TryCreateLocalWinRTType
EndGroup: TerminalAppLocalTests::TabTests::TryCreateLocalWinRTType [Blocked]

StartGroup: TerminalAppLocalTests::TabTests::TryCreateXamlObjects
EndGroup: TerminalAppLocalTests::TabTests::TryCreateXamlObjects [Blocked]

StartGroup: TerminalAppLocalTests::TabTests::TryCreateTab
EndGroup: TerminalAppLocalTests::TabTests::TryCreateTab [Blocked]

Summary of TAEF Warnings:
    Warning: TAEF: Using TE.ProcessHost.exe as the Application Executable for UAP tests
      is deprecated, please consider using TE.ProcessHost.UAP.exe in your custom manifest.

Summary of Errors Outside of Tests:
    Error: TAEF: [HRESULT: 0x80270254] Failed to create the test host process for
      out of process test execution. (The IApplicationActivationManager::
      ActivateApplication call failed while using a default host. TAEF's ETW logs
      which are gathered with the /enableEtwLogging switch should contain
      events from relevant providers that may help to diagnose the failure.)

Summary of Non-passing Tests:
    TerminalAppLocalTests::TabTests::TryInitXamlIslands [Blocked]
    TerminalAppLocalTests::TabTests::TryCreateLocalWinRTType [Blocked]
    TerminalAppLocalTests::TabTests::TryCreateXamlObjects [Blocked]
    TerminalAppLocalTests::TabTests::TryCreateTab [Blocked]

Summary: Total=4, Passed=0, Failed=0, Blocked=4, Not Run=0, Skipped=0

Creating the ETW log and parsing it with Windows' built-in tracerpt.exe tool yields a number of interesting errors. As best as I can tell from my very limited knowledge, there was a problem with the AppX package extensions (<Extension> elements in AppXManifest.xml).

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="CombaseTraceLoggingProvider" Guid="{1aff6089-e863-4d36-bdfd-3581f07440be}" />
    <EventID>0</EventID>
    <Version>0</Version>
    <Level>2</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x0</Keywords>
    <TimeCreated SystemTime="2019-08-18T22:19:27.701635500-07:00" />
    <Execution ProcessID="8272" ThreadID="11028" ProcessorID="0" KernelTime="315" UserTime="405" />
    <Channel />
    <Computer />
  </System>
  <EventData>
    <!-- Michael: This is 0x80040154 == REGDB_E_CLASSNOTREG
         "Class not registered" -->
    <Data Name="hresult">-2147221164</Data>
    <Data Name="code"></Data>
    <Data Name="message"></Data>
    <Data Name="function"></Data>
    <Data Name="fileName">onecore\com\combase\extensioncatalog\extensioncatalog.cpp</Data>
    <Data Name="lineNumber">    1215</Data>
  </EventData>
  <RenderingInfo Culture="en-US">
    <Task>ComError</Task>
  </RenderingInfo>
</Event>

[snip snip]

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="ActivationManager" Guid="{cf7f94b3-08dc-5257-422f-497d7dc86ab3}" />
    <EventID>0</EventID>
    <Version>0</Version>
    <Level>2</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x0</Keywords>
    <TimeCreated SystemTime="2019-08-18T22:19:27.702335700-07:00" />
    <Execution ProcessID="8272" ThreadID="11028" ProcessorID="2" KernelTime="315" UserTime="405" />
    <Channel />
    <Computer />
  </System>
  <EventData>
    <Data Name="PartA_PrivTags">16777216</Data>
    <ComplexData Name="wilActivity">
      <!-- Michael: This is 0x80270254 == E_APPLICATION_NOT_REGISTERED
           "This app does not support the contract specified or is not installed." -->
      <Data Name="hresult">2150040148</Data>
      <Data Name="fileName">onecoreuap\base\appmodel\execmodel\modern\activationmanager\extensionhelpers.cpp</Data>
      <Data Name="lineNumber">     221</Data>
      <Data Name="module">activationmanager.dll</Data>
      <Data Name="failureType">       1</Data>
      <Data Name="message"></Data>
      <Data Name="threadId">   11028</Data>
      <Data Name="callContext">\ForegroundActivation</Data>
      <Data Name="originatingContextId">       8</Data>
      <Data Name="originatingContextName">ForegroundActivation</Data>
      <Data Name="originatingContextMessage"></Data>
      <Data Name="currentContextId">       8</Data>
      <Data Name="currentContextName">ForegroundActivation</Data>
      <Data Name="currentContextMessage"></Data>
      <Data Name="failureId">      32</Data>
      <Data Name="failureCount">      32</Data>
      <Data Name="function"></Data>
    </ComplexData>
  </EventData>
  <RenderingInfo Culture="en-US">
    <Task>ActivityError</Task>
  </RenderingInfo>
</Event>
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="ActivationManager" Guid="{cf7f94b3-08dc-5257-422f-497d7dc86ab3}" />
    <EventID>0</EventID>
    <Version>0</Version>
    <Level>5</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x0</Keywords>
    <TimeCreated SystemTime="2019-08-18T22:19:27.702384600-07:00" />
    <Execution ProcessID="8272" ThreadID="11028" ProcessorID="2" KernelTime="315" UserTime="405" />
    <Channel />
    <Computer />
  </System>
  <EventData>
    <!-- Michael: This is E_APPLICATION_NOT_REGISTERED
         "This app does not support the contract specified or is not installed." -->
    <Data Name="HRESULT">0x80270254</Data>
    <Data Name="File">onecoreuap\base\appmodel\execmodel\modern\activationmanager\extensionhelpers.cpp</Data>
    <Data Name="Line">     221</Data>
    <Data Name="failure-&gt;pszFunction"></Data>
    <Data Name="failure-&gt;pszMessage"></Data>
    <Data Name="failure-&gt;pszCode"></Data>
    <Data Name="GetCurrentThreadId()">   11028</Data>
    <Data Name="static_cast&lt;DWORD&gt;(failure-&gt;type)">       1</Data>
    <Data Name="failure-&gt;cFailureCount">32</Data>
  </EventData>
  <RenderingInfo Culture="en-US">
    <Task>WIL_Failover</Task>
  </RenderingInfo>
</Event>

[zadjii-msft]

So for the record, these tests are designed to not run in CI. Unfortunately the tests require Windows 1903, and the CI lab machines are all running Server 2019, which is actually 1809 :(.

This is a known issue with the TabTests currently, but I haven't had a chance to dig in deeper. When I wrote them originally, I was under the impression that it was an OS bug. If you could help us figure out why these don't work, I'd officially owe you a beer when I get back to campus :P