ILT0005 Out of memory when building release mode #9

Arlodotexe · 2022-07-29T01:52:41Z

Describe the bug

Here goes nothing.

In the CI, we've hit ILT0005, and an out of memory error. Only WinUI 2 failed to build, and no app-affecting code was changed for this CI run.

error : ILT0005: 'C:\Users\runneradmin\.nuget\packages\runtime.win10-x64.microsoft.net.native.compiler\2.2.10-rel-29722-00\tools\x64\ilc\Tools\nutc_driver.exe @"D:\a\Labs-Windows\Labs-
error : Error: NUTC300F:Internal Compiler Error: Native compilation failed due to out of memory error

I've seen this before - I've been having this issue in a project's CI for several months, and I've already spent 2 weeks researching and trying to fix this error.

This is the first time I've seen it happen in Labs, but if this is related, then it's more or less random and only present on mid to low-spec machines. It's not reproducible in a VM and it's going to happen more frequently the more code we put into Labs.

There are some mitigations that we can try anyway, but I'm not expecting it to be that easy.

Steps to reproduce

See https://github.com/CommunityToolkit/Labs-Windows/runs/7550122176?check_suite_focus=true

Expected behavior

A buildable repo should consistently pass CI

Screenshots

Code Platform

Windows Build Number

Other Windows Build number

No response

App minimum and target SDK version

Windows 10, version 1809 (Build 17763)
Windows 10, version 1903 (Build 18362)
Windows 10, version 1909 (Build 18363)
Windows 10, version 2004 (Build 19041)
Other (specify)

Other SDK version

No response

Visual Studio Version

No response

Visual Studio Build Number

No response

Device form factor

No response

Additional context

No response

Help us help you

Yes, but only if others can assist.

The text was updated successfully, but these errors were encountered:

michael-hawker · 2022-08-09T16:02:24Z

For CommunityToolkit/Labs-Windows#211 it was just a general exit code, but it may be the same thing. The guidance from the threads in dotnet/corert#5604 was to turn the 64-bit mode on for .NET Native. I tried that in CommunityToolkit/Labs-Windows#211 but then I think it causes issues with all the MSTest warnings we have about something that's misconfigured there.

Did you want to try just bringing over 92ecbc47988d76bc1057e96fda413a80f9abf6ce in a separate branch and seeing if you can add the missing dependencies (if that's the issue) to the test projects?

"D:\a\Labs-Windows\Labs-Windows\tests\CommunityToolkit.Labs.Tests.Uwp\CommunityToolkit.Labs.Tests.Uwp.csproj" (default target) (14:6) ->
  C:\Users\runneradmin\.nuget\packages\microsoft.net.native.compiler\2.2.10-rel-29722-00\tools\Microsoft.NetNative.targets(805,5): error : Internal Compiler Error [D:\a\Labs-Windows\Labs-Windows\tests\CommunityToolkit.Labs.Tests.Uwp\CommunityToolkit.Labs.Tests.Uwp.csproj]
  C:\Users\runneradmin\.nuget\packages\microsoft.net.native.compiler\2.2.10-rel-29722-00\tools\Microsoft.NetNative.targets(805,5): error : Type 'System.Xml.XmlDocumentXPathExtensions' from assembly 'System.Xml.XPath.XmlDocument' was not included in compilation, but was referenced in method 'ClientUtilities.FixRelativePathsInRunSettings(XmlDocument, string)'. There may have been a missing assembly, or a dependency on a more recent Windows SDK release. [D:\a\Labs-Windows\Labs-Windows\tests\CommunityToolkit.Labs.Tests.Uwp\CommunityToolkit.Labs.Tests.Uwp.csproj]
  C:\Users\runneradmin\.nuget\packages\microsoft.net.native.compiler\2.2.10-rel-29722-00\tools\Microsoft.NetNative.targets(805,5): error : Method 'ClientUtilities.FixRelativePathsInRunSettings(XmlDocument, string)' will always throw an exception due to the missing method 'XmlDocumentXPathExtensions.SelectSingleNode(XmlNode, string)' in assembly 'System.Xml.XPath.XmlDocument'. There may have been a missing assembly, or a dependency on a more recent Windows SDK release. [D:\a\Labs-Windows\Labs-Windows\tests\CommunityToolkit.Labs.Tests.Uwp\CommunityToolkit.Labs.Tests.Uwp.csproj]
  C:\Users\runneradmin\.nuget\packages\microsoft.net.native.compiler\2.2.10-rel-29722-00\tools\Microsoft.NetNative.targets(805,5): error : Method 'InferRunSettingsHelper.MakeRunsettingsCompatible(string)' will always throw an exception due to the missing method 'XmlDocumentXPathExtensions.CreateNavigator(XmlDocument)' in assembly 'System.Xml.XPath.XmlDocument'. There may have been a missing assembly, or a dependency on a more recent Windows SDK release. [D:\a\Labs-Windows\Labs-Windows\tests\CommunityToolkit.Labs.Tests.Uwp\CommunityToolkit.Labs.Tests.Uwp.csproj]
  C:\Users\runneradmin\.nuget\packages\microsoft.net.native.compiler\2.2.10-rel-29722-00\tools\Microsoft.NetNative.targets(805,5): error : Method 'InferRunSettingsHelper.UpdateRunSettingsWithUserProvidedSwitches(XmlDocument, Architecture, Framework, string)' will always throw an exception due to the missing method 'XmlDocumentXPathExtensions.CreateNavigator(XmlDocument)' in assembly 'System.Xml.XPath.XmlDocument'. There may have been a missing assembly, or a dependency on a more recent Windows SDK release. [D:\a\Labs-Windows\Labs-Windows\tests\CommunityToolkit.Labs.Tests.Uwp\CommunityToolkit.Labs.Tests.Uwp.csproj]
  C:\Users\runneradmin\.nuget\packages\microsoft.net.native.compiler\2.2.10-rel-29722-00\tools\Microsoft.NetNative.targets(805,5): error : Method 'InferRunSettingsHelper.TryGetLegacySettingElements(string, Dictionary<string, string>)' will always throw an exception due to the missing method 'XmlDocumentXPathExtensions.CreateNavigator(XmlDocument)' in assembly 'System.Xml.XPath.XmlDocument'. There may have been a missing assembly, or a dependency on a more recent Windows SDK release. [D:\a\Labs-Windows\Labs-Windows\tests\CommunityToolkit.Labs.Tests.Uwp\CommunityToolkit.Labs.Tests.Uwp.csproj]

https://github.com/CommunityToolkit/Labs-Windows/runs/7735671450?check_suite_focus=true#step:9:16694

Arlodotexe · 2022-08-10T18:47:04Z

@michael-hawker In case it helps, we discovered and applied a number of fixes in here (different project) that we could try.

michael-hawker · 2022-10-27T20:40:40Z

Also see Sergio's comment here: CommunityToolkit/Labs-Windows#290 (comment)

Arlodotexe · 2022-10-27T21:27:54Z

@michael-hawker This is one of the fixes I intended to try when I got time.

From here, any one of these could help us mitigate the error for now:

    <Use64BitCompiler>true</Use64BitCompiler>
    <PreferredToolArchitecture>x64</PreferredToolArchitecture>

    <SingleThreadNUTC>true</SingleThreadNUTC>
    <UseDotNetNativeToolchain>true</UseDotNetNativeToolchain>
    <EnableGatekeeperAnalysis>true</EnableGatekeeperAnalysis>
    <ShortcutGenericAnalysis>true</ShortcutGenericAnalysis> <!-- Can help stop runaway analysis of generic types and reduce overall generation requirements. -->
    <OutOfProcPDB>true</OutOfProcPDB> <!-- Opt-in to generate PDB's out of process -->

From past experience with this bug, I don't expect any of these to solve the problem permanently, but we can try them.

Arlodotexe · 2022-11-17T15:31:44Z

In CommunityToolkit/Labs-Windows#211, we managed to make ILT0005 happen on every CI run. After investigating, we found:

[View CI run] ILT0005 happens every time when <Use64BitCompiler> is applied to main equivalent.
[View CI run] ILT0005 only happens occasionally when <Use64BitCompiler> is removed.

michael-hawker · 2022-11-17T17:05:37Z

@Arlodotexe <Use64BitCompiler> was added afterwards to try and resolve the error though...

Arlodotexe · 2022-11-18T20:12:36Z

@Arlodotexe <Use64BitCompiler> was added afterwards to try and resolve the error though...

ILT0005 is an out of memory error, it could be that using the 64 bit compiler increased the memory used beyond the 4GB limit on 32-bit processes and caused the error to happen more often.

michael-hawker · 2023-03-14T04:47:47Z

Hitting this a ton on CommunityToolkit/Labs-Windows#405, thought we had some of the flags enabled, but don't see them.

From @Sergio0694:

<Use64BitCompiler>true</Use64BitCompiler> and <OutOfProcPDB>true</OutOfProcPDB>
These can help a bit. Need .NET Native 6.2.14 for the later

@Arlodotexe I think these caused other warnings to turn as errors at one point too which we had to resolve, but I'll try pushing a branch with just them enabled in the UWP props file and see what happens.

Add Use64BitCompiler and OutOfProcPDB flags to release mode build Update to 6.2.14 of Microsoft.NETCore.UniversalWindowsPlatform .NET Native package

Arlodotexe · 2023-05-08T18:42:22Z

A workaround suggested by a member of the UWP Community (see here and here) seems to do what we need in the CI.

There are other variants of ILT0005 that can happen locally (involving PDB write errors), but these are no longer appearing in this project. We'll close this for now, if CommunityToolkit/Labs-Windows#431 passes.

Arlodotexe added bug 🐛 Something isn't working CI/pipeline 🔬 labels Jul 29, 2022

Arlodotexe mentioned this issue Aug 9, 2022

🧪 Bring over StackedNotificationsBehavior from Toolkit CommunityToolkit/Labs-Windows#211

Merged

Arlodotexe self-assigned this Nov 10, 2022

michael-hawker mentioned this issue Dec 7, 2022

🚧 Testing input injector with 1.2.221109.1 WinAppSDK in CI CommunityToolkit/Labs-Windows#183

Draft

michael-hawker mentioned this issue Feb 10, 2023

Optimize Build Jobs CommunityToolkit/Labs-Windows#365

Open

34 tasks

This was referenced Mar 14, 2023

Attempt to mitigate #245 with .NET Native Compiler Memory Issue CommunityToolkit/Labs-Windows#406

Closed

Rename common directory to tooling CommunityToolkit/Labs-Windows#405

Merged

Arlodotexe transferred this issue from CommunityToolkit/Labs-Windows Mar 15, 2023

michael-hawker mentioned this issue Mar 15, 2023

Moved tooling folder to submodule CommunityToolkit/Labs-Windows#409

Merged

michael-hawker mentioned this issue Mar 23, 2023

Fixed SettingsCard not lost force when touch leave CommunityToolkit/Labs-Windows#410

Merged

Arlodotexe mentioned this issue Mar 28, 2023

🧪 [Experiment] TokenView CommunityToolkit/Labs-Windows#348

Merged

This was referenced May 2, 2023

Re-enable Testspaces Test Reporting CommunityToolkit/Windows#40

Merged

Workaround/fix for ILT0005 in CI CommunityToolkit/Labs-Windows#431

Merged

Arlodotexe mentioned this issue May 10, 2023

Mitigations for ILT0005 #50

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ILT0005 Out of memory when building release mode #9

ILT0005 Out of memory when building release mode #9

Arlodotexe commented Jul 29, 2022 •

edited

Loading

michael-hawker commented Aug 9, 2022

Arlodotexe commented Aug 10, 2022 •

edited

Loading

michael-hawker commented Oct 27, 2022

Arlodotexe commented Oct 27, 2022 •

edited

Loading

Arlodotexe commented Nov 17, 2022 •

edited

Loading

michael-hawker commented Nov 17, 2022

Arlodotexe commented Nov 18, 2022 •

edited

Loading

michael-hawker commented Mar 14, 2023

Arlodotexe commented May 8, 2023 •

edited

Loading

ILT0005 Out of memory when building release mode #9

ILT0005 Out of memory when building release mode #9

Comments

Arlodotexe commented Jul 29, 2022 • edited Loading

Describe the bug

Steps to reproduce

Expected behavior

Screenshots

Code Platform

Windows Build Number

Other Windows Build number

App minimum and target SDK version

Other SDK version

Visual Studio Version

Visual Studio Build Number

Device form factor

Additional context

Help us help you

michael-hawker commented Aug 9, 2022

Arlodotexe commented Aug 10, 2022 • edited Loading

michael-hawker commented Oct 27, 2022

Arlodotexe commented Oct 27, 2022 • edited Loading

Arlodotexe commented Nov 17, 2022 • edited Loading

michael-hawker commented Nov 17, 2022

Arlodotexe commented Nov 18, 2022 • edited Loading

michael-hawker commented Mar 14, 2023

Arlodotexe commented May 8, 2023 • edited Loading

Arlodotexe commented Jul 29, 2022 •

edited

Loading

Arlodotexe commented Aug 10, 2022 •

edited

Loading

Arlodotexe commented Oct 27, 2022 •

edited

Loading

Arlodotexe commented Nov 17, 2022 •

edited

Loading

Arlodotexe commented Nov 18, 2022 •

edited

Loading

Arlodotexe commented May 8, 2023 •

edited

Loading