GenerateDepsFile allocates 25GB of heap on large project #48814
Description
When building dotnet runtime on CI, we run OOM on WASM legs.
Top offenders are
GenerateDepsFileis at 25GB over 1056 runs on the same build. 😮ValidateAssembliesTask0.5GB over 59 runsResolveRuntimePackAssets0.4GB over 451 runs
The issue was identified by using new MSBuild telemetry
It would be good to fix the root cause (memory consumption) in those Tasks.
I believe that customers with large solutions could suffer with the same problems.
Workaround
We applied workaround to run those tasks out-of-process for WASM leg.
Metadata
Metadata
Assignees
Labels
Type
Projects
Milestone
Relationships
Participants
Activity
dotnet-policy-service commentedon May 5, 2025
Thanks for creating this issue! We believe this issue is related to NuGet tooling, which is maintained by the NuGet team. Thus, we closed this one and encourage you to raise this issue in the NuGet repository instead. Don’t forget to check out NuGet’s contributing guide before submitting an issue!
If you believe this issue was closed out of error, please comment to let us know.
Happy Coding!
baronfel commentedon May 5, 2025
Bad bot.
Forgind commentedon May 7, 2025
@pavelsavara, are there logs I can look at telling me where in GenerateDepsFile the 25GB were allocated?
pavelsavara commentedon May 7, 2025
@Forgind you can delete workarounds/overrides in
src/mono/browser/build/WasmApp.InTree.CI.targetsof the runtime repo and run the same that the CI pipeline does for WASM on linux.I think that the binlong would not be much different from this random one from yesterday.
There is nothing specific in the binlog, unless you also enable the new telemetry. And that only marks the Task as the guilty party.
We don't know what's the allocation distribution either. Is that one/few specific projects which cause that ? Or that's evenly distributed problem ?
I think the next step is to do GC heap dump on the msbuild before and after running
GenerateDepsFileonce.