dotnet watch grows my app's memory until it freezes my system

[MeikelLP]

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

Similar to #59027

I have the same issue though my symptoms are quite different:

• I ever only get 1 hot reload and have to restart after that. I never got a second hot reload.
• The app keeps taking memory (idle is ~250mb, after hot reload growing and growing 20+GB until my system freezes and the kernel kills the process)

• The memory growing starts exactly after the second try of hot reload

I assume this may be a bug in my application but I get zero errors from dotnet watch or in the Aspire dashboard.

dotnet watch --verbose

dotnet watch ⌚ File updated: /src/Service.BrokerPortal/Components/Pages/Page.razor.cs
dotnet watch ⌚ Updating document text of '/src/Service.BrokerPortal/Components/Pages/Page.razor.cs'.
dotnet watch ⌚ Solution after document update: v3
dotnet watch 🔥 Hot reload capabilities: AddExplicitInterfaceImplementation AddFieldRva AddInstanceFieldToExistingType AddMethodToExistingType AddStaticFieldToExistingType Baseline ChangeCustomAttributes GenericAddFieldToExistingType GenericAddMethodToExistingType GenericUpdateMethod NewTypeDefinition UpdateParameters.
dotnet watch 🔃 [Service.BrokerPortal (net10.0)] Updating diagnostics.
dotnet watch 🔥 [Service.BrokerPortal.AppHost (net10.0)] Sending update batch #1
dotnet watch 🕵 [Service.BrokerPortal.AppHost (net10.0)] Applying updates to module 04552c6a-2367-41c3-a4a2-d48f2e9dbf72.
dotnet watch 🕵 [Service.BrokerPortal.AppHost (net10.0)] Invoking metadata update handlers.
dotnet watch 🕵 [Service.BrokerPortal.AppHost (net10.0)] System.Reflection.Metadata.RuntimeTypeMetadataUpdateHandler.ClearCache
dotnet watch 🕵 [Service.BrokerPortal.AppHost (net10.0)] System.Text.Json.JsonSerializerOptionsUpdateHandler.ClearCache
dotnet watch 🕵 [Service.BrokerPortal.AppHost (net10.0)] Microsoft.Extensions.DependencyInjection.ActivatorUtilities+ActivatorUtilitiesUpdateHandler.ClearCache
dotnet watch 🕵 [Service.BrokerPortal.AppHost (net10.0)] System.ComponentModel.ReflectionCachesUpdateHandler.ClearCache
dotnet watch 🕵 [Service.BrokerPortal.AppHost (net10.0)] Microsoft.AspNetCore.Routing.RouteValueDictionary+MetadataUpdateHandler.ClearCache
dotnet watch 🕵 [Service.BrokerPortal.AppHost (net10.0)] Microsoft.Extensions.Internal.PropertyHelper+MetadataUpdateHandler.ClearCache
dotnet watch 🕵 [Service.BrokerPortal.AppHost (net10.0)] Microsoft.Extensions.Internal.PropertyHelper+MetadataUpdateHandler.ClearCache
dotnet watch 🕵 [Service.BrokerPortal.AppHost (net10.0)] Microsoft.Extensions.Internal.PropertyHelper+MetadataUpdateHandler.ClearCache
dotnet watch 🕵 [Service.BrokerPortal.AppHost (net10.0)] Updates applied.
dotnet watch 🔥 [Service.BrokerPortal.AppHost (net10.0)] Update batch #1 completed.

From what I can see the second hot reload try isn't even logged but at that point the app is stuck at ~20% CPU and 2GB (from ~250MB when idle)

What other options do I have to diagnose this?

Expected Behavior

No response

Steps To Reproduce

Unknown

Exceptions (if any)

No response

.NET Version

10.0.300

Anything else?

.NET SDK:
 Version:           10.0.300
 Commit:            caa81fa497
 Workload version:  10.0.300-manifests.502164c0
 MSBuild version:   18.6.3+caa81fa49

Runtime Environment:
 OS Name:     debian
 OS Version:  13
 OS Platform: Linux
 RID:         linux-x64
 Base Path:   /usr/share/dotnet/sdk/10.0.300/

.NET workloads installed:
There are no installed workloads to display.
Configured to use workload sets when installing new manifests.
No workload sets are installed. Run "dotnet workload restore" to install a workload set.

Host:
  Version:      10.0.8
  Architecture: x64
  Commit:       94ea82652c

.NET SDKs installed:
  8.0.420 [/usr/share/dotnet/sdk]
  9.0.313 [/usr/share/dotnet/sdk]
  10.0.300 [/usr/share/dotnet/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 8.0.26 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 9.0.15 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 10.0.8 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 8.0.26 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 9.0.15 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 10.0.8 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

Other architectures found:
  None

Environment variables:
  Not set

global.json file:
  Not found

Learn more:
  https://aka.ms/dotnet/info

Download .NET:
  https://aka.ms/dotnet/download

[dariatiurina]

@tmat Can you take a look at this, please?

[tmat]

@MeikelLP It's unlikely that this is dotnet watch issue. To diagnose you can either capture a memory dump at the point when the app consumes a lot of memory and then inspect the memory to see what's allocated on the heap. Or you can record perf trace and inspect the trace.

Some tools that might help:
https://learn.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-dump
https://learn.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-gcdump
https://learn.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-trace

[MeikelLP]

@tmat thanks for confirming that it's probably not a dotnet watch issue. I analyzed the memory with dotMemory and got following suspicious stacktrace:

Microsoft.AspNetCore.Components.RenderTree.RenderTreeDiff[]
  Objects : n/a
  Bytes   : 6442450968

 100%  Rent • 6.00 GB / 6.00 GB • System.Buffers.SharedArrayPool<T>.Rent(Int32)
   100%  RenderIntoBatch • 6.00 GB / - • Microsoft.AspNetCore.Components.Rendering.ComponentState.RenderIntoBatch(RenderBatchBuilder, RenderFragment, Exception)
     100%  ProcessRenderQueue • 6.00 GB / - • Microsoft.AspNetCore.Components.RenderTree.Renderer.ProcessRenderQueue()
       100%  StateHasChanged • 6.00 GB / - • Microsoft.AspNetCore.Components.ComponentBase.StateHasChanged()
         100%  CallOnParametersSetAsync • 6.00 GB / - • Microsoft.AspNetCore.Components.ComponentBase.CallOnParametersSetAsync()
           100%  SetParametersAsync • 6.00 GB / - • Microsoft.AspNetCore.Components.ComponentBase.SetParametersAsync(ParameterView)
             100%  SupplyCombinedParameters • 6.00 GB / - • Microsoft.AspNetCore.Components.Rendering.ComponentState.SupplyCombinedParameters(ParameterView)
               100%  SetDirectParameters • 6.00 GB / - • Microsoft.AspNetCore.Components.Rendering.ComponentState.SetDirectParameters(ParameterView)
                 100%  <RenderRootComponentsOnHotReload>b__64_0 • 6.00 GB / - • Microsoft.AspNetCore.Components.RenderTree.Renderer.<RenderRootComponentsOnHotReload>b__64_0()
                   100%  <InvokeAsync>g__Execute|8_0 • 6.00 GB / - • Microsoft.AspNetCore.Components.Rendering.RendererSynchronizationContext.<InvokeAsync>g__Execute|8_0(ValueTuple<T1, T2, T3>)
                     100%  InvokeAsync • 6.00 GB / - • Microsoft.AspNetCore.Components.Rendering.RendererSynchronizationContext.InvokeAsync(Action)
                       100%  MoveNext • 6.00 GB / - • Microsoft.AspNetCore.Components.RenderTree.Renderer+<RenderRootComponentsOnHotReload>d__64.MoveNext()
                         100%  Start • 6.00 GB / - • System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start<TStateMachine>(TStateMachine)
                           100%  Start • 6.00 GB / - • System.Runtime.CompilerServices.AsyncVoidMethodBuilder.Start<TStateMachine>(TStateMachine)
                             100%  RenderRootComponentsOnHotReload • 6.00 GB / - • Microsoft.AspNetCore.Components.RenderTree.Renderer.RenderRootComponentsOnHotReload()
                               100%  RunInternal • 6.00 GB / - • System.Threading.ExecutionContext.RunInternal(ExecutionContext, ContextCallback, Object)
                                 100%  <GetUpdateHandlerActions>b__6 • 6.00 GB / - • Microsoft.DotNet.HotReload.MetadataUpdateHandlerInvoker+<>c__DisplayClass19_0.<GetUpdateHandlerActions>b__6(Type[])
                                   100%  MetadataUpdated • 6.00 GB / - • Microsoft.DotNet.HotReload.MetadataUpdateHandlerInvoker+RegisteredActions.MetadataUpdated(AgentReporter, Type[])
                                     100%  MetadataUpdated • 6.00 GB / - • Microsoft.DotNet.HotReload.MetadataUpdateHandlerInvoker.MetadataUpdated(Type[])
                                       100%  ApplyManagedCodeUpdates • 6.00 GB / - • Microsoft.DotNet.HotReload.HotReloadAgent.ApplyManagedCodeUpdates(IEnumerable<T>)
                                         100%  MoveNext • 6.00 GB / - • Microsoft.DotNet.HotReload.Listener+<ReadAndApplyManagedCodeUpdateAsync>d__8.MoveNext()
                                           100%  ExecutionContextCallback • 6.00 GB / - • System.Runtime.CompilerServices.AsyncTaskMethodBuilder+AsyncStateMachineBox<TResult, TStateMachine>.ExecutionContextCallback(Object)
                                             100%  RunInternal • 6.00 GB / - • System.Threading.ExecutionContext.RunInternal(ExecutionContext, ContextCallback, Object)
                                               100%  MoveNext • 6.00 GB / - • System.Runtime.CompilerServices.AsyncTaskMethodBuilder+AsyncStateMachineBox<TResult, TStateMachine>.MoveNext(Thread)
                                                 100%  MoveNext • 6.00 GB / - • System.Runtime.CompilerServices.AsyncTaskMethodBuilder+AsyncStateMachineBox<TResult, TStateMachine>.MoveNext()
                                                   100%  RunOrScheduleAction • 6.00 GB / - • System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox, Boolean)
                                                     100%  RunContinuations • 6.00 GB / - • System.Threading.Tasks.Task.RunContinuations(Object)
                                                       100%  TrySetResult • 6.00 GB / - • System.Threading.Tasks.Task<TResult>.TrySetResult(TResult)
                                                         100%  SetExistingTaskResult • 6.00 GB / - • System.Runtime.CompilerServices.AsyncTaskMethodBuilder<TResult>.SetExistingTaskResult(Task<TResult>, TResult)
                                                           100%  SetResult • 6.00 GB / - • System.Runtime.CompilerServices.AsyncValueTaskMethodBuilder<TResult>.SetResult(TResult)
                                                             100%  MoveNext • 6.00 GB / - • Microsoft.DotNet.HotReload.ManagedCodeUpdateRequest+<ReadAsync>d__11.MoveNext()
                                                               100%  ExecutionContextCallback • 6.00 GB / - • System.Runtime.CompilerServices.AsyncTaskMethodBuilder+AsyncStateMachineBox<TResult, TStateMachine>.ExecutionContextCallback(Object)
                                                                 100%  RunInternal • 6.00 GB / - • System.Threading.ExecutionContext.RunInternal(ExecutionContext, ContextCallback, Object)
                                                                   100%  MoveNext • 6.00 GB / - • System.Runtime.CompilerServices.AsyncTaskMethodBuilder+AsyncStateMachineBox<TResult, TStateMachine>.MoveNext(Thread)
                                                                     100%  MoveNext • 6.00 GB / - • System.Runtime.CompilerServices.AsyncTaskMethodBuilder+AsyncStateMachineBox<TResult, TStateMachine>.MoveNext()
                                                                       100%  RunOrScheduleAction • 6.00 GB / - • System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox, Boolean)
                                                                         100%  RunContinuations • 6.00 GB / - • System.Threading.Tasks.Task.RunContinuations(Object)
                                                                           100%  TrySetResult • 6.00 GB / - • System.Threading.Tasks.Task<TResult>.TrySetResult(TResult)
                                                                             100%  SetExistingTaskResult • 6.00 GB / - • System.Runtime.CompilerServices.AsyncTaskMethodBuilder<TResult>.SetExistingTaskResult(Task<TResult>, TResult)
                                                                               100%  SetResult • 6.00 GB / - • System.Runtime.CompilerServices.AsyncValueTaskMethodBuilder<TResult>.SetResult(TResult)
                                                                                 100%  MoveNext • 6.00 GB / - • Microsoft.DotNet.HotReload.StreamExtesions+<ReadByteAsync>d__9.MoveNext()
                                                                                   100%  ExecutionContextCallback • 6.00 GB / - • System.Runtime.CompilerServices.AsyncTaskMethodBuilder+AsyncStateMachineBox<TResult, TStateMachine>.ExecutionContextCallback(Object)
                                                                                     100%  RunInternal • 6.00 GB / - • System.Threading.ExecutionContext.RunInternal(ExecutionContext, ContextCallback, Object)
                                                                                       100%  MoveNext • 6.00 GB / - • System.Runtime.CompilerServices.AsyncTaskMethodBuilder+AsyncStateMachineBox<TResult, TStateMachine>.MoveNext(Thread)
                                                                                         100%  MoveNext • 6.00 GB / - • System.Runtime.CompilerServices.AsyncTaskMethodBuilder+AsyncStateMachineBox<TResult, TStateMachine>.MoveNext()
                                                                                           100%  RunOrScheduleAction • 6.00 GB / - • System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox, Boolean)
                                                                                             100%  RunContinuations • 6.00 GB / - • System.Threading.Tasks.Task.RunContinuations(Object)
                                                                                               100%  SetResult • 6.00 GB / - • System.Runtime.CompilerServices.AsyncValueTaskMethodBuilder<TResult>.SetResult(TResult)
                                                                                                 100%  MoveNext • 6.00 GB / - • Microsoft.DotNet.HotReload.StreamExtesions+<ReadExactlyAsync>d__16.MoveNext()
                                                                                                   100%  ExecutionContextCallback • 6.00 GB / - • System.Runtime.CompilerServices.AsyncTaskMethodBuilder+AsyncStateMachineBox<TResult, TStateMachine>.ExecutionContextCallback(Object)
                                                                                                     100%  RunInternal • 6.00 GB / - • System.Threading.ExecutionContext.RunInternal(ExecutionContext, ContextCallback, Object)
                                                                                                       100%  MoveNext • 6.00 GB / - • System.Runtime.CompilerServices.AsyncTaskMethodBuilder+AsyncStateMachineBox<TResult, TStateMachine>.MoveNext(Thread)
                                                                                                         100%  MoveNext • 6.00 GB / - • System.Runtime.CompilerServices.AsyncTaskMethodBuilder+AsyncStateMachineBox<TResult, TStateMachine>.MoveNext()
                                                                                                           100%  RunOrScheduleAction • 6.00 GB / - • System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox, Boolean)
                                                                                                             100%  RunContinuations • 6.00 GB / - • System.Threading.Tasks.Task.RunContinuations(Object)
                                                                                                               100%  SetResult • 6.00 GB / - • System.Runtime.CompilerServices.AsyncValueTaskMethodBuilder<TResult>.SetResult(TResult)
                                                                                                                 100%  MoveNext • 6.00 GB / - • System.IO.Pipes.PipeStream+<ReadAsyncCore>d__82.MoveNext()
                                                                                                                   100%  ExecutionContextCallback • 6.00 GB / - • System.Runtime.CompilerServices.AsyncTaskMethodBuilder+AsyncStateMachineBox<TResult, TStateMachine>.ExecutionContextCallback(Object)
                                                                                                                     100%  RunInternal • 6.00 GB / - • System.Threading.ExecutionContext.RunInternal(ExecutionContext, ContextCallback, Object)
                                                                                                                       100%  MoveNext • 6.00 GB / - • System.Runtime.CompilerServices.AsyncTaskMethodBuilder+AsyncStateMachineBox<TResult, TStateMachine>.MoveNext(Thread)
                                                                                                                         100%  MoveNext • 6.00 GB / - • System.Runtime.CompilerServices.AsyncTaskMethodBuilder+AsyncStateMachineBox<TResult, TStateMachine>.MoveNext()
                                                                                                                           100%  Execute • 6.00 GB / - • System.Net.Sockets.SocketAsyncEngine.Execute()
                                                                                                                             100%  Dispatch • 6.00 GB / - • System.Threading.ThreadPoolWorkQueue.Dispatch()
                                                                                                                               100%  WorkerThreadStart • 6.00 GB / - • System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart()
                                                                                                                                 100%  StartCallback • 6.00 GB / - • System.Threading.Thread.StartCallback()
                                                                                                                                  ►  100%  [AllThreadsRoot] • 6.00 GB / - • [AllThreadsRoot]

#stacktrace

The only place where I myself am using a SharedArrayPool is in my Login.razor.cs in the OnLogin callback:

var arr = ArrayPool<byte>.Shared.Rent(1024);
var buffer = arr.AsSpan(0, 1024);
Random.Shared.NextBytes(buffer);
try
{
    // do login
}
finally
{
    ArrayPool<byte>.Shared.Return(arr);
}

So I think my code is not the issue? The code shouldn't even be called.

Any way I can provide more details for you to analyze?

[tmat]

@javiercn