Still missing crashing container logs in 9.3 & 9.4 preview in tests

[afscrome]

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

Continuation of #8206, but that issue has been locked.

If a containers crashes upon startup it still continues to report no logs in both 9.3 and 9.4.0-preview.1.25351.3.

9.4 does have some improvement in that it tells me the container crashed (all be it with a rather confusing error), but the logs are still missing.

    fail: Aspire.Hosting.Dcp.dcpctrl.ContainerReconciler[0]
      failed to start Container	{"Container": {"name":"REDCTED-mkvtbbkr"}, "Reconciliation": 39, "ContainerID": "aeecc55e407fb644a0fd24dda32536c7a52511566f67bd54da7c19d451d967dd", "error": "container 'REDACTED-mkvtbbkr' start failed (exit code 1)\nwriter is closed\nnot all requested objects were returned\nonly 0 out of 1 containers were successfully started"}

This issue is a significant blocker towards us using Distributed Application Testing builder - when (not if) we screw something up and cause the app to crash on startup, we're pretty much unable to debug. Not only does DCP not report the logs, but it also immediately deletes the container from docker when it crashes, so you can't even pull the logs of the stopped container through docker.

The fix for our crash is almost certainly a silly missing environment variable, but lack of logs means we're unable to see exactly what we're missing.

Expected Behavior

No response

Steps To Reproduce

No response

Exceptions (if any)

No response

.NET Version info

No response

Anything else?

No response

[afscrome]

In 9.3, the logs do also show up in the Console log page of the dashboard, just not through DistributedApplicationTestingBuilder

      var builder = DistributedApplicationTestingBuilder.Create();
      builder.Services.AddLogging(logging =>
      {
         logging
            .SetMinimumLevel(LogLevel.Debug)
            .AddFilter(builder.Environment.ApplicationName, LogLevel.Debug)
            .AddFilter("Aspire.", LogLevel.Debug);
      });

[dbreshears]

Could be that the DistributedApplicationTestingBuilder is not subscribing to the startup logs. That needs to be investigated.

[dbreshears]

@afscrome , do you have a repro or way that we can simulate the crash

[afscrome]

Anything that instantly crashes out - missing mandatory argument/env var, unexpected values

        appHost.AddContainer("nginx", "nginx")
            .WithArgs("--not-a-valid-arg");

Running this container via docker cli (docker run nginx --fake-arg):

/docker-entrypoint.sh: 47: exec: --fake-arg: not found

Full test case (based on the aspire sample app)

    [Test]
    public async Task GetWebResourceRootReturnsOkStatusCode()
    {
        // Arrange
        var cancellationToken = TestContext.CurrentContext.CancellationToken;

        var appHost = await DistributedApplicationTestingBuilder.CreateAsync<Projects.SdkContainerPackage_AppHost>(cancellationToken);
        appHost.Services.AddLogging(logging =>
        {
            logging.SetMinimumLevel(LogLevel.Debug);
            // Override the logging filters from the app's configuration
            logging.AddFilter(appHost.Environment.ApplicationName, LogLevel.Debug);
            logging.AddFilter("Aspire.", LogLevel.Debug);
        });
        appHost.Resources.Clear();
        appHost.AddContainer("nginx", "nginx")
            .WithArgs("--fake-arg")
            .WithHttpEndpoint(targetPort: 80)
            .WithHttpHealthCheck();

        await using var app = await appHost.BuildAsync(cancellationToken).WaitAsync(DefaultTimeout, cancellationToken);
        await app.StartAsync(cancellationToken).WaitAsync(DefaultTimeout, cancellationToken);

        await app.ResourceNotifications.WaitForResourceHealthyAsync("nginx", cancellationToken).WaitAsync(DefaultTimeout, cancellationToken);
    }

test output:

    info: Aspire.Hosting.DistributedApplication[0]
      Aspire version: 9.4.0-preview.1.25355.1+8e2d33c6570641dd90de2089f6e12bc56034968d
info: Aspire.Hosting.DistributedApplication[0]
      Distributed application starting.
info: Aspire.Hosting.DistributedApplication[0]
      Application host directory is: S:\SdkContainerPackage\SdkContainerPackage.AppHost
dbug: Aspire.Hosting.Cli.BackchannelService[0]
      Backchannel socket path was not specified.
dbug: Aspire.Hosting.ApplicationModel.ResourceNotificationService[0]
      Resource nginx/nginx-nubrfzut changed state: Starting
info: Aspire.Hosting.DistributedApplication[0]
      Distributed application started. Press Ctrl+C to shut down.
dbug: Aspire.Hosting.ApplicationModel.ResourceNotificationService[0]
      Waiting for resource 'nginx' to enter the 'Healthy' state.
dbug: Aspire.Hosting.ApplicationModel.ResourceNotificationService[0]
      Resource nginx/nginx-nubrfzut changed state: Starting
info: SdkContainerPackage.AppHost.Resources.nginx[0]
      1: 2025-07-07T17:38:42.0980000Z 8ec1413e49556513c73d25e249f281aa0b27229eb9931002830ac08e434c697a
fail: Aspire.Hosting.Dcp.dcpctrl.ContainerReconciler[0]
      failed to start Container	{"Container": "/nginx-nubrfzut", "Reconciliation": 8, "ContainerID": "8ec1413e4955", "ContainerName": "nginx-nubrfzut", "error": "container 'nginx-nubrfzut' start failed (exit code 127)\nwriter is closed\nnot all requested objects were returned\nonly 0 out of 1 containers were successfully started"}
dbug: Aspire.Hosting.ApplicationModel.ResourceNotificationService[0]
      Resource nginx/nginx-nubrfzut changed state: Starting -> FailedToStart
fail: Aspire.Hosting.ApplicationModel.ResourceNotificationService[0]
      Stopped waiting for resource 'nginx' to become healthy because it failed to start.

[afscrome]

Absolute simplest scenario would be something like

        appHost.AddContainer("nginx", "nginx") // or any base image of your choice with bash
            .WithEntrypoint("bash")
            .WithArgs("-c", "echo hello world; exit 123");

[afscrome]

Also looks like it's not container specific - you can get similar behaviour with a short lived executable. (Logs showing up in dashboard, but not in test output)

        appHost.AddExecutable("cmd", "cmd.exe", ".")
            .WithArgs("/c", "echo Hello World!");

[davidfowl]

I'm gonna declare this fixed with #11895.

@afscrome When there's a new build can you give it a try. I think this fixes missing logs randomly.

[afscrome]

It's definitely gotten better, but I'm still loosing logs in some situations. The below test reproduces it very consistently for me - If you uncomment the Task.Delay, this test nearly always passes. If you comment it out the test nearly always fails.

    [Fact]
    public async Task ContainerExitsImmediatelyAfterStart()
    {
        using var cts = DefaultCancellationTokenSource();
        await using var builder = DistributedApplicationTestingBuilder.Create()
            .WithTestingDefaults();

        var container = builder.AddContainer("nginx", "nginx")
            .WithArgs("--illegal-argument");
        AddFakeLogging(container);

        await using var app = builder.Build();
        await app.StartAsync(cts.Token);

        //IMO, not sure this case should really go to `FailedToStart`.  The container did start, it just exited immediately.
        // So I'd expect it to go to `Running`, and then immediately to `Exited`.
        await app.ResourceNotifications.WaitForResourceAsync(container.Resource.Name, KnownResourceStates.FailedToStart, cts.Token);

        //await Task.Delay(1_000);
        var snapshot = app.Services.GetFakeLogCollector().GetSnapshot();
        //first log entry will be the container id
        Assert.Contains("/docker-entrypoint.sh: 47: exec: --illegal-argument: not found", snapshot[3].Message);
    }

Output without delay

    info: Aspire.Hosting.DistributedApplication[0]
      Aspire version: 13.1.0+8a4db1775c3fbae1c602022b636299cb04971fde
info: Aspire.Hosting.DistributedApplication[0]
      Distributed application starting.
info: Aspire.Hosting.DistributedApplication[0]
      Application host directory is: S:\aspire-otel-testing\tests\AppHostPerTest
info: Aspire.Hosting.Dcp.DcpHost[0]
      Starting DCP with arguments: start-apiserver --monitor 31252 --detach --kubeconfig "C:\Users\alex\AppData\Local\Temp\aspire-dcpmif0c3nu.kxc\kubeconfig"
info: Aspire.Hosting.DistributedApplication[0]
      Distributed application started. Press Ctrl+C to shut down.
info: AppHostPerTest.Resources.nginx[0]
      1: 2026-01-06T00:07:42.3570000Z e4da3f660a815bf6771c4a3df1fe2ceaf7249c0a6dd877c9537b15945b227083
info: AppHostPerTest.Resources.nginx[0]
      2: [sys] Added new ContainerNetworkConnection: ContainerName = nginx-cznuqrxg
fail: AppHostPerTest.Resources.nginx[0]
      3: [sys] Failed to start Container: ContainerName = nginx-cznuqrxg, ContainerId = e4da3f660a81:
[sys] container 'nginx-cznuqrxg' start failed (exit code 127)
[sys] writer is closed
[sys] not all requested objects were returned
[sys] only 0 out of 1 containers were successfully started

Output with delay:

 AppHostPerTest.Failures.Docker.ContainerExitsImmediatelyAfterStart
   Source: Docker.cs line 83
   Duration: 6.9 sec

  Standard Output: 
info: Aspire.Hosting.DistributedApplication[0]
      Aspire version: 13.1.0+8a4db1775c3fbae1c602022b636299cb04971fde
info: Aspire.Hosting.DistributedApplication[0]
      Distributed application starting.
info: Aspire.Hosting.DistributedApplication[0]
      Application host directory is: S:\aspire-otel-testing\tests\AppHostPerTest
info: Aspire.Hosting.Dcp.DcpHost[0]
      Starting DCP with arguments: start-apiserver --monitor 34560 --detach --kubeconfig "C:\Users\alex\AppData\Local\Temp\aspire-dcpnerykkcr.kyb\kubeconfig"
info: Aspire.Hosting.DistributedApplication[0]
      Distributed application started. Press Ctrl+C to shut down.
info: AppHostPerTest.Resources.nginx[0]
      1: 2026-01-06T00:06:50.5370000Z 41c835f7105b7daf5fa78822fc2e64cc527593bdb2f39ef493404f54aa54720d
info: AppHostPerTest.Resources.nginx[0]
      2: [sys] Added new ContainerNetworkConnection: ContainerName = nginx-ywuwtdrr
fail: AppHostPerTest.Resources.nginx[0]
      3: [sys] Failed to start Container: ContainerName = nginx-ywuwtdrr, ContainerId = 41c835f7105b:
[sys] container 'nginx-ywuwtdrr' start failed (exit code 127)
[sys] writer is closed
[sys] not all requested objects were returned
[sys] only 0 out of 1 containers were successfully started
fail: AppHostPerTest.Resources.nginx[0]
      4: 2026-01-06T00:06:51.9841661Z /docker-entrypoint.sh: 47: exec: --illegal-argument: not found

This seems to be specifically if logs come in after the contaienr starts. Errors

[afscrome]

    [Fact]
    public async Task ExecutableDoesNotExist()
    {
        using var cts = DefaultCancellationTokenSource();
        await using var builder = DistributedApplicationTestingBuilder.Create();

        var container = builder.AddExecutable("exe", "does-not-exist", "");
        AddFakeLogging(container);

        await using var app = builder.Build();
        await app.StartAsync(cts.Token);
        await app.ResourceNotifications.WaitForResourceAsync(container.Resource.Name, KnownResourceStates.FailedToStart, cts.Token);

        //await Task.Delay(1_000);

        var snapshot = app.Services.GetFakeLogCollector().GetSnapshot();
        Assert.True(snapshot.Any());
        Assert.Contains("[sys] Failed to start a process: Cmd = does-not-exist, Args = [], Error = exec: \"does-not-exist\": executable file not found in %PATH%", snapshot[1].Message);
        Assert.Contains("[sys] Failed to start Executable: Error = exec: \"does-not-exist\": executable file not found in %PATH%", snapshot[2].Message);
    }

is another example that looses logs