Configure Container Startup Timeout

[luisquintanilla]

Containers like HuggingFace text-generation-inference take a long time to startup because they're pulling down the model which is usually multi-GB files.

This causes the app to time out.

info: Aspire.Hosting.DistributedApplication[0]
      Distributed application starting.
info: Aspire.Hosting.DistributedApplication[0]
      Application host directory is: C:\Users\luquinta\source\repos\TestAspireDeployment\TestAspireDeployment.AppHost
info: Aspire.Hosting.DistributedApplication[0]
      Now listening on: http://localhost:15201
info: Aspire.Hosting.DistributedApplication[0]
      Distributed application started. Press CTRL-C to stop.
info: Grpc.AspNetCore.Server.ServerCallHandler[14]
      Error reading message.
      System.IO.IOException: The client reset the request stream.
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpRequestPipeReader.ReadAsync(CancellationToken cancellationToken)
         at Grpc.AspNetCore.Server.Internal.PipeExtensions.ReadSingleMessageAsync[T](PipeReader input, HttpContextServerCallContext serverCallContext, Func`2 deserializer)
fail: Grpc.AspNetCore.Server.ServerCallHandler[6]
      Error when executing service method 'WatchResourceConsoleLogs'.
      System.IO.IOException: The client reset the request stream.
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpRequestPipeReader.ReadAsync(CancellationToken cancellationToken)
         at Grpc.AspNetCore.Server.Internal.PipeExtensions.ReadSingleMessageAsync[T](PipeReader input, HttpContextServerCallContext serverCallContext, Func`2 deserializer)
         at Grpc.AspNetCore.Server.Internal.CallHandlers.ServerStreamingServerCallHandler`3.HandleCallAsyncCore(HttpContext httpContext, HttpContextServerCallContext serverCallContext)
         at Grpc.AspNetCore.Server.Internal.CallHandlers.ServerCallHandlerBase`3.<HandleCallAsync>g__AwaitHandleCall|8_0(HttpContextServerCallContext serverCallContext, Method`2 method, Task handleCall)

Provide a way to enable containers that take a long time to start up.

To repro in Aspire, add this to AppHost project.

var hf =
    builder.AddContainer("hf-generation", "ghcr.io/huggingface/text-generation-inference",tag: "1.4")
    .WithVolumeMount("$PWD/data", "/data", VolumeMountType.Bind)
    .WithHttpEndpoint(hostPort: 8080, name: "generation")
    .WithArgs("--model-id", "HuggingFaceH4/zephyr-7b-beta");

[DamianEdwards]

I tried to repro this on latest bits and it seemed to indicate there's an error in your WithArgs call?

[luisquintanilla]

Yeah. Good catch. I don't think that's needed in this case so you can still run it without that arg. Updated the repro command

[DamianEdwards]

For reference, --shm-size is an arg for docker run itself which we don't support right now (similar to #2737).

I'm trying the repro again now...

[DamianEdwards]

Also note, your .WithVolumeMount("$PWD/data", "/data", VolumeMountType.Bind) needs to be changed to .WithBindMount("./data", "/data") with latest bits and to work the way you intended I think (the $PWD just gets taken literally, not interpreted as the current directory).

[luisquintanilla]

For reference, --shm-size is an arg for docker run itself which we don't support right now (similar to #2737).

I'm trying the repro again now...

I see. So to clarify docker run args aren't supported yet but args passed to to the container are. In this case --model-id being something you pass to the container.

[DamianEdwards]

Ok I got this working locally on latest (daily) bits.

I had to change your code to the following to ensure the endpoint was registered correctly:

var hf =
    builder.AddContainer("hf-generation", "ghcr.io/huggingface/text-generation-inference", tag: "1.4")
        .WithBindMount("./data", "/data")
        .WithHttpEndpoint(containerPort: 80, hostPort: 8080)
        .WithArgs("--model-id", "HuggingFaceH4/zephyr-7b-beta");

I watched the logs in Docker Desktop and on the Aspire Dashboard and it took about 5 minutes to fully startup (having previously downloaded the model into the ./data directory):

At that point I opened a terminal and ran the following command:

 curl localhost:8080/generate -X POST -d '{"inputs":"What is Deep Learning?","parameters":{"max_new_tokens":20}}' -H 'Content-Type: application/json'

And got the following reply:

{"generated_text":"\n\nDeep Learning is a subset of Machine Learning that uses Artificial Neural Networks with multiple"}

I could see the generation details in the log too:

If there was a problem before it seems it's fixed now, perhaps by the recent change to streaming logs.

[luisquintanilla]

Awesome! Is this the right guidance for setting up the latest daily version of Aspire?

https://github.com/dotnet/aspire/blob/main/docs/using-latest-daily.md

[DamianEdwards]

Yep! Let us know if you run into any issues. Getting the feeds and workload just right can be a little tricky.