InvalidOperationException cannot hijack chunked ... stream (Rancher Desktop, Docker Desktop 4.10.0)

[wesley-pattison]

Describe the bug
Please see this bug reported by someone else in Docker DotNet:
dotnet/Docker.DotNet#554

My own response to this:
dotnet/Docker.DotNet#554 (comment)

To Reproduce
Steps to reproduce the behavior:

1. Use Rancher Desktop with WSL2 enabled and dockerd (moby)
2. Run unit tests in dotnet-testcontainers solution (I am only focused on RabbitMq myself)
3. Exception thrown

Expected behavior
Unit test passes

Desktop (please complete the following information):

• Rancher Desktop (Windows install) v1.1.1
• Windows 10 with WSL2 Ubuntu

Workaround

• Use snapshot release 2.1.0-beta.2698157448
• Downgrade to Docker Desktop 4.9.x

[piotrke]

any update on this? I'd really like to use this library but it's not possible with Rancher Desktop

[HofmeisterAn]

With the latest changes (in the develop) you might be able to use Testcontainers. As long as you don't need access to the I/O streams. I did some minor improvements, but I still need to look into the underlying issue. There is no snapshot release yet, but you can, checkout the repository in the meantime.

[joebone]

So I tried the latest changes in Develop, to no avail, however I have made a local hack that at least allows this to work..
dotnet/Docker.DotNet#554 (comment)
At the same time i had to manually replace the "localhost" in the generated connectionstring with "127.0.0.1" in order to connect properly, so in the end my factory method is something like this:

 static Func<Task<IDbConnection>> GetDockerizedDb = async () =>
 {
     var pw = "tmpTest(!)Password123";
     var dbConfig = new MsSqlTestcontainerConfiguration("mcr.microsoft.com/mssql/server:2019-latest");
     dbConfig.Password = pw;
     var testcontainersBuilder = new TestcontainersBuilder<MsSqlTestcontainer>().WithDatabase(dbConfig);
     var containers = testcontainersBuilder.Build();
     await containers.StartAsync();
     var conn = new SqlConnection(containers.ConnectionString.Replace("localhost,", "127.0.0.1,"));
     await conn.OpenAsync();
     await conn.ExecuteAsync(File.ReadAllText("SharedTests/sundb.setup.sql"));
     return conn;
 };

[joebone]

Update: I have an open PR to that repo that adds an option to the configuration that will allow this project to specify the http version
dotnet/Docker.DotNet#566
Being a first time contributor, I have no idea how long that process will take or even if they will accept the change, but I have applied the changes to a local version and "everything works on my machine", as we say. I don't know what your feelings are on taking dependencies on prerelease versions, but at least if this is a stopper that is a workaround...

[joebone]

@HofmeisterAn Could we have a discussion about how you would best like to integrate the changes required in this library? I had hoped to make as non-intrusive a change as possible for backwards compatibility but I've gone down a rabbit hole of changes in order to include it in the configuration, and the issue is that the instance of the dockerclient that is used is currently pulled in by calling a new DotNet.Testcontainers.Configurations.Windows
which eventually winds up in NpipeEndpointAuthenticationProvider.GetAuthConfig()

Basically the client is instantiated in DockerEndpointAuthenticationConfiguration in the method, and the HttpVersion (configured by the user, or the context detecting that rancher desktop is being run ) would be have to be configured there, but because its internal, the end user has no access / control over that..

public DockerClientConfiguration GetDockerClientConfiguration()
{
  return new DockerClientConfiguration(this.Endpoint) { HttpVersion = this.HttpVersion };
}

Initially I just wanted to add a public Version HttpVersion { get; } to that DockerEndpointAuthenticationConfiguration , but as it is all called internally by the TestcontainersContainer constructor via the TestcontainersClient, the changes I was making were starting to sprawl.

How would be your preferred way of solving this issue? Autodetect the runtime / api that is running? Some kind of extension method like the following?

       dbConfig.Password = pw;
       var testcontainersBuilder = new TestcontainersBuilder<MsSqlTestcontainer>()
           .WithHttpVersion(new(1, 0))
           .WithDatabase(dbConfig);

Just need some guidance here so I don't waste a whole bunch of everybody's time..

[HofmeisterAn]

I know this is an important issue for anyone that can't use Docker. I haven't had enough time to take a close look into the issue yet, but I prepared a test environment and read the Docker.DotNet sources. Just give me some more time, to get a better understanding of the underlying issue.

I don't think, changing the HTTP version to 1.0 is the right approach. If you look into the Docker Engine API documentation, you'll see that ContainerAttach returns 1.1 for instance. I would assume it (should) supports chunked transfer encoding:

it allows streaming of content as chunks and explicitly signaling the end of the content

[joebone]

I'm glad I asked! =) Thanks for the update, I guess meanwhile we'll internally use this workaround, but I look forward to hearing your thoughts on how you'd like to solve this issue when you find the time. Obviously, whatever help I can give is yours. Great project btw!

[jplumhoff]

Since this issue specifically mentions Rancher, I'm not sure if this is the place to mention this or not. But immediately after upgrading to Docker Desktop 4.10, I started seeing the same "cannot hijack chunked or content length stream" exception. I uninstalled 4.10, reinstalled 4.9, and everything works again.

[HofmeisterAn]

I can confirm. It's broken for Docker Desktop for Windows (4.10.0) too. We need a fix in dotnet/Docker.DotNet#554.

[databrecht]

Same on Docker Desktop for Mac (4.10.0) it seems, downgrading to 4.9.1 makes the error go away.

System.AggregateException : One or more errors occurred. (cannot hijack chunked or content length stream) (cannot hijack chunked or content length stream)
---- System.InvalidOperationException : cannot hijack chunked or content length stream
---- System.InvalidOperationException : cannot hijack chunked or content length stream
 Stack Trace:
----- Inner Stack Trace #1 (System.InvalidOperationException) -----
  at [Microsoft.Net](http://microsoft.net/).Http.Client.HttpConnectionResponseContent.HijackStream()
  at Docker.DotNet.DockerClient.MakeRequestForHijackedStreamAsync(IEnumerable`1 errorHandlers, HttpMethod method, String path, IQueryString queryString, IRequestContent body, IDictionary`2 headers, TimeSpan timeout, CancellationToken cancellationToken)
  at Docker.DotNet.ContainerOperations.AttachContainerAsync(String id, Boolean tty, ContainerAttachParameters parameters, CancellationToken cancellationToken)
  at DotNet.Testcontainers.Clients.DockerContainerOperations.AttachAsync(String id, IOutputConsumer outputConsumer, CancellationToken ct)
  at DotNet.Testcontainers.Containers.Modules.TestcontainersContainer.Start(String id, CancellationToken ct)
  at DotNet.Testcontainers.Containers.Modules.TestcontainersContainer.StartAsync(CancellationToken ct)```

[PatrickGhosn]

Getting this bug as well after updating to 4.10.

[HofmeisterAn]

Quick update because I know a couple of devs waiting for a fix. We’re trying to contact the maintainers. If that somehow won’t work we'll fork and fix it.

[Jejuni]

Getting this bug as well.

We're using TestContainers quite heavily for integration testing.
Luckily we didn't update docker on our Build Server, but none of our developers can run integration tests on their machines anymore.

Hoping for a quick fix.

[HofmeisterAn]

@Jejuni Your devs can downgrade to 4.9.1. The patch I'm suggesting works fine on my environment. I just need to get it to the upstream. I'll think about how we could mange this with a fork in the meantime.

[HofmeisterAn]

I've emailed the @dotnet-foundation and asked if they can help us, or if I can get access to the repository. I prefer a fix in the upstream. A fork should be the last option.

[SeanKilleen]

@HofmeisterAn just coming across this. I also reached out to the .NET Foundation Maintainers committee on the Discord & Slack to see if pinging there might help move it along.

In a way, I'm relieved -- was testing Testcontainers for the first time (LOVE the idea!) and thought I'd done something wrong 😅

[HofmeisterAn]

@SeanKilleen Thanks, for reaching out too. I hope we find a way to fix the upstream soon. Especially, because I have a patch and it's a bad first impression using the latest Docker Desktop version.

[SeanKilleen]

@HofmeisterAn another thought -- many of us, including me, now pay for Docker Pro. Since this changed as of 4.10, perhaps we can open a support request / issue and see if docker has any flexibility on their end? I know it doesn't seem to be a "breaking change" in the classical sense, but given the impact, maybe they'd consider options.

[SeanKilleen]

Just to update the group here -- the maintainer responded and Docker.Dotnet has a new release that includes @HofmeisterAn's fix.

So I think as soon as we get a new release of testcontainers-dotnet with the updated Docker.Dotnet dependency, we should be good.

[HofmeisterAn]

I've deployed a snapshot version 2.1.0-beta.2698157448. Can anyone test it? Feedback would be appreciated.

[kron0s19]

I just updated to 2.1.0-beta.2698157448, and I can confirm that it is working now.

Great job (Y)

I'm running Windows Docker Desktop 4.10.1 (82475)

[piotrke]

works like a charm.
I'm running Rancher 1.4.1 on Windows

[SeanKilleen]

Worked for me too! 🎉

[Xitric]

We can also confirm that 2.1.0-beta.2698157448 has fixed our integration tests that use testcontainers.

Thank you for the great work, and great to see the amount of support and activity around this project!

[wesley-pattison]

I can also confirm that using 2.1.0-beta.2698157448 has resolved the issue, and integration tests are succeeding.

Thanks all!

[HofmeisterAn]

Awesome 🥳 I'll publish a release version later that day. Thanks for your response.

[macalbert]

The version 2.1.0-beta.2698157448 fix the problem! 😄

Tried out using docker desktop version 4.9.1 and 4.10.1.

Thank you!!

[rossmasday]

I can confirm that using 2.1.0-beta.2698157448 works on Mac OSX Arm64 and on windows with Rancher desktop 1.4.1