WebSocket compression issues when testing server-side with Autobahn

[BrennanConroy]

When trying to add WebSockets compression support to ASP.NET Core I ran the Autobahn test suite against the server and hit multiple test cases failing.

I'm going to focus on a single one in this issue because it's already complicated enough (and hopefully same root issue).

This specific issue is looking at test 12.4.11 which states: "Send 1000 compressed messages each of payload size 8192, auto-fragment to 256 octets. Use default permessage-deflate offer." I modified the client window bits to be 12 (was testing various window bits and hitting issues on different tests with different window bits)

I manually looped over all the messages until I found one that wasn't 8192 in size and have isolated that individual message for the repro below.

Single message test bytes from client
compressed8192fromClient12bit.txt

[Fact]
public async Task CompressTest()
{
    WebSocketTestStream stream = new();
    using WebSocket server = WebSocket.CreateFromStream(stream, new WebSocketCreationOptions
    {
        IsServer = true,
        KeepAliveInterval = TimeSpan.Zero,
        DangerousDeflateOptions = new()
        {
            ClientMaxWindowBits = 12,
            ServerMaxWindowBits = 15,
            ClientContextTakeover = false,
        }
    });
    using WebSocket client = WebSocket.CreateFromStream(stream.Remote, new WebSocketCreationOptions
    {
        KeepAliveInterval = TimeSpan.Zero,
        DangerousDeflateOptions = new()
        {
            ClientMaxWindowBits = 12,
            ServerMaxWindowBits = 15,
            ClientContextTakeover = false,
        }
    });

    using var fileStream = File.OpenRead("<path to>/compressed8192fromClient12bit.txt");
    int r = 1;
    var fileBuf = new byte[2000];
    while (r != 0)
    {
        r = await fileStream.ReadAsync(fileBuf);
        await stream.Remote.WriteAsync(fileBuf.AsMemory(0, r), default);
    }

    var buf = new byte[8192];
    int total = 0;
    while (true)
    {
        var res = await server.ReceiveAsync(buf, default);
        total += res.Count;
        if (res.EndOfMessage == true)
        {
            Assert.Equal(8192, total);
            break;
        }
    }
}

Some more information about this specific message:
Inflated sections (bytes): 320, 3226, 822, 1736, 2115 - total = 8219
WebSocket frame lengths: 256, 256, 256, 256, 243

cc @zlatanov in case you're interested

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

When trying to add WebSockets compression support to ASP.NET Core I ran the Autobahn test suite against the server and hit multiple test cases failing.

I'm going to focus on a single one in this issue because it's already complicated enough (and hopefully same root issue).

This specific issue is looking at test 12.4.11 which states: "Send 1000 compressed messages each of payload size 8192, auto-fragment to 256 octets. Use default permessage-deflate offer." I modified the client window bits to be 12 (was testing various window bits and hitting issues on different tests with different window bits)

I manually looped over all the messages until I found one that wasn't 8192 in size and have isolated that individual message for the repro below.

Single message test bytes from client
compressed8192fromClient12bit.txt

[Fact]
public async Task CompressTest()
{
    WebSocketTestStream stream = new();
    using WebSocket server = WebSocket.CreateFromStream(stream, new WebSocketCreationOptions
    {
        IsServer = true,
        KeepAliveInterval = TimeSpan.Zero,
        DangerousDeflateOptions = new()
        {
            ClientMaxWindowBits = 12,
            ServerMaxWindowBits = 15,
            ClientContextTakeover = false,
        }
    });
    using WebSocket client = WebSocket.CreateFromStream(stream.Remote, new WebSocketCreationOptions
    {
        KeepAliveInterval = TimeSpan.Zero,
        DangerousDeflateOptions = new()
        {
            ClientMaxWindowBits = 12,
            ServerMaxWindowBits = 15,
            ClientContextTakeover = false,
        }
    });

    using var fileStream = File.OpenRead("<path to>/compressed8192fromClient12bit.txt");
    int r = 1;
    var fileBuf = new byte[2000];
    while (r != 0)
    {
        r = await fileStream.ReadAsync(fileBuf);
        await stream.Remote.WriteAsync(fileBuf.AsMemory(0, r), default);
    }

    var buf = new byte[8192];
    int total = 0;
    while (true)
    {
        var res = await server.ReceiveAsync(buf, default);
        total += res.Count;
        if (res.EndOfMessage == true)
        {
            Assert.Equal(8192, total);
            break;
        }
    }
}

Some more information about this specific message:
Inflated sections (bytes): 320, 3226, 822, 1736, 2115 - total = 8219
WebSocket frame lengths: 256, 256, 256, 256, 243

cc @zlatanov in case you're interested

Author:	BrennanConroy
Assignees:	-
Labels:	area-System.Net, untriaged
Milestone:	-

[scalablecory]

CC @CarnaViire

[zlatanov]

Thanks for the ping @BrennanConroy!

What I find interesting is that the data contained after 8192nd byte contains valid UTF string - class="case_text_block cas. Can you share how did you manage to create the deflated file?

Or if you have a branch where I can run the Autobahn server tests, I can try and reproduce it locally.

[BrennanConroy]

What I find interesting is that the data contained after 8192nd byte contains valid UTF string - class="case_text_block cas. Can you share how did you manage to create the deflated file?

I ran the test, captured the bytes before the server did anything with them, then found a message that wasn't 8192 in length after decompression and isolated that message.

One interesting thing is that the test failed with the message "(got length 8175, expected length 8192)", but we're seeing a size of 8219 on the server side.

I do have a branch and a server app at https://github.com/dotnet/aspnetcore/tree/brecon/wscompression/src/Middleware/WebSockets/test/ConformanceTests/AutobahnTestApp
Running the server with: dotnet run --urls=http://localhost:9001

To set the client window bits, modify this line https://github.com/dotnet/aspnetcore/blob/71c1b11a136ca4c415764caa2493381fc1bf7ba3/src/Middleware/WebSockets/src/WebSocketMiddleware.cs#L169

fuzzingclient.json

{
	"servers": [
	{
	  "agent": "Server",
	  "url": "ws://host.docker.internal:9001"
	}
	],
	"outdir": "./reports/clients",
	"cases": ["12.4.11"],
	"exclude-cases": [
	  "12.2.*"
	],
	"exclude-agent-cases": {}
}

To run the test use: docker run -it --rm -v "${PWD}/config:/config" -v "${PWD}/reports:/reports" -p 9001:9001 --name fuzzingclient crossbario/autobahn-testsuite wstest --mode fuzzingclient -s config/fuzzingclient.json

[zlatanov]

@BrennanConroy I added a barebone websocket server here: https://github.com/zlatanov/websockets-compliance. After running the test case 12.4.11 it passed without problems. Actually all tests pass without any issues.

I ran the tests using the latest build from the runtime and corerun.exe "C:\Projects\Github\websockets-compliance\src\bin\Debug\net5.0\WebSocketCompliance.dll".

I didn't find any issues in your code at first glance. Can you try and run my project and see if the test succeeds / fails?

kestrel_case_12_4_1.zip

[BrennanConroy]

Looks like you tried 12.4.1? Not 12.4.11?
Also, how do you run your app?

[zlatanov]

I've attached the results for all 12.* cases, all green.

kestrel_case_12.zip

Also, how do you run your app?

Assuming that you have https://github.com/dotnet/runtime in x:\runtime, I build it with build.cmd clr+libs -rc Release. Then copy the output of Microsoft.AspNetCore.App from C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\5.0.5 to x:\runtime\artifacts\bin\testhost\net6.0-windows-Debug-x64\shared\Microsoft.NETCore.App\6.0.0.

Build my project in either debug or release. Then start it with:

cd "x:\runtime\artifacts\bin\testhost\net6.0-windows-Debug-x64\shared\Microsoft.NETCore.App\6.0.0"
corerun.exe "x:\websockets-compliance\src\bin\$(Configuration)\net5.0\WebSocketCompliance.dll" --urls http://localhost:9001

[BrennanConroy]

Ran your server and saw the tests all passing which was super confusing. The only difference I saw between your test app and mine was the buffer passed into ReceiveAsync and SendAsync is 16*1024 in your app and 4*1024 in mine. Changing mine got the tests passing. And changing yours to 4*1024 showed the failures like I was seeing earlier.

You can also still see errors with the 16*1024 buffer if you set the client max window bits to 12 from the server side. 12.4.16 fails in that case, but if you then increase the buffer to something larger than the tests message size (131072) the test passes again.

[zlatanov]

Thanks for the findings. I will investigate what is happening.