RequestHeaderEncodingSelector doesn't default to UTF8

[daniel-liuzzi]

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

According to the docs, RequestHeaderEncodingSelector should use the default UTF8Encoding when null is return, but this doesn't seem to be the case.

Either returning null or leaving RequestHeaderEncodingSelector unset results in 400 Bad Request, whereas explicitly returning Encoding.UTF8 works as expected.

Expected Behavior

Setting RequestHeaderEncodingSelector shouldn't be necessary to use default UTF8 encoding.

Steps To Reproduce

Program.cs

var builder = WebApplication.CreateBuilder(args);

builder.WebHost.ConfigureKestrel(
    options =>
    {
        // Replacing null below with System.Text.Encoding.UTF8 works as expected
        options.RequestHeaderEncodingSelector = _ => null;
    });
var app = builder.Build();

app.MapGet("/", (HttpContext context) => context.Request.Headers);
app.Run();

test.http

GET http://localhost:5170/
dummy: Ç

Response

< HTTP/1.1 400 Bad Request
< Content-Length: 0
< Connection: close
< Date: Thu, 27 Mar 2025 15:53:23 GMT
< Server: Kestrel

Exceptions (if any)

No response

.NET Version

9.0.101

Anything else?

As an aside, the following answer in Stack Overflow confirms this behavior:

Eventually we managed to solve this issue via setting the encoding of the Http Headers to UTF-8. I find it quiet interesting, because UTF-8 should be the default value anyways. *We have no idea why it‘s not working without setting it implicitly, but until now we haven’t had any side effects.

[BrennanConroy]

It's very likely you're sending the Ç character in Unicode form which is value 199 instead of as UTF8 which would be 195 135, see https://en.wikipedia.org/wiki/%C3%87#Computer

The reason it "works" when you explicitly set Encoding.UTF8 is because you're allowing the decoder to replace invalid bytes (see second bullet of https://learn.microsoft.com/dotnet/api/system.text.encoding.utf8?view=net-9.0#remarks) and the resulting header value will not be what you expect, it'll be 65533.
Internally Kestrel uses new UTF8Encoding(encoderShouldEmitUTF8Identifier: false, throwOnInvalidBytes: true); instead of Encoding.UTF8 because we only want to allow UTF8.

What you likely want is to set RequestHeaderEncodingSelector = _ = Encoding.Latin1;
See this code sample for example:

using System;
using System.Text;

var encoding = new UTF8Encoding(encoderShouldEmitUTF8Identifier: false, throwOnInvalidBytes: true);
var noThrowEncoding = new UTF8Encoding(encoderShouldEmitUTF8Identifier: false, throwOnInvalidBytes: false);
var s = encoding.GetString([195, 135]);
var s2 = noThrowEncoding.GetString([195, 135]);
Console.WriteLine(s);
Console.WriteLine(s2);

s2 = noThrowEncoding.GetString([199]);
Console.WriteLine(s2);

Console.WriteLine(Encoding.Latin1.GetString([199]));

s = encoding.GetString([199]); // <--- This throws
Console.WriteLine(s);

This stackoverflow also talks about Utf8 and Latin1 encoding and might be insightful.
https://stackoverflow.com/questions/56761626/how-to-detect-and-fix-incorrect-character-encoding

[dotnet-policy-service]

This issue has been resolved and has not had any activity for 1 day. It will be closed for housekeeping purposes.

See our Issue Management Policies for more information.