HttpClient Content-Type Header

[drain]

Description

When patching KnownHeaders using reflection and adding the content-type header, another gets added to the request.

Reproduction Steps

After executing this code there will be two content-type headers in the request.

var assembly = typeof(System.Net.Http.Headers.HttpRequestHeaders).Assembly;
var assemblyTypes = assembly.GetTypes();
var knownHeaderType = assemblyTypes.FirstOrDefault(n => n.Name == "KnownHeader");
var headerTypeField = knownHeaderType?
    .GetFields(System.Reflection.BindingFlags.NonPublic | System.Reflection.BindingFlags.Instance)
    .FirstOrDefault(n => n.Name.Contains("HeaderType"));
if (headerTypeField is null) return;
var headerTypeFieldType = headerTypeField.FieldType;            
var newValue = Enum.Parse(headerTypeFieldType, "All");
var knownHeadersType = assemblyTypes.FirstOrDefault(n => n.Name == "KnownHeaders");
var contentTypeObj = knownHeadersType.GetFields().FirstOrDefault(n => n.Name == "ContentType").GetValue(null);
if (contentTypeObj is null) return;
headerTypeField.SetValue(contentTypeObj, newValue);    
var contentLengthObj = knownHeadersType.GetFields().FirstOrDefault(n => n.Name == "ContentLength").GetValue(null);
if (contentLengthObj is null) return;
headerTypeField.SetValue(contentLengthObj, newValue);

using (var request = new HttpRequestMessage(new HttpMethod("POST"), "https://www.example.com/"))
{
    request.Version = new Version(2, 0);
    request.Headers.TryAddWithoutValidation("Content-Type", "application/json; charset=UTF-8");
    request.Headers.TryAddWithoutValidation("content-length", "93");
    request.Headers.TryAddWithoutValidation("accept-encoding", "gzip");
    request.Headers.TryAddWithoutValidation("pragma", "no-cache");
    request.Headers.TryAddWithoutValidation("cache-control", "no-cache");

    request.Content = GetBody();

    var response = await _module.client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead);

    return await ResponseHandler(response);

} 

Expected behavior

1 content-type header in the request.

Actual behavior

2 content-type headers in the request.

Regression?

No response

Known Workarounds

No response

Configuration

No response

Other information

Why??

Some websites require headers to be sent in a specific order otherwise the request gets rejected. cough akamai

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

When patching KnownHeaders using reflection and adding the content-type header, another gets added to the request.

Reproduction Steps

After executing this code there will be two content-type headers in the request.

            var assembly = typeof(System.Net.Http.Headers.HttpRequestHeaders).Assembly;
            var assemblyTypes = assembly.GetTypes();
            var knownHeaderType = assemblyTypes.FirstOrDefault(n => n.Name == "KnownHeader");
            var headerTypeField = knownHeaderType?
                .GetFields(System.Reflection.BindingFlags.NonPublic | System.Reflection.BindingFlags.Instance)
                .FirstOrDefault(n => n.Name.Contains("HeaderType"));
            if (headerTypeField is null) return;
            var headerTypeFieldType = headerTypeField.FieldType;            
            var newValue = Enum.Parse(headerTypeFieldType, "All");
            var knownHeadersType = assemblyTypes.FirstOrDefault(n => n.Name == "KnownHeaders");
            var contentTypeObj = knownHeadersType.GetFields().FirstOrDefault(n => n.Name == "ContentType").GetValue(null);
            if (contentTypeObj is null) return;
            headerTypeField.SetValue(contentTypeObj, newValue);    
            var contentLengthObj = knownHeadersType.GetFields().FirstOrDefault(n => n.Name == "ContentLength").GetValue(null);
            if (contentLengthObj is null) return;
            headerTypeField.SetValue(contentLengthObj, newValue);



            using (var request = new HttpRequestMessage(new HttpMethod("POST"), "https://www.example.com/"))
            {
                request.Version = new Version(2, 0);
                request.Headers.TryAddWithoutValidation("Content-Type", "application/json; charset=UTF-8");
                request.Headers.TryAddWithoutValidation("content-length", "93");
                request.Headers.TryAddWithoutValidation("accept-encoding", "gzip");
                request.Headers.TryAddWithoutValidation("pragma", "no-cache");
                request.Headers.TryAddWithoutValidation("cache-control", "no-cache");

                request.Content = GetBody();

                var response = await _module.client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead);

                return await ResponseHandler(response);

            } 

Expected behavior

1 content-type header in the request.

Actual behavior

2 content-type headers in the request.

Regression?

No response

Known Workarounds

No response

Configuration

No response

Other information

Why??

Some websites require headers to be sent in a specific order otherwise the request gets rejected. cough akamai

Author:	drain
Assignees:	-
Labels:	area-System.Net.Http, untriaged
Milestone:	-

[filipnavara]

The Content-Type header is part of request.Content.Headers. You are bypassing the explicit validation that would point that out.

[drain]

thanks, I have managed to fix the duplicated content-type header by doing request.Content.Headers.Remove("Content-Type"); but doesn't seem to work with content-length.

[scalablecory]

This is the first report I've seen about an incompatibility with Akamai -- it is likely a specific endpoint and not all of Akamai, or we'd have many more reports.

Can you provide more details? Headers are orderless per HTTP standard. Typically we don't entertain changes to assist in non-compliant server behavior, unless it's something especially common.

[drain]

You are correct. It is only on specific endpoints protected with https://www.akamai.com/products/bot-manager that are causing an issue. This trick is becoming common within the "anti bot" industry, it would be nice to have some customization. I wasn't sure wherever to post here or on so, I apologize if it isn't the place for this.

[drain]

The tldr is these products often check the request and compare them to what the originated client sends - e.g h2 order, h1 order as mentioned in this issue, cipher sets etc..

[karelz]

Triage: This is advanced scenario to mimic browser. We should be able to achieve it with LLHTTP.
We do not think we should modify HttpClient APIs to do that as it would make the API more complicated.

Closing this issue in favor of LLHTTP.