HttpClient instrumentation does not handle case when tracing headers are content headers

[nrcventura]

The HttpClient instrumentation currently handles the case when the distributed tracing and CAT headers are defined in the request headers but it does not handle the case when the tracing headers are already defined as content headers.

Description

The HttpClient instrumentation currently handles the case when the distributed tracing and CAT headers are defined in the request headers. It handles this by first removing the existing headers, and then adding the expected headers for the request. However, if these tracing headers are present in the content headers collection, the agent does not detect them and it results in 2 tracing headers being present in the external request. This results in broken traces.

Expected Behavior

The agent should replace any tracing headers defined on both the request and the content, and the appropriate headers should be added to the request (not the content).

Troubleshooting or NR Diag results

Steps to Reproduce

1. Add a tracing header to an outbound HttpClient request using code like the following.
   request.Content.Headers.TryAddWithoutValidation("newrelic", "newrelic2");
2. Perform the outbound call in an instrumented app.
3. Observe in the downstream request that there are multple newrelic header values. One added by the agent and the other added using the above code.

Note that request.Headers.Add("newrelic", $"newrelic1"); does not result in multiple header values being observed in the downstream request.

Your Environment

Additional context

For Maintainers Only or Hero Triaging this bug
Suggested Priority (P1,P2,P3,P4,P5): P5
Suggested T-Shirt size (S, M, L, XL, Unknown): M (most of the effort is probably in the integration tests for the different tracing types -- W3C, New Relic DT, CAT)

[workato-integration]

https://issues.newrelic.com/browse/NR-150012