Add header propagation functionality

[sw-joelmut]

Fixes #176

Description

This PR adds header propagation functionality, allowing to choose which headers to transition from incoming to outgoing requests.
This implementation uses an Http middleware, a context to share values, and the options to choose which headers to propagate.
Additionally, it will also work with hosted services, as it registers the new request headers into AsyncLocal for the current context by passing the headers via ActivityWithClaims.

How to use it

Most of the propagation configuration is done internally, so, a way to use this feature is adding the UseHeaderPropagation method call to the application builder as follows in the Program.cs file.

var app = builder.Build();
app.UseHeaderPropagation();

How it knows which headers to propagate

The header propagation functionality exposes a way to set which headers to propagate by using C# Attributes. Whenever a class uses the [HeaderPropagation] attribute, it will be automatically loaded into the header propagation context.
This functionality provides a collection to Add, Override, Append, and Propagate incoming request headers o outgoing ones.

Here is an example:

[HeaderPropagation]
internal class HeaderPropagation : IHeaderPropagationAttribute
{
    public static void LoadHeaders(HeaderPropagationEntryCollection collection)
    {
        // Propagate headers to the outgoing request by adding them to the HeaderPropagationEntryCollection.
    }
}

Detailed Changes

• Added HeaderPropagation folder containing all classes of this functionality.
  • Added HeaderPropagationMiddleware which will be used to intercept any incoming request, assigning the headers into a context for later use.
  • Added HeaderPropagationContext which is in charge of maintaining the headers over the current asynchronous control flow using AsyncLocal, also, it filters which headers have to save for the propagation by looking at the options configured in the middleware configuration.
• Added UseHeaderPropagation to the application builder extension that lets a customer enable the header propagation functionality.
• Added AddHeaderPropagation as a private application host extension to register the header propagation related classes.
• Added to the HostedActivityService the ability to read the headers from another http context by obtaining the headers from the ActivityWithClaims instance and assigning them to the new header propagation context.

Note

Working with Skills:
To propagate headers coming from a Root bot, the Skill will also need to register the middleware via UseHeaderPropagation.

Testing

The following image shows a testing header being propagated.

[tracyboehrer]

@sw-joelmut Oh nice. I have a little POC branch, but I like how this uses existing features. I'll take a look.

[sw-joelmut]

Hi @tracyboehrer, we added the C# Attributes to set which headers require propagation to outgoing requests. The implementation is working when we tested it, but after merging with the latest main changes, things changed and may not work for some scenarios (we suspect Skills).

We marked this PR as draft as it is not fully ready to merge with main, and we wanted to know if you like this new approach better, so we can do the final touches and ensure it is working correctly.

Let us know if there are some improvements that we could apply, thanks!

Note

The app.UseHeaderPropagation() middleware will still be required in the bot, but without parameters, as it is used to capture all incoming request headers.

[sw-joelmut]

Hi @tracyboehrer, we pushed the latest improvements for setting the header to propagate easier, and updated the PR's description to include attributes.

[Copilot]

Pull Request Overview

This PR introduces header propagation functionality by adding middleware to capture incoming request headers and propagate them to outgoing requests.

• Updates test mocks to include header parameters.
• Implements the UseHeaderPropagation extension and associated middleware.
• Adds core header propagation support with new context and attribute-based configuration.

Reviewed Changes

Copilot reviewed 18 out of 18 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
src/tests/Microsoft.Agents.Hosting.AspNetCore/CloudAdapterTests.cs	Updated unit tests to adjust for the new header parameter.
src/libraries/Hosting/AspNetCore/ServiceCollectionExtensions.cs	Added UseHeaderPropagation extension to register the middleware.
src/libraries/Hosting/AspNetCore/HeaderPropagationMiddleware.cs	New middleware for capturing headers from incoming requests.
src/libraries/Hosting/AspNetCore/CloudAdapter.cs	Updated to propagate headers via updated activity queue.
src/libraries/Hosting/AspNetCore/BackgroundQueue/*	Modified QueueBackgroundActivity signature and header copy handling.
src/libraries/Core/Microsoft.Agents.Core/HeaderPropagation/*	Added core header propagation types and attribute processing.
src/libraries/Client/*	Updated clients to include header propagation when creating HttpClient.

[Copilot]

The fallback using '[]' when headers is null may cause a type mismatch. Consider creating an empty HeaderDictionary instead, e.g. 'new HeaderDictionary()'.

Suggested change

	var copyHeaders = headers != null ? new HeaderDictionary(headers.ToDictionary()) : [];
	var copyHeaders = headers != null ? new HeaderDictionary(headers.ToDictionary()) : new HeaderDictionary();

[Copilot]

Passing '[header.Value]' wraps header.Value in an array, which may not match the expected type. It is likely more appropriate to pass 'header.Value' directly.

Suggested change

	httpClient.DefaultRequestHeaders.TryAddWithoutValidation(header.Key, [header.Value]);
	httpClient.DefaultRequestHeaders.TryAddWithoutValidation(header.Key, header.Value);

[Copilot]

When invoking a static method via reflection, the instance parameter should be 'null' rather than 'assembly'. Use 'loadHeaders.Invoke(null, new object[]{ HeaderPropagationContext.HeadersToPropagate })'.

Suggested change

	loadHeaders.Invoke(assembly, [HeaderPropagationContext.HeadersToPropagate]);
	}
	loadHeaders.Invoke(null, new object[] { HeaderPropagationContext.HeadersToPropagate });

[tracyboehrer]

@sw-joelmut Getting back to this. I'll let you know.

[tracyboehrer]

@sw-joelmut Does this also work for the Invoke/ExpectReplies/Streaming handling? Which doesn't use the queue.

[sw-joelmut]

@sw-joelmut Does this also work for the Invoke/ExpectReplies/Streaming handling? Which doesn't use the queue.

Hi Tracy, both queue and non-queue handling have the header propagation functionality. We will need to test the Streaming one as it was added around the same time as we created this PR and wasn't taken into account when testing.
We'll add a task to test this.

Thanks!

[tracyboehrer]

I suspect Stream won't impact the outgoing at all. It just skips the queue as Invoke and ExpectReplies does.

[sw-joelmut]

I suspect Stream won't impact the outgoing at all. It just skips the queue as Invoke and ExpectReplies does.

Hi Tracy, we tried the Stream delivery mode, and it is working as expected as this scenario don't send another request.
Additionally, while testing Stream with Agent-to-Agent communication, we found a small bug that we fixed in PR #300.

[tracyboehrer]

I believe this is populated by virtue of the attribute handling, yes? There is a question about concurrency. We do something similar with the serializer options init (via attributes). I added a lock there because I wasn't sure about the exact module loading thread safety. But I'm not 100% that is required.

[ceciliaavila]

We added a lock to the Add operations in HeaderPropagationEntryCollection. Thanks!

[tracyboehrer]

Take a look at SerializationInitAttribute.GetLoadOnInitTypes. We had to add exception handling because some modules failed on this, and the app would crash. See if this is relevant here too.

[ceciliaavila]

Thanks Tracy, we applied the changes.

[ceciliaavila]

Hi @tracyboehrer, we added unit tests to illustrate the different operations that can be done over the headers (add, override, append, and propagate).