Formalize the concept of a configuration server

[samsp-msft]

What should we add or change to make your life better?

In the scenarios where multiple YARP instances are being used, you need to manage the configuration for each instance independently, which can be a bunch of work.
The model when integrating YARP with a different process management system such as k8s or Service Fabric, is to typically have a process that takes their model, and maps it over to YARP's, which is then serialized and given to the YARP process.

Why is this important to you?

While you can roll your own today, we should build the capability into YARP for proxy instances to be able to talk to a configuration server and make the server easy to operate.

Proposal

Using a configuration server in a YARP proxy instance, should be as simple as pointing YARP at the configuration endpoint

 services.AddReverseProxy()
    .UseConfigServer("https://my.config.server/YarpConfig", sharedKey);

To create the config server should be very similar to how you configure YARP today. You should be able to use exactly the same configuration files, custom providers, filters etc that you would with a local instance.

services.AddReverseProxyConfigServer(sharedKey)
    .LoadFromConfig(Configuration.GetSection("ReverseProxy"))

You then expose an endpoint for the proxy instances to talk to using:

public void Configure(IApplicationBuilder app, IProxyStateLookup lookup)
{
    app.UseRouting();
    app.UseEndpoints(endpoints =>
    {
        endpoints.MapReverseProxyConfig("/YarpConfig");
    });
}

This will create an endpoint that the proxy will listen on. The endpoint will provide APIs for the configuration data, and a way that the proxy instances can listen for change notifications. Making a change to configuration on the config server should notify each of the clients and then they can pull the new configuration down.

There needs to be a way to authenticate the proxies, probably using some form of shared secret, that is then hashed and sent over the wire.

Dynamic destination registration

One way this could be extended would be for there to be a way for destinations to register and unregister themselves against specific clusters.

[thompson-tomo]

For efficiency I feel that this should be achieved by Aspire (dotnet/aspire#3985) so that other projects/app can benefit.