How does Microsoft.Extensions.Compliance.Redaction work

[bitbonk]

In the .NET Conf 2023 talk "Improving your application telemetry using .NET 8 and Open Telemetry" with @noahfalk and @samsp-msft Sam mentioned that there are now Microsoft.Extensions that help with redacting PII from logs. I suppose Microsoft.Extensions.Compliance.Redaction is what they meant with that. Are there any resources, articles, samples that help me understand how to use it?

I have tried to get a simple hello world worker service running where log message text is redacted in the console output but I could not figure out how a Redactor, a DataClassificationSet and the log entry relate to each other.

using Microsoft.Extensions.Compliance.Classification;
using Microsoft.Extensions.Compliance.Redaction;
using TelemetrySource;

var builder = Host.CreateApplicationBuilder(args);
builder.Services.AddHostedService<Worker>()
    .AddLogging(
        loggingBuilder => loggingBuilder
            // .EnableEnrichment()
            .EnableRedaction())
    // .AddProcessLogEnricher(o => o.ProcessId = true)
    .AddRedaction(
        r => { r.SetRedactor<MyRedactor>(new DataClassificationSet(new DataClassification("foo", "bar"))); } // what is "foo" and "bar"
        // .SetRedactor<ErasingRedactor>(new DataClassificationSet(new DataClassification("foo", "bar")))
        // .SetFallbackRedactor<ErasingRedactor>()
    );

var host = builder.Build();
host.Run();

public class MyRedactor : Redactor
{
    public override int Redact(ReadOnlySpan<char> source, Span<char> destination)
    {
        throw new NotImplementedException(); // never called
    }

    public override int GetRedactedLength(ReadOnlySpan<char> input)
    {
        throw new NotImplementedException(); // never called
    }
}

public class Worker : BackgroundService
{
    private readonly ILogger<Worker> _logger;

    public Worker(ILogger<Worker> logger)
    {
        _logger = logger;
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        while (!stoppingToken.IsCancellationRequested)
        {
            if (_logger.IsEnabled(LogLevel.Information))
            {
                _logger.LogInformation("foo bar: {foo} {bar}", "foo", "bar");
            }

            await Task.Delay(1000, stoppingToken);
        }
    }
}

[noahfalk]

@geeknoid - Any updated docs or doc plans to call out here? Could you also double check my explanation below appears accurate?

Hi @bitbonk. As best I know the folks building the new Microsoft.Extensions.* libraries are planning to start with README files for each library that covers the basics and then broader documentation on learn.microsoft.com is a work in progress. In the case of redacting for logging I think you are probably looking for something like this:

1. Define one or more attributes that derive from DataClassificationAttribute. For example:

[AttributeUsage(...)]
public class PersonalDataAttribute : DataClassificationAttribute
{
    public PersonalDataAttribute() : base("NameOfTaxonomy", "NameOfClassification") {}
    // both of those strings are arbitrary identifiers you can pick. You would use them
    // later when configuring redaction to set the policies for your different named classifications.
}

2. Annotate type properties or parameters to source generated logging methods with these attributes.

// Order type from the presentation
public class Order
{
    public int Id;
    public Sandwich Sandwich;
    public int StoreId;
    [PersonalData] public int UserId; // annotated property
}

// logging code that logs an Order
public static partial class Log
{
    [LoggerMessage(LogLevel.LogInformation, "New order created {order}")]
    public static partial void OrderCreated(this ILogger logger, [LogProperties] Order order);
}

3. By default parameters/fields marked with any data classification will have the ErasingRedactor applied to them, but you can use the AddRedaction() method to customize that redaction behavior. In the presentation UserId was included as a property in the logged message, but once you add this attribute it no longer will be there.

As an alternative to step 1+2 above you can get the IRedactorProvider from the service container, then get a redactor from the provider using GetRedactor(DataClassificationSet classifications). Once you have the redactor you can use it to transform any data you like explicitly.

Hope this helps as a starting point and I defer to @geeknoid when more broader docs will available.

[bitbonk]

@noahfalk Thanks for the quick reply. ❤️ I have create a demo repo to put it into practice.
But I do get two roslyn compiler errors here.

Error LOGGEN035 : Parameter "user" of logging method "UserLoggedIn" has a sensitive field/property in its type (https://aka.ms/dotnet-extensions-warnings/LOGGEN035)
Error CS8795 : Partial method 'Log.UserLoggedIn(ILogger, User)' must have an implementation part because it has accessibility modifiers.

My guess is that CS8795 is just the aftereffect of LOGGEN035. But how would I fix LOGGEN035?

[geeknoid]

The CS8795 warning is because the code generator is not actively being triggered and is not generating the implementation of the logging method. Let me take a look at your repo....

[geeknoid]

@joperezr Jose, I think we might have a problem with the msbuild magic to control logging code generators. This is a pretty straightforward use of the new logging generator, but the generator is not being run at build time. If I load the solution into VS and click on the Dependencies node in Solution Explorer, I can see the reference to Microsoft.Gen.Logging in there, but it says "this generator is not providing any files", which indicates the thing is not being run for some reason.

[joperezr]

@bitbonk I'll take a look at your demo repo and help figuring out what is wrong here.

[joperezr]

@joperezr When I convert the User record into a good old class, it works: bitbonk/LogRedactionDemo#1

Lol, seems like we commented the same thing at the same time 😄

[joperezr]

@xakep139 if what we are doing now is to disable the generator when we find a record type, we should fix that and instead throw an error so that the user understands what is happening, as opposed to just not emitting the generated code.

[bitbonk]

@joperezr @xakep139 @noahfalk Why is it that you do not support records with this? Was this a deliberate decision?
What just occurred to me is that records have a ToString implementation that writes out all properties. The way redaction currently works it seems that the data that should actually be redacted would end up in any case in the log entry. It would be kind of weird that redaction breaks if a logged type happens to have a ToString implementation. How can we protect ourselves from such error?

[xakep139]

@bitbonk Records are supported, the issue is that you had {User} in your [LoggerMessage] template - "User {User} logged in".
The source generator prevents you from logging records with sensitive properties, that's why you get LOGGEN035.
If you just remove {User} from the template, you'd still get record's values because you have used [LogProperties] - and all sensitive properties will be redacted.

[xakep139]

This is a PR with suggested changes: https://github.com/bitbonk/LogRedactionDemo/pull/2/files
And here's the console output of the program with these changes:

[bitbonk]

[joperezr]

LogProperties attribute will add the properties of the object as tags in a structured log. If what you are looking at is just at the regular logs in the Console, those are not structured so would still have the same output which is just calling User.ToString(). If you want to see the structured logs through the console to see the tags being sent (and the data being redacted), then simply modify your code to look like:

var builder = Host.CreateApplicationBuilder(args);
builder.Services.AddLogging(lb =>
{
    lb.EnableRedaction();
    lb.AddJsonConsole(); // <-- This line makes structured logs also get sent to the console.
    lb.Services.AddRedaction(); // <-- Make sure the redactor provider is hooked up so that the logger can get a redactor.
})
    .AddHostedService<Worker>();
var host = builder.Build();
host.Run();

And that should do the trick.

[geeknoid]

Yeah, we should add support to the console logger to output structured state. We had this as a work item at some point, but I guess it didn't get done. We have this in one of the internal R9 packages somewhere...

[bitbonk]

@joperezr Oh yes you are right, when User was still a record, I could see all properties on the console but that was due to the ToString override of the record. I was wrongly assuming this was due to the [LogProperties] attribute. I assumed it had the same effect as the @ ("{@User}") used for destructuring in Serilog.

[bitbonk]

@geeknoid

Yeah, we should add support to the console logger to output structured state.

I kind of like the destructuring approach in @nblumhardt's Serilog I mentioned above: https://nblumhardt.com/2014/07/using-attributes-to-control-destructuring-in-serilog/

[alexathomases]

As a follow-up, is it possible to use this library for redacting sensitive information from distributed traces? I tried folowing the implemention provided by @noahfalk and here, but the usage of LogProperties suggested that this redaction only applies to HTTP logging rather than tracing.

[geeknoid]

The code redaction infrastructure just operates against strings. You give it a data classification, get back a redactor, and then apply redaction on data N times.

Various higher-level components, take advaantage of the redaction infrastructure as part of their own features. This includes logging, such as when you use [LoggerMessage] and [LoggerProperties]. There is currently no integration for redaction in distributed tracing, but you can likely roll your own by buinding directly on the redaction infra.

[MCLifeLeader]

I have another follow up:

I forked the LogRedactionDemo from @bitbonk and discovered that inner objects do not seem to apply the Redaction Attribute to those property members on the inner object.

The following code in the following branch highlights what I found: https://github.com/MCLifeLeader/LogRedactionDemo/tree/failed_redaction_inner_record.

I was expecting that any inner objects with the appropriately applied redaction attribute would also be redacted as the logger serialized the original object for output. Should redaction be applied to the entire object graph on log serialization of an object or just the root object's property members?

public record User
{
    public User(string Id, [PersonalData] string Name, [PersonalData] string Email, InnerUserData innerData)
    {
        this.Id = Id;
        this.Name = Name;
        this.Email = Email;
        this.InnerData = innerData;
    }

    public string Id { get; }

    [PersonalData] 
    public string Name { get; }

    [PersonalData] 
    public string Email { get; }

    public InnerUserData InnerData { get; }
}

public record InnerUserData
{
    [PersonalData]
    public string RedactedData { get; } = Guid.NewGuid().ToString();
    public string PublicData { get; } = DateTime.UtcNow.ToString(CultureInfo.InvariantCulture);
}

[dariusclay]

Hi @MCLifeLeader -- we have this feature available from >= 8.1. You just need to use the transitive property on the LogPropertiesAttribute. I've created a small pull request on your repo which highlights the usage:
https://github.com/MCLifeLeader/LogRedactionDemo/pull/1/files

public static partial class Log
{
    [LoggerMessage(LogLevel.Information, "User logged in")]
#pragma warning disable EXTEXP0003 // Type is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed.
    public static partial void UserLoggedIn(this ILogger logger, [LogProperties(Transitive = true)] User user);
#pragma warning restore EXTEXP0003 // Type is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed.
}