How to use structured logging

NLog 4.5 introduces structured logging - also called semantic logging. This document describes how to use structural logging. The implementation in NLog supports the message templates syntax, so it is recommended to check: https://messagetemplates.org/

Why structured logging

Structured logging makes it easier to store and query log-events, as the logevent message-template and message-parameters are preserved, instead of just transforming them into a formatted message.

The normal .NET string.Format(...) will only accept input strings like this:

logger.Info("Logon by user:{0} from ip_address:{1}", "Kenny", "127.0.0.1");
logger.Debug("Shopitem:{0} added to basket by user:{1}", "Jacket", "Kenny");

When storing these log-events in a database (or somewhere else), then it can be difficult to query all actions performed by a certain user. Also it could be hard to group similar events.

The workaround would then be to perform RegEx-queries to recognize logevent messages and convert them into a more structured format, and register which parameter index is the user. The RegEx might even have to extract the needed details from the formatted message, making it even more fragile. Maintaining these RegEx-queries can become rather cumbersome.

Further history: messagetemplates.org

Using structured logging

NLog has always supported log-event metadata called event-properties, but it requires a little effort to generate a log-event with properties. See also the Fluent-Logger-API

NLog 4.5 makes it possible to capture log-event-properties from the log-message-template, so they can be easily processed by the NLog destination target:

logger.Info("Logon by {user} from {ip_address}", "Kenny", "127.0.0.1"); // Logon by "Kenny" from "127.0.0.1"
logger.Debug("{shopitem} added to basket by {user}", new { Id=6, Name = "Jacket", Color = "Orange" }, "Kenny");

Any NLog destination target that is able to handle log-event-properties will automatically experience the benefit of doing structured logging.

Formatting of the message

The formatting of message depends on the datatype of the parameters.

Example:

logger.Info("Order {orderid} created for {user}", 42, "Kenny");

Will be formatted as:

Order 42 created for "Kenny"

The formatting is controlled by the datatype of the parameter:

• string: surrounded with double quotes, e.g "hello"
• number: no quotes
• null: printed as NULL
• list/ienumerable/array: "item1", "item2"
• dictionary: "key1"="value1", "key2"="value2"
• objects: ToString()

It's possible to control formatting by preceding @ or $:

• @ will format the object as JSON
• $ forces ToString()

Examples

Object o = null;

logger.Info("Test {value1}", o); // null case. Result:  Test NULL
logger.Info("Test {value1}", new DateTime(2018,03, 25)); // datetime case. Result:  Test 25-3-2018 00:00:00 (locale TString)
logger.Info("Test {value1}", new List<string> { "a", "b" }); // list of strings. Result: Test "a", "b"
logger.Info("Test {value1}", new[] { "a", "b" }); // array. Result: Test "a", "b"
logger.Info("Test {value1}", new Dictionary<string, int> { { "key1", 1 }, { "key2", 2 } }); // dict. Result:  Test "key1"=1, "key2"=2

var order = new Order
{
    OrderId = 2,
    Status = OrderStatus.Processing
};

logger.Info("Test {value1}", order);  // object Result: Test MyProgram.Program+Order
logger.Info("Test {@value1}", order); // object Result: Test {"OrderId":2, "Status":"Processing"}
logger.Info("Test {value1}", new { OrderId = 2, Status = "Processing"});  // anonymous object. Result: Test { OrderId = 2, Status = Processing }
logger.Info("Test {@value1}", new { OrderId = 2, Status = "Processing"}); // anonymous object. Result: Test {"OrderId":2, "Status":"Processing"}

Output captured properties

When formatting the log message, then it will automatically serialize the parameters (see Formatting of the message). It is also possible to configure NLog Layout to serialize the individual properties. This can be done by using ${event-properties} or ${all-event-properties}, for example:

    <target xsi:type="File" name="jsonFile" fileName="c:\temp\nlog-json-${shortdate}.log">
      <layout>${longdate}|${level}|${logger}|${message}|${all-event-properties}{exception:format=tostring}</layout>
    </target>

NLog Layouts with option includeEventProperties="true" (Before NLog 5.0 it was includeAllProperties="true") will output captured properties from structured logging:

• Json Layout
• Log4JXml Layout
• Xml Layout

It can be configured like this:

    <target xsi:type="File" name="jsonFile" fileName="c:\temp\nlog-json-${shortdate}.log">

      <layout xsi:type="JsonLayout" includeEventProperties="true">
        <attribute name="time" layout="${longdate}" />
        <attribute name="level" layout="${level:upperCase=true}"/>
        <attribute name="message" layout="${message}" />
      </layout>

    </target>

NLog 4.5 adds support for rendering the raw message-template, instead of just the formatted message:

• ${message:raw=true}

NLog 4.5 extends the following NLog LayoutRenderers with support for serializing with property reflection into JSON when format="@"

• ${event-properties}
• ${exception}
• ${gdc} - Global Context
• ${scopeproperty} - Thread Context (Before NLog 5.0 it was ${mdc} + ${mdlc})

In code you can use LogEventInfo.Properties. If having special interest in format-options for the captured parameters then one can use LogEventInfo.MessageTemplateParameters

NLog Target Support

Any NLog destination target that handles event-properties will be able to take advantage of structured logging. Many targets already has support for the option includeEventProperties="true", that comes automatically when inheriting from TargetWithContext.

NLog Targets with supports for NLog Layout as payload will automatically support structured logging:

• NLogViewer Target makes use of Log4JXml Layout.
• WebService Target can perform JSON-post with Json Layout.
• Network Target can publish both Json Layout and Xml Layout to any TCP/UDP-endpoint.
• Database Target can write to an XML column using Xml Layout

There are already many custom NLog targets that provide the ability to store log-events in central storage, and allow a Web-Application to query the log-events along with their event-properties. These targets will automatically benefit from structured logging, and allow the Web-Application to perform effective queries.

Transform captured properties

Extract single property

NLog 4.6.3 allows you to navigate into a complex object and extract a single property. Ex:

logger.Debug("{shopitem} added to basket by {user}", new { Id=6, Name = "Jacket", Color = "Orange" }, "Kenny");

Then one can extract specific details from shopItem-property:

      <layout xsi:type="JsonLayout">
        <attribute name="shopitemId" layout="${event-properties:shopitem:objectPath=Id}" />
      </layout>

Customize Object Reflection

To avoid object reflection for a custom type, then just implement the IFormattable-interface, and NLog will call ToString() instead of doing reflection of properties.

NLog 4.7 allows you to transform large (or dangerous) objects into a more reduced/safe object:

LogManager.Setup().SetupSerialization(s =>
   s.RegisterObjectTransformation<System.Net.WebException>(ex => new {
      Type = ex.GetType().ToString(),
      Message = ex.Message,
      StackTrace = ex.StackTrace,
      Source = ex.Source,
      InnerException = ex.InnerException,
      Status = ex.Status,
      Response = ex.Response.ToString(),  // Call your custom method to render stream as string
   })
);

This can also be used to output object-fields, as the NLog Object Reflection only sees object-properties. By transforming into an Anonymous Type, and assigning the wanted object-fields to named properties.

If more dynamic reflection is required, then one can also just return a Dictionary<string, object> instead of returning an Anonymous Type (The performance hit is a little higher).

Note when making an override for an object-type, then the returned object-type must be of a different object-type. Because NLog caches the reflection of the return object-type, and afterwards NLog skips checking for transformation for already cached object-types.

Advanced

How to override parameter value formatter

Create a new class that NLog.IValueFormatter and set NLog.Config.ConfigurationItemFactory.Default.ValueFormatter

How to override JSON formatter

You can override the builtin NLog JSON converter with a different implementation.

You need a custom NLog.IJsonConverter and set NLog.Config.ConfigurationItemFactory.Default.JsonConverter

Example of using JSON.NET that supports JSON.NET object annotations like [JsonIgnore]:

    internal class JsonNetSerializer : NLog.IJsonConverter
    {
        readonly JsonSerializerSettings _settings;

        public JsonNetSerializer()
        {
           _settings = new JsonSerializerSettings
                {
                    Formatting = Formatting.Indented,
                    ReferenceLoopHandling = ReferenceLoopHandling.Ignore
                };
        }

        /// <summary>Serialization of an object into JSON format.</summary>
        /// <param name="value">The object to serialize to JSON.</param>
        /// <param name="builder">Output destination.</param>
        /// <returns>Serialize succeeded (true/false)</returns>
        public bool SerializeObject(object value, StringBuilder builder)
        {
            try
            {
                var jsonSerializer = JsonSerializer.CreateDefault(_settings);
                var sw = new System.IO.StringWriter(builder, System.Globalization.CultureInfo.InvariantCulture);
                using (var jsonWriter = new JsonTextWriter(sw))
                {
                    jsonWriter.Formatting = jsonSerializer.Formatting;
                    jsonSerializer.Serialize(jsonWriter, value, null);
                }
            }
            catch (Exception e)
            {
                NLog.Common.InternalLogger.Error(e, "Error when custom JSON serialization");
                return false;
            }
            return true;
        }
    }

Combine indexed and structured logging

e.g. logging "Hello {0} with {Message}"

For backward compatibility (and performance), then NLog will by default skip full parsing if it detects the first parameter as being positional index. This will ensure repeated positional-placeholders supported by string.Format will continue to work:

string.Format("{1}{0}{2}{0}", Environment.NewLine, "Hello", "World");

When parseMessageTemplates='true' then NLog will always parse all parameters, and if one is non-numeric, then all the parameters will be treated as structured parameters. Mixing and numeric and structured parameters is not recommended, as it hurts performance (backtrack) and numeric parameters are most of the time not so descriptive.

Parameter Names

The Names of the parameters should be unique

I just need the template

e.g. Order {orderId}

Use ${message:raw=true}

Disabling Structured Logging

NLog will by default attempt to parse log-events as structured log-events. This gives a minor overhead, that will not be noticable by most.

It is possible to disable this behavior, telling NLog it should not attempt to parse log-events as structured log-events, with this xml-setting:

<nlog parseMessageTemplates="false">
    ...
</nlog>