V8 LogViewer

[warrenbuckley]

This is a work in progress to consider adding in a built in LogViewer to Umbraco V8 Core

Why do this?

Currently, for all Cloud sites, we use a community package called Trace Log Viewer which is a popular community package.

It makes sense to have this part of the core Umbraco product to have a way to view, filter and query the logs that we are reporting from Umbraco, as opposed to a community package. This way everyone benefits from it, not just sites running on Umbraco Cloud.

With the upcoming Umbraco Headless project, there is no direct file access and this will be the only way for customers to see & view the logs that the Umbraco headless project is reporting.

What does it do?

This PR adds in a new tree to the Developer section called Logs that allows users to:

• View logs directly in the backoffice
• Filter/toggle logs by log level
• Quick glance of number of errors
• Chart displaying the distribution of log level types
• List of most common log message templates, to determine common log patterns or errors
• Perform advanced expression/searches to find logs

Extending/swapping out the LogViewer datasource

It is possible to swap out the default LogViewer datasource from reading the JSON files off disk, by disabling the Umbraco core component and registering our own component to put into the DI container.

The example below reads JSON logs stored in Azure Table Storage, that a user may configure V8 to store logs with Serilog to be logged into Azure Table storage with the off shelf Serilog sink.

Setup

• Install Serilog.Sinks.AzureTableStorage from Nuget (Latest pre-release)
• Update `config/serilog.user.config with credentials (so logs persist to Azure)

<add key="serilog:using:AzureTableStorage" value="Serilog.Sinks.AzureTableStorage" />
<add key="serilog:write-to:AzureTableStorage.connectionString" value="DefaultEndpointsProtocol=https;AccountName=ACCOUNT_NAME;AccountKey=KEY;EndpointSuffix=core.windows.net" />
<add key="serilog:write-to:AzureTableStorage.formatter" value="Serilog.Formatting.Compact.CompactJsonFormatter, Serilog.Formatting.Compact" />

Register component (that disables the core one)

using Umbraco.Core.Components;
using Umbraco.Core.Composing;
using Umbraco.Core.Logging.Viewer;

namespace Umbraco.Web.UI
{
    [DisableComponent(typeof(LogViewerComponent))]
    public class AzureTableLogComponent : UmbracoComponentBase, IUmbracoUserComponent
    {
        public override void Compose(Composition composition)
        {
            composition.Container.RegisterSingleton<ILogViewer, AzureTableLogViewer>();
        }
    }
}

Implement ILogViewer

using Microsoft.WindowsAzure.Storage;
using Microsoft.WindowsAzure.Storage.Table;
using Serilog.Events;
using Serilog.Formatting.Compact.Reader;
using Serilog.Sinks.AzureTableStorage;
using System;
using System.Collections.Generic;
using Umbraco.Core.Logging.Viewer;

namespace Umbraco.Web.UI
{
    public class AzureTableLogViewer : LogViewerSourceBase
    {
        
        public override bool CanHandleLargeLogs => true;

        public override bool CheckCanOpenLogs(DateTimeOffset startDate, DateTimeOffset endDate)
        {
            //This method will not be called - as we have indicated that this 'CanHandleLargeLogs'
            throw new NotImplementedException();
        }

        public override IEnumerable<LogEvent> GetLogs(DateTimeOffset startDate, DateTimeOffset endDate, ILogFilter filter, int skip, int take)
        {
            var cloudStorage = CloudStorageAccount.Parse("DefaultEndpointsProtocol=https;AccountName=warrenlogstorage;AccountKey=/NuiVmUiRkPmkg+zY/Gnz5/Lw1ErxZm8cW2XwR4DwhoMvJ0TSo987nSW7lDJHrciW2EyaZv948DLz+GJujqjnA==;EndpointSuffix=core.windows.net");
            var tableClient = cloudStorage.CreateCloudTableClient();
            var table = tableClient.GetTableReference("LogEventEntity");

            var logs = new List<LogEvent>();
            var count = 0;
            
            // Create the table query
            // TODO: Use a range query to filter by start & end date on the Timestamp
            var query = new TableQuery<LogEventEntity>();
            var results = table.ExecuteQuery(query);

            // Loop through the results, displaying information about the entity.
            foreach (var entity in results)
            {
                //Reads the compact JSON format stored in the 'Data' column back to a LogEvent
                //Same as the JSON txt files does
                var logItem = LogEventReader.ReadFromString(entity.Data);
                
                if (count > skip + take)
                {
                    break;
                }

                if (count < skip)
                {
                    count++;
                    continue;
                }

                if (filter.TakeLogEvent(logItem))
                {
                    logs.Add(logItem);
                }

                count++;
            }

            return logs;
        }
    }
}

TODO:

• Date picker range & filtering for logs Removed for first iteration
• Total count for distinct message templates
• Search/Filter dropdown for common searches (Base it on Content Node Name Variants)
• Fix Error exception message styling - rather than big ugly red background
• Click a common logtemplate - performs the search/expression for that messagetemplate

[warrenbuckley]

@Shazwazza & @zpqrtbnk I would love some early tech/code review on this please before I code myself into a corner & to ensure it is performant as it can be, Thx 😄