Filtering Duplicate logs

[iamterryclark]

Aim:

I would like to create a hook which skips duplicate logs that are created but prints one line with the number of skips. I am trying out a similar approach to SDPlogs handling of this. I am not able to get the desired result.

What I have done:

In the filter function I want to capture all duplicate logs within a 10 second span so max_skip_duration = 10000. I make json objects from the data to compare them, as they could be arrays.

I created another logger dup_logger which then logs to the same file instead of returning out. This results in all my duplicate log message logging out but my other logs do not get through unless i return message instead of false inside the filtered check.

skip_counter and last_msg are global value I have set.

I would appreciate if someone could point me in the right direction for this to work properly?

Example Code:

const log = require('electron-log');

const setupLogger = () => {
    //Create Instance of Logger
    let logger = log.create("logger");
    let dup_logger = log.create("duplicate_logger");
  
    //Create string for filename
    let filename = 'test-duplicate-log.log';
  
    logger.transports.file.resolvePath = () => filename;
    dup_logger.transports.file.resolvePath = () => filename;

    logger.transports.file.maxSize = 10 * 1024 * 1024;
    dup_logger.transports.file.maxSize = 10 * 1024 * 1024;

    let last_msg = {date: new Date()};
    let max_skip_duration = 2000;
    let skip_counter = 0;

    const filter= (msg) => {
        let filter_duration = msg.date - last_msg.date;
        return (filter_duration > max_skip_duration) || (JSON.stringify({msg: msg.data}) !== JSON.stringify({msg: last_msg.data}))
    }
    
    //Setup duplicate log filter
    logger.hooks.push(
        (message) => {
            let filtered= filter(message);

            if (!filtered)
            {
                skip_counter += 1;
                return false;
            }
            
            // log the "skipped.." message
            if (skip_counter > 0)
            {
                let skipped_msg = 'Skipped above duplicate message ' + skip_counter + ' times ..';
                dup_logger.info(skipped_msg);
            }

            // log current message
            skip_counter = 0;
            last_msg = message;

            // return message;
            //fixed by sending message to duplicate logger
            dup_logger.logMessageWithTransports(message, [logger.transports.file, logger.transports.console]);
        }
    );
    return logger;
}

let logger = setupLogger();

logger.info("Test1");
logger.error("Test2");

let i = 0;
while (i < 100000)
{
    logger.info("duplicate-test");
}

[megahertz]

The code looks fine, but the result depends on the other details. Could you share the whole code to help me reproduce that?

[iamterryclark]

Updated the example above... you will see that test1 and test2 will not print unless return message instead of false.

[iamterryclark]

FYI I was using this as a base where they sink the log rather than return out

https://github.com/gabime/spdlog/blob/c23430b438e9740419375205eadbcd84360e40f0/include/spdlog/sinks/dup_filter_sink.h

[megahertz]

I'll look at the source in the evening. At the moment, please be aware that the hook is called once per each transport, so if file and console transports are configured it's called two times per each message.

[iamterryclark]

ah that makes sense I am seeing them on the console just not on the log itself.

[iamterryclark]

I have fixed it, it seems that I just needed to parse the message over to the dup_logger instead of returning out
Edited above