API Design

[NachoVazquez]

This discussion will contain documents focused on helping design Lumberjack's Core following API.

We will start describing the current API followed by other libraries' APIs. This will help us understand the state of the art regarding logging libraries and inspire us to find the best compromise.

Important The Lumberjack Core must be designed to make it easy to use directly, but its primary usage will be through framework-specific clients wrapping it.

The library must find the right balance between ease of use and extensibility.

Since the library will be combined with other libraries, it should be as light as possible.

[NachoVazquez]

Current Lumberjack Core API

Creating and configuring a lumberjack instance

To create a lumberjack instance, you need to provide a list of drivers and a
configuration object.

The lumberjack instance is generic over the type of the payload of the logs.
The default payload type is void. See more at TypeScript.

const drivers = [...];
const isProduction = true;
const options = {};
const config = createLumberjackConfig(isProduction, options);

let lumberjack = createLumberjack<TPayload>({
  drivers: drivers,
  config: config,
});

We will discuss the different parts of the creation API in the following sections.

Drivers

The drivers are the plugins used to write the logs.

Other logging libraries call them "transports," "sinks," "monitors," etc.

A valid driver is an object whose structure matches the LumberjackDriver interface.

/**
 * The interface implemented by drivers. Optionally supports a log payload.
 */
export interface LumberjackDriver<
  TPayload extends LumberjackLogPayload | void = void,
> {
  /**
   * Driver settings.
   */
  readonly config: LumberjackDriverConfig;
  /**
   * A critical log and its text representation is passed to this method.
   */
  logCritical(driverLog: LumberjackDriverLog<TPayload>): void;
  /**
   * A debug log and its text representation are passed to this method.
   */
  logDebug(driverLog: LumberjackDriverLog<TPayload>): void;
  /**
   * An error log and its text representation is passed to this method.
   */
  logError(driverLog: LumberjackDriverLog<TPayload>): void;
  /**
   * An info log and its text representation are passed to this method.
   */
  logInfo(driverLog: LumberjackDriverLog<TPayload>): void;
  /**
   * A trace log and its text representation is passed to this method.
   */
  logTrace(driverLog: LumberjackDriverLog<TPayload>): void;
  /**
   * A warning log and its text representation are passed to this method.
   */
  logWarning(driverLog: LumberjackDriverLog<TPayload>): void;
}

A LumberjackDriver is composed of a configuration object and six methods, one
for each supported log level.

The driver config can be extended, but in its basic form, it is composed
by a levels property indicating the log leaves this particular driver
is allowed to log and an identifier property, which is a string used
to identify the driver (This can support driver querying
if we plan to implement complex log restrictions beyond the level filter).

Each log method receives a LumberjackDriverLog object, which is composed by
the structured log object and the formatted text representation of the log.

/**
 * The data structure is passed to a driver by Lumberjack. Optionally supports
 * a log payload.
 */
export interface LumberjackDriverLog<
  TPayload extends LumberjackLogPayload | void = void,
> {
  /**
   * The text representation of the log.
   */
  readonly formattedLog: string;
  /**
   * The log. Optionally supports a log payload.
   */
  readonly log: LumberjackLog<TPayload>;
}

We will learn more about the LumberjackLog type in the Writing logs section.

With that said, we can create a driver by using a class or a factory
a function that returns an object with that structure.

Therefore, creating the driver list to pass to the createLumberjack function
can look like this:

const drivers = [createConsoleDriver(...), new HttpDriver(...), ...]

Config

The Lumberjack configuration is an object with two properties:
A format function and a levels property.

/**
 * Settings used internally by Lumberjack services.
 */
export interface LumberjackConfig<
  TPayload extends LumberjackLogPayload | void = void,
> {
  /**
   * The Lumberjack format function used to generate the text representation of
   * a log.
   */
  readonly format: LumberjackFormatFunction<TPayload>;
  /**
   * The default log level filter.
   */
  readonly levels: LumberjackConfigLevels;
}

This object comprises a format function and a levels property.

The format function is used to generate the text representation of a log.
This is the generated text passed to the drivers as seen
in the drivers section.

The levels property is a list of the globally allowed log levels.
This is used to filter the logs before passing them to the drivers.

It can either be a list of log levels or a list of a single log level:
LumberjackLevel.Verbose, which represents all levels.

/**
 * A set of Levels used to configure Lumberjacks and drivers. Lumberjack filters logs
 * passed to the driver based on its configured log levels.
 */
export type LumberjackConfigLevels =
  | LumberjackLogLevel[]
  | [LumberjackLevel.Verbose];

Lumberjack also provides a helper function to create the config object.

/**
 * Helps combine the default Lumberjack configurations with custom developer configurations.
 * @param isProductionEnvironment - Lumberjack uses different default log levels based on its running environment.
 * @param options - LumberjackOptions that overwrite the default configuration.
 * @returns - The combination of default configs and custom overwrites.
 */
export function createLumberjackConfig<
  TPayload extends LumberjackLogPayload | void = void,
>(
  isProductionEnvironment: boolean,
  options: LumberjackOptions = {},
): LumberjackConfig<TPayload> {
  return {
    format: lumberjackFormatLog<TPayload>,
    levels: isProductionEnvironment
      ? defaultProductionLevels
      : defaultDevelopmentLevels,
    ...options,
  } as LumberjackConfig<TPayload>;
}

This function takes in a boolean indicating if the environment is production or not
and an optional object with the custom options.

The isProductionEnvironment flag is used to set the default log levels.
In a production environment, the default log levels are LumberjackLogLevel.Warning
and LumberjackLogLevel.Error. In a nonproduction environment, the default log levels
are LumberjackLogLevel.Verbose. These are reasonable defaults but can be overwritten
by the developer through the options.

Another default set by this function is the format function. This function is used to generate
the text representation of the log. The default implementation is lumberjackFormatLog
which combines the log properties to form a string representation.

/**
 * Default function used by Lumberjack to format the Lumberjack logs.
 *
 * This function formats a LumberjackLog object into a string with a specific format:
 * "<level> <timestamp> [<scope>] <message>". If the scope is not provided, it will be omitted.
 *
 * @example
 * const log = {
 *  scope: 'Application',
 *  createdAt: new Date(),
 *  level: 'ERROR',
 *  message: 'An unexpected error occurred',
 * };
 *
 * console.log(lumberjackFormatLog(log));
 * // Outputs: "ERROR 2023-06-22T15:23:42Z [Application] An unexpected error occurred"
 *
 */
export function lumberjackFormatLog<
  TPayload extends LumberjackLogPayload | void = void,
>({
  scope,
  createdAt: timestamp,
  level,
  message,
}: LumberjackLog<TPayload>): string {
  const formattedScope = scope ? ` [${scope}]` : "";

  return `${level} ${utcTimestampFor(timestamp)}${formattedScope} ${message}`;
}

Both the format and the levels can be overwritten by
the developer through the options object.
This object is a partial of the LumberjackConfig interface.

/**
 * Shared Lumberjack settings used by `LumberjackModule`.
 */
export type LumberjackOptions = Partial<LumberjackConfig>;

Writing logs

Once created, the lumberjack instance can be used to write logs.

The basic syntax looks like this:

const lumberjack = createLumberjack(...);

lumberjack.log({...});

Where the log method receives a LumberjackLog object.

The LumberjackLog object contains the basic information of the log:

• An optional scope or context: useful to mark the origin of the log.
• The level: the severity of the log.
• The message: a description of the event.
• The createAt timestamp: when the log was created.
• The optional payload or metadata: additional information about the event.

/**
 * A Lumberjack log. Optionally supports a payload.
 */
export interface LumberjackLog<
  TPayload extends LumberjackLogPayload | void = void,
> {
  /**
   * Unix epoch ticks in milliseconds, representing when the log was created.
   */
  readonly createdAt: number;
  /**
   * Level of severity.
   */
  readonly level: LumberjackLogLevel;

  /**
   * Log message, for example, describing an event that happened.
   */
  readonly message: string;
  /**
   * Optional payload with custom properties.
   *
   * NOTE! Make sure that your drivers support these properties.
   */
  readonly payload?: TPayload;
  /**
   * Scope, for example, domain, application, component, or service.
   */
  readonly scope?: string;
}

If more specific information is needed, it can be passed to the payload object.

If we used every property of the LumberjackLog interface,
the log object looks like this:

lumberjack.log({
  createdAt: new Date(dateTime).valueOf(),
  level: LumberjackLogLevel.Info,
  mesaage: "A log message",
  payload: {
    correlationId: "123",
    email: "test@tester.com",
  },
  scope: "auth.signin",
});

Note: The createdAt property is exposed to allow the developer to have control
over this important value. The philosophy of Lumberjack is to be as flexible as
possible and let the developer decide how to use the library. However, the library
also attempts to provide reasonable defaults to ease simple use cases.
See more at LumberjackLogBuilder.

This can be a bit lengthy, so the library provides a set of helpers.

The foundation of all log creation assistance is the LumberjackLogBuilder class.

LubmerjackLogBuilder

The constructor of the LumberjackLogBuilder class receives three arguments:

• the log level
• the message
• an optional getUnixEpochTicks.

This basic usage looks like this:

const logBuilder = new LumberjackLogBuilder(
  LumberjackLogLevel.Info,
  "A log message",
);

The default value for the getUnixEpochTicks argument is the following function:

() => new Date().valueOf();

Once the builder is created, it can be used to create a log object.

// Simple log
lumberjack.log(logBuilder.build());

// Enhanced log
lumberjack.log(
  logBuilder
    .withScope("auth.signin")
    .withPayload({
      correlationId: "123",
      email: "test@tester.com",
    })
    .build(),
);

// Or
lumberjack.log(
  logBuilder.withScope("auth.signin").build({
    correlationId: "123",
    email: "test@tester.com",
  }),
);

Note: To add a payload to the log the instance of the builder needs to have
a generic type. See more at TypeScript.

But the builder can be still verbose. For that reason the library provides
other helper functions based on the builder.

They all have the same signature, there is one for each log level.

// Split into three lines
const infoLogBuilderFactory = createInfoLogBuilder();
const infoLogBuilder = infoLogBuilderFactory("A log message");
lumberjack.log(infoLogBuilder.build());

// Split into two lines
const infoLogBuilder = createInfoLogBuilder()("A log message");
lumberjack.log(infoLogBuilder.build());

// Or in one line
lumberjack.log(createInfoLogBuilder()("A log message").build());

// Complex log
luumberjack.log(
  createInfoLogBuilder()
    .withScope("auth.signin")
    .withPayload({
      correlationId: "123",
      email: "test@tester.com",
    })
    .build(),
);

TypeScript

The Lumberjack library has a strong focus on type safety and aims
to provide a good developer experience.

The library is generic over the type of the payload of the logs.
This drives how the library is used and how the logs are written.

The default payload type is void which means that the logs
do not have a payload.

The specific payload type can be passed to the createLumberjack function
and it is used across all the library APIs.

const lumberjack = createLumberjack<CustomPayload>(...);

lumberjack.log(createInfoLogBuilder<CustomPayload>()("A log message").build({myCustomPayloadObject}));

If there is a mismatch between the payload type and the payload object
a TypeScript error is shown.

const lumberjack = createLumberjack<CustomPayload>(...);

//  Argument of type 'LumberjackLog<void>' is not assignable to parameter
// of type 'LumberjackLog<CustomPayload>'.Type 'void' is not assignable to type 'CustomPayload'.
lumberjack.log(createInfoLogBuilder()("A log message").build());

If the user wants to send a payload but it doesn't want to specify the payload type,
it can use the LumberjackLogPayload type.

```ts
const lumberjack = createLumberjack<LumberjackLogPayload>(...);
lubmerjack.log(createInfoLogBuilder<LumberjackLogPayload>()("A log message").build(anyObject));

LumberjackLogPayload is an alias for Record<string, unknown>.

export interface LumberjackLogPayload {
  /**
   * A custom property for a log.
   *
   * NOTE! Make sure that this property is supported by your drivers.
   */
  readonly [property: string]: unknown;
}

Error handling

Although nothing is obviously exported as error handling by the library it is worth mentioning about some "hidden behaviors" that the application would perform and that directly affects the users.

Lumberjack implements an error recovering mechanism that cannot be modified (at the moment of writing this document). From the error handling perspective the algorithm behaves in the following way:

1. A logs is passed to Lumberjack
2. Lumberjack iterates over all drivers and tries to send the log to them
3. If a driver was processed correctly it will be added to a "stable driver" list
4. If a driver errors, a LumberjackDriverError is created and added to the driver's error list.
5. After processing all drivers the driver's error list is processed
6. If the list of stable drivers is empty the errors are logged to console
7. If the list of stable drivers has drivers the log errors are attempt to send to the stable drivers using the same algorithm and the driver is removed from the stable driver's list.
8. This process is recursive, eventually all driver's errors are logged to the remaining stable drivers or to console.

[pBouillon]

From my point of view, when compared to winston, js-logger or ngx-logger, lumberjack is lacking of simplicity to log things.

The path from setting up the logger to actually logging something is a too complex and might drive people away from lumberjack as a logging library for their projects.

However, lumberjack also has several advantages that makes its strength like its flexibility, its bundle size and its design as a framework agnostic tool. The only think I feel left behind is the developer experience while using it.

In order to help on that part, we could consider an additional library, acting as an adapter on top of the core. This way, parts that requires the "native API" could still rely on it, while others could use the simpler one.

It would be important that this library:

• Does not depend on another framework itself
• Is not used by framework-specific implementations rather than the core package, in order to reduce the layers and thus the bundle size
• As light as possible not to negate lumberjack's focus on small bundle

[NachoVazquez]

Agreed. That should be our goal. To improve the DX and remove complexity from the most common uses of the library while keeping it flexible.

One of the solutions you suggest is to keep the core less accessible but ship wrappers from day one to simplify its usage.

We should then discuss both the API of the core library and the initial wrapper.

Let's consider that many clients or wrappers are planned, for example, for Angular, React, Solid, NestJS, etc. So, this initial wrap can be shallow but still easy to use if developers prefer the small, agnostic wrapper or core.