Enabling error message truncation

[devpolo]

Is your feature request related to a problem? Please describe.

Entire app is crashing with Log entry with size 300.0K exceeds maximum size of 256.0K error. This is happening for exemple with prisma when some errors are catched after huge failed SQL queries.

Describe the solution you'd like

Add a default error message truncation and an maxEntrySize param. E.g. with winston

Describe alternatives you've considered

Truncating error message in NestJs logger class extension:

import { Injectable } from '@nestjs/common';
import { Logger } from 'nestjs-pino';

@Injectable()
export class LoggerService extends Logger {
  error(message: any, ...context: any[]): void {
    super.error(this.limitErrorMessageTo256KB(message));
  }

  // log, warn, etc..

   /**
   * Temporary fix for the following issue:
   * @see https://github.com/metcoder95/cloud-pine/issues/34
   *
   * Google Cloud Platform doesn't accept more than 256KB for Logging Message
   * see: https://cloud.google.com/logging/quotas
   */
  private limitErrorMessageTo256KB(error: unknown) {
    if (!(error instanceof Error)) {
      return error;
    }

    const MAX_BYTES = 50_000; // Set the limit to 50KB

    const encoder = new TextEncoder();
    const messageBytes = encoder.encode(error.message);
    const stackBytes = encoder.encode(error.stack);

    if (messageBytes.length + stackBytes.length <= MAX_BYTES) {
      return error;
    } else {
      const decoder = new TextDecoder();
      const truncatedErrorMessage =
        decoder.decode(messageBytes.slice(0, MAX_BYTES)) + '...';

      error.message = truncatedErrorMessage;
      error.stack = undefined;
      return error;
    }
  }
}

Way to reproduce

import Pino from ('pino')

const logger = Pino({
   transport: {
      target: 'cloud-pine',
      options: {
         cloudLoggingOptions: {
            skipInit: true,
            sync: true,
         }
      }
   }
})

function generateLargeJSON(sizeInBytes) {
  const characters =
    'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789';
  const dataSize = sizeInBytes - 2; // Account for braces '{}' in JSON

  // Generate random string data for the given size
  let data = '';
  while (data.length < dataSize) {
    const randomIndex = Math.floor(Math.random() * characters.length);
    data += characters[randomIndex];
  }

  // Convert the data string into a JSON object
  const jsonObject = { data };

  return JSON.stringify(jsonObject);
}

// Generate a JSON object of size ~260 KB (260,000 bytes)
const jsonSizeInBytes = 260 * 1024; // 260 KB in bytes
const largeJSON = generateLargeJSON(jsonSizeInBytes);

logger.error(JSON.stringify(largeJSON))

[metcoder95]

Hey 👋
Thanks for the feedback.

I think that should be doable, so far we have two things here, we can start by extending the options also to allow configure split2 with the args maxLength and skipOverflow so the behaviour is customizable.

The second one is that the current implementation is quite basic, and does not implement backpressure, which can be implemented as further enhancement.

Let's start simple, and just allow configure split2 implementation. Would you like to submit a PR for it? 🙂