Transport is not being waited for in lambda

[YeeP79]

I apologize if this is the wrong place to post something like this, but I am currently at a point where I feel like my only option is to change logging libraries. I really like pino and want to make this work.

I have a transport that I am working on, and even when running the code outside the lambda, you can see that the sending of logs through an async process is completed after the main thread is complete. I would consider this "expected" when talking about a worker thread as one does not really need to wait for the other. The problem arises when running inside a lambda, where apparently AWS considered the event loop empty, and terminates the function.

My transport does not buffer logs currently, it sends them immediately to their final location. As far as I know, when using a transport, the {destination:sync} that is shown in the docs is not applicable, as I am determining the syncing mechanism in the transport. Also I do have the lambda context set to wait for an empty event loop.

Because I do not have access to the instantiation of the worker thread, I cannot leverage parentPort.postMessage to let the main thread know when the worker is done.

This has to be an issue that someone has already dealt with and I am missing something. Can anyone give any tips on how to handle this scenario?

Just as a reference because Im sure someone might ask, here is the instantiation of the transport:

// main TS file

this._pinoTransport = pino.transport({
        options: { datadogIntegrationOptions },
        target: transportTarget,
      });
      this._pinoInstance = pino(this._loggerOptions, this._pinoTransport);

And the transport itself:

// transport:

export default async function (opts) {
  const { configuration, context } = opts.datadogIntegrationOptions;
  const apiConfiguration = client.createConfiguration(configuration);
  const apiInstance = new v2.LogsApi(apiConfiguration);
  console.log('Transport initialized');

  return build(
    async (source) => {
      for await (const log of source) {
        const { bindings, rootRequirements } = getParsedLog(log);
        const logBaseOptions = getLogBaseOptions(rootRequirements);
        const ddLog = getDDLogOptions(
          logBaseOptions,
          rootRequirements,
          context,
          bindings,
        );

        try {
          const logItems = { message: JSON.stringify(ddLog) };
          const logObject = { body: [logItems], ddtags: rootRequirements.tags };
          await apiInstance.submitLog(logObject);
        } catch (error) {
          console.log('logging error: ', error);
          emitter.emit('NWServerLogger:serverTransport:error', error);
        }
      }
    },
    {
      close(err, cb) {
        console.log('closing transport');
        cb(err);
      },
    },
  );
}

Some things have been added here to help diagnose issues with it, and it is not complete. I really am just looking for someone to help me understand how to handle this scenario. I do not want to have to change logging libraries, but I am getting no logs from my quick running lambdas as they are terminated too early.

[mcollina]

You should be calling transport.flushSync() before terminating your lambda. Unfortunately this is a bad side effect of lambda.

Note that you are likely gaining nothing from using transports in lambda, so you might just use streams (but you'll still need a flush mechanism).

[YeeP79]

I was under the impression that if I needed to communicate with an external source (datadog) I needed to use a transport that could be run in a worker thread.

So you are referring to transport.flushSync(), not pino.flush(). I will start researching that. Eventually I will have some sort of buffering system in here, not sure if I can do that in a stream as well. Definitely interested in any input you have.

Edit: I see flush inside transport, but not flushSync.

pino/lib/transport.js

Line 69 in 06df1df

function flush (stream) {

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.