RFC: Per-record Logger keys and correlation id in Batch processing

[dreamorosi]

Is this related to an existing feature request or issue?

No. This came out of a design session for a closed source use case. The closest prior work is #5554, which introduced InvocationScoped in commons.

Which area does this RFC relate to?

Batch

Summary

Batch processes records in parallel by default. Logger's temporary keys live in invocation-wide state, so when a record handler calls logger.appendKeys() or logger.setCorrelationId(), every other record running at the same time sees those keys. The only fixes available today are to process records one at a time, or to thread a child logger through every function the handler calls.

We propose a recordScope option on the Batch processors. When it's on, each record gets its own copy of the Logger temporary keys, starting from the keys the invocation already has. The record handler and any module it imports keep using the ordinary module-level logger. A companion correlationId option sets a correlation id per record, from a JMESPath expression or a function. Both options are opt-in. Whether recordScope becomes the default in v3 depends on the feedback here.

Use case

Here's a handler that looks correct and isn't:

const logger = new Logger();
const processor = new BatchProcessor(EventType.SQS);

const recordHandler = async (record: SQSRecord) => {
  logger.appendKeys({ orderId: record.messageId });
  await chargeCustomer(record); // logs through the same `logger`
};

export const handler = async (event: SQSEvent, context: Context) =>
  processPartialResponse(event, recordHandler, processor, { context });

With ten records in the batch, the log lines from chargeCustomer carry whichever orderId was appended last. BasePartialProcessor.process() defaults processInParallel to true and runs Promise.all(records.map(processRecord)). Logger keeps temporary keys in InvocationScoped slots. Those are only separated per async context when AWS_LAMBDA_MAX_CONCURRENCY is set, and even then they're separated per invocation, not per record.

The downstream case that triggered this RFC is a platform that stamps a request_id on every DynamoDB item a request writes. The writes flow through a DynamoDB stream, an EventBridge Pipe, a bus and per-consumer SQS queues, to Lambda consumers. Each consumer wants every log line for a record to carry that record's request_id as the correlation id. EventBridge can't set SQS message attributes, so the id is inside the body, at detail.dynamodb.NewImage.request_id.S. Because parallel batches can't do this today, the team moved their consumers to sequential processing as a stopgap.

Sequential processing doesn't fully fix it either. Keys appended in record 1 are still there when record 2 starts, because nothing clears them between records.

Today's workarounds:

• processInParallel: false. This is slower, and keys still carry over from one record to the next.
• logger.createChild() per record, passed to every function the handler calls. It works, but it changes the signature of every function that logs.
• A custom formatter that reads from an AsyncLocalStorage the user manages, which means rebuilding what Powertools should provide.

Proposal

What users write

import { BatchProcessor, EventType, processPartialResponse } from '@aws-lambda-powertools/batch';
import { Logger } from '@aws-lambda-powertools/logger';
import { search } from '@aws-lambda-powertools/logger/correlationId';

const logger = new Logger({ correlationIdSearchFn: search });
const processor = new BatchProcessor(EventType.SQS, { logger });

const recordHandler = async (record: SQSRecord) => {
  logger.appendKeys({ orderId: record.messageId }); // only this record sees it
  await chargeCustomer(record); // any module that imports `logger` gets this record's keys
};

export const handler = async (event: SQSEvent, context: Context) =>
  processPartialResponse(event, recordHandler, processor, {
    context,
    recordScope: true, // name to be defined
    correlationId: 'powertools_json(body).detail.dynamodb.NewImage.request_id.S',
    // or: correlationId: (record) => extractRequestId(record), or a JMESPath expression, etc.
  });

The handler has no wrapper and keeps its signature. The two new options sit in the processing options, next to processInParallel.

Semantics

• When a record starts, it sees the temporary keys the invocation already has. For example, logger.appendKeys({ tenant }) called before processPartialResponse shows up in every record.
• Keys a record appends, removes or resets stay inside that record. After the batch, the logger has exactly the temporary keys it had before.
• This applies to parallel and sequential processing, to BatchProcessorSync, and to both FIFO processors.
• It applies to every Logger instance in the process, child loggers included, not only the one passed to the processor. A library that creates its own Logger gets record-level keys too.
• Persistent keys, the Lambda context, log level, log buffering and sampling stay invocation-wide. A persistent key appended inside a record is still there after the record, and the usual rule that the most recent write wins applies between temporary and persistent keys.
• Metrics, and Batch's own state such as success and failure lists, are unaffected. Metrics added inside records are flushed once, as today.
• logger.resetKeys() inside a record drops the keys that record added and goes back to the invocation's keys. A record can't wipe keys like tenant for itself by accident.
• Setting correlationId turns recordScope on. Without a scope, a per-record correlation id would leak into the next record.
  • A string is a JMESPath expression. Batch calls logger.setCorrelationId(record, path), so the Logger resolves it with its own correlationIdSearchFn, the same way injectLambdaContext({ correlationIdPath }) works.
  • A function receives the record, typed from the event, so an SQSEvent gives an SQSRecord. It returns the id, and undefined or an empty string sets nothing.
  • If the function throws, that record fails like any other record error. In FIFO processors this short-circuits the group as usual.
• Work started inside a record keeps that record's keys, including promises the handler doesn't await. That's standard AsyncLocalStorage behaviour.
• A batch processed inside another batch's record handler starts from the outer record's keys.
• With AWS_LAMBDA_MAX_CONCURRENCY set, record scopes sit inside each invocation's InvokeStore context. Request id, X-Ray trace id and tenant id read the same inside a record as outside it.
• injectLambdaContext({ resetKeys: true }) works unchanged. It runs after the handler, outside any record, and clears invocation keys as it does today.

How it works

Commons. We add a record scope next to InvocationScoped:

• an AsyncLocalStorage<Map<symbol, unknown>>;
• runInRecordScope(fn), which works for sync and async functions;
• an optional fork: (parent: T) => T on InvocationScopedOptions.

Only slots that declare fork look at the record scope. Inside a scope, the first access to such a slot stores a fork of the parent value, and later reads and writes in that record use the copy. The parent value comes from the nearest enclosing record scope that has one, and otherwise from the invocation, resolved exactly as today. reset() inside a scope discards the copy, so the next access forks again. Slots without fork never read the record scope, so Batch's stores, Metrics' stores and the Logger slots we don't fork behave exactly as they do now.

The AsyncLocalStorage lives on globalThis under Symbol.for('powertools.commons.recordScope.v1'), and the first runInRecordScope call creates it. Functions that never open a scope never read it. When a bundle ends up with two copies of commons, they share one scope instead of quietly ignoring each other. The v1 suffix lets us change the shape of the stored map later without breaking older copies.

runInRecordScope and fork are @internal. We export them from the existing @aws-lambda-powertools/commons/utils/invocation-scoped subpath, which is how Logger and Metrics already import InvocationScoped.

Logger. Today LogAttributesStore keeps a key map recording, for each key, whether the temporary or the persistent value wins. We replace it with write stamps. Every write gets an increasing number, and when a key is both temporary and persistent, the most recent write wins. Temporary attributes and their stamps live in one slot that declares fork. Persistent attributes and their stamps stay invocation-wide, as persistent attributes are today. That keeps persistent writes made inside a record visible after it, without any API to reach past the record scope. It also makes the store smaller. The prototype's Logger change is +51 / -63 lines. The fork deep-copies the attributes, because appendTemporaryKeys deep-merges into nested objects in place and a shallow copy would bleed back into the invocation. The public API doesn't change.

Batch.

• BatchProcessor.processRecord and BatchProcessorSync.processRecordSync run the record inside runInRecordScope when recordScope or correlationId is set. Both FIFO processors extend these classes and call processRecord from their own loops, so every loop is covered without touching the loops.
• The correlation id is applied inside the scope and inside each processor's existing try, so a throwing function goes through the normal failure handler. AsyncHandlerNotSupportedError from the sync processor is still thrown.
• processPartialResponse and processPartialResponseSync infer the record type from event.Records, so the correlationId function's record parameter is typed. BatchProcessingOptions<T, R = any> keeps existing call sites compiling. Loosely typed events give the BaseRecord union, which the callback has to narrow.
• The logger option's type widens, without breaking anyone, to Pick<GenericLogger, 'debug' | 'warn' | 'error'> & { setCorrelationId?: (value: unknown, path?: string) => void }.
  • If correlationId is set and the logger has no setCorrelationId, Batch warns once and carries on.
  • Without a logger, records are still scoped. Only the correlation id is skipped.
• SqsFifoPartialProcessor and SqsFifoPartialProcessorAsync gain an optional constructor config with logger. Today they take no arguments.

Why opt-in first. Turning this on by default changes behaviour some users may depend on, perhaps without knowing it. Today a key appended in record N is visible in record N+1 when processing sequentially, and keys appended inside records are visible after the batch. We'd consider flipping the default in v3, depending on how people use the option in v2.

Investigation

We started from four independent design write-ups, then built two working prototypes. Both prototypes ran the same 13 acceptance tests and the same benchmark.

The tests cover:

• parallel isolation with interleaved awaits, through a separately imported module;
• invocation keys visible inside records;
• nothing left after the batch;
• sequential, sync and FIFO processing;
• partial-failure reporting;
• a single metrics flush containing every data point;
• the correlation id as a function, as a path, and empty;
• two concurrent invocations under AWS_LAMBDA_MAX_CONCURRENCY, each with two records, keeping their request ids;
• a second, independent Logger instance, and a child logger.

On main, 11 of the 13 fail. The two that pass are the partial-failure and metrics checks, which guard against regressions.

	Proposal (commons fork)	Alternative B (Logger-owned scope)
Acceptance tests	13/13	13/13
Existing and new unit tests	807/807	807/807
Source lines	+212 / -22	+321 / -22
Needs a logger passed to the processor	No	Yes

The benchmark used Node 24 on a shared machine. We ran three rounds one after another; the table shows the best run in ms.

	main	Proposal	Alternative B
10k-record batch, recordScope off	310	327	324
10k-record batch, recordScope on	not applicable	348	343
10k-record batch, on, with correlationId	not applicable	371	367
200k logger.info outside any batch	2494	2548	2560
200k appendKeys outside any batch	263	278	280
200k metrics.addMetric	399	411	401

In the batch benchmark, each record appends a key, logs twice and awaits once. The cost of the scope is about 7 to 10% of batch time. Outside a batch, Logger calls cost about 2 to 6% more. Metrics didn't change beyond noise.

We then resolved the open points from the first prototype in the proposal's prototype: write stamps, nested inheritance, typed correlationId, error handling, resetKeys(). It now passes 823 tests, including the 13 acceptance tests and new tests for each point. The source diff is +298 / -81 lines, split as commons +139 / -3, logger +51 / -63 and batch +108 / -15. We ran the benchmark again, taking the median of three runs. The machine wasn't idle, so treat ±5% as noise.

	Before follow-up	After follow-up
200k logger.info outside any batch	2650	2676
200k appendKeys outside any batch	287	194
10k-record batch, recordScope on	347	377
10k-record batch, on, with correlationId	380	400

appendKeys got faster because each write now reads one slot instead of two.

We also tried to avoid reading the AsyncLocalStorage when no scope is open, using a process-wide counter of open scopes. It saved about 15 ns per access, about 0.3% of a logger.info call. It also broke work that outlives its record: once the counter dropped to zero, a fire-and-forget promise started inside a record went back to reading invocation-level keys. We dropped the counter, and kept the test that caught it.

Out of scope

• Per-record Metrics dimensions. Dimensions stay invocation-wide. A Metrics slot could opt in with fork later, but flushing would need its own design.
• Per-record persistent keys. appendPersistentKeys inside a record still changes the logger for the whole invocation.
• A public scope API. We're not exposing runInRecordScope to users or to other utilities, such as Event Handler or Kafka, in this RFC. If demand shows up, it can go public later without changing the semantics.
• Per-record Tracer subsegments.
• Passing the parsed record to the correlationId function. The function receives the raw record, before any parsing.
• Reading the scope only while one is open. We measured it and dropped it. See the investigation.

Potential challenges

• The opt-in is per slot, not per caller. Any slot that declares fork is separated in every record scope, process-wide. That's what lets a second Logger instance work. It also means a future fork on some other slot changes behaviour for every Batch user who turned the scope on. We'll document the rule in InvocationScoped, and treat adding fork to a slot as a behaviour change.
• resetKeys() inside a record goes back to the invocation's keys rather than clearing everything. That's a small difference from what resetKeys() means outside a batch, and the docs need to say so.
• Work that outlives its record, such as a fire-and-forget promise, keeps writing to that record's copy. Anything it appends after the record ends is silently dropped. While that work is pending, it also keeps the record's scope, and any outer scopes, in memory. We'll document both.
• Loosely typed events. If event isn't typed, the correlationId function gets the BaseRecord union and has to narrow it. Annotating the parameter as SQSRecord is rejected, and the error points at the event argument rather than the callback. We tried NoInfer to move the error to the callback, but it broke any-typed events that annotate the record.
• Persistent keys and the InvokeStore. With write stamps, persistent keys no longer touch an InvocationScoped slot. That changes one edge case, where AWS_LAMBDA_MAX_CONCURRENCY is set and no InvokeStore is installed. There, appendPersistentKeys used to throw straight away, and now the error appears at the first log call instead. Normal imports can't produce that setup, because commons imports @aws/lambda-invoke-store, which installs the store. We don't expect anyone to notice.

Dependencies and Integrations

• No new dependencies. AsyncLocalStorage comes from node:async_hooks, which every supported Lambda Node.js runtime provides.
• Changes are in @aws-lambda-powertools/commons, logger and batch. Batch already depends on commons and doesn't depend on Logger; that stays true. Batch only calls setCorrelationId through the widened logger type.
• Logger: correlationId reuses the correlationIdSearchFn and @aws-lambda-powertools/logger/correlationId search that users already configure for injectLambdaContext.
• Multi-concurrency: when AWS_LAMBDA_MAX_CONCURRENCY is set, record scopes use their own AsyncLocalStorage and never call InvokeStore.run, so the InvokeStore context and its protected keys are untouched.
• Metrics, Tracer, Idempotency: no changes. Metrics was covered by the acceptance tests. Tracer and Idempotency don't use InvocationScoped.
• Docs to update:
  • Batch: a new "Per-record logging context" section.
  • Logger, "Working with keys" and "correlation id": what is per record under Batch, and the caveat about work that outlives a record.

Alternative solutions

B. A Logger-owned scope, reached by Batch through a symbol on the logger. Logger keeps its own module-level AsyncLocalStorage. Batch calls a symbol-keyed method on the logger passed to the processor. Commons only provides the symbol. We prototyped this and it passes the same tests, with the same performance. It leaves InvocationScoped untouched, which is its main advantage. We prefer the proposal for four reasons:

• it's about a third less code;
• there's no hidden contract between Batch and Logger to version;
• scoping works without passing a logger to the processor;
• duplicate copies of commons share one scope, while duplicate copies of Logger in B don't.

Treat an open scope as scoped for every InvocationScoped slot. This was the first idea. It changes the shared lookup in commons so that any open scope separates every slot. It would make Batch's errors and failureMessages, which are written from inside the record, and Metrics' stored metrics look empty or get lost. Opting in per slot avoids that.

A wrapper such as logger.withScope(() => ...). It's easy to build, but every handler has to remember it. Code reached through other modules is only covered if the user wraps at the right level. We'd rather users configure this once on the processor.

logger.createChild() per record. This works today, and it's what we'd recommend while this RFC is open. It forces every function that logs to accept a logger argument.

Reuse InvokeStore.run from @aws/lambda-invoke-store. run(context, fn) replaces the whole invocation context, protected keys included. Outside multi-concurrency mode it also doesn't isolate anything. It doesn't fit nesting inside an invocation.

Turn it on for everyone now. It's nicer DX, but it changes the behaviour described in "Why opt-in first" without a major version.

Acknowledgment

• This RFC meets Powertools for AWS Lambda (TypeScript) Tenets
• Should this be considered in other Powertools for AWS Lambda languages? i.e. Python, Java, and .NET

---

Disclaimer: After creating an issue, please wait until it is triaged and confirmed by a maintainer before implementing it. This will reduce amount of rework and the chance that a pull request gets rejected.

Metadata information for admin purposes, please leave them empty.

• RFC PR:
• Approved by: ''
• Reviewed by: ''

Future readers

Please react with 👍 and your use case to help us understand customer demand.