Docs: Version Mismatch Errors

[alexbaileyuk]

What were you searching in the docs?

There is a recommendation for esbuild and the CDK which excludes some packages when bundling function code.

new NodejsFunction(this, 'Function', {
  ...
  bundling: {
    externalModules: [
      '@aws-lambda-powertools/*',
      '@aws-sdk/*',
    ],
  }
});

This can cause version mismatches and cause unexpected behaviour in code if versions are not managed properly. Namely, when using instanceof as a condition, the result will always be false if the Lambda Layer version and package version do not match.

As an example, take this function definition deployed through the CDK:

import type { LambdaInterface } from '@aws-lambda-powertools/commons/types';
import { Logger } from '@aws-lambda-powertools/logger';
import { ParseError, parser } from '@aws-lambda-powertools/parser';
import { Tracer } from '@aws-lambda-powertools/tracer';
import { APIGatewayProxyEventSchema } from '@aws-lambda-powertools/parser/schemas';
import type { APIGatewayProxyResult, Context } from 'aws-lambda';
import { z, ZodError, ZodIssue } from 'zod';

const logger = new Logger();
const tracer = new Tracer();

export const requestSchema = APIGatewayProxyEventSchema.extend({
  queryStringParameters: z.object({
    message: z.string().max(10),
  }),
});

const handleZodError = (error: ZodError) => {
  const flattenedErrors = error.flatten((issue: ZodIssue) => ({
    field: issue.path.join('.'),
    issue: issue.message,
  }));

  return {
    statusCode: 422,
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({
      message: 'Validation error',
      data: { validationErrors: flattenedErrors.fieldErrors },
    }),
  };
};

export function httpErrorHandler(): MethodDecorator {
  return (_target, _propertyKey, descriptor: PropertyDescriptor) => {
    const originalMethod = descriptor.value;

    descriptor.value = async function (...args: unknown[]) {
      try {
        return await originalMethod.apply(this, args);
      } catch (error) {
        if (error instanceof Error) {
          logger.error('Error in API handler', error);
          tracer.addErrorAsMetadata(error);
        } else {
          logger.error('Unknown error in API handler', { error });
        }

        if (error instanceof ParseError && error.cause instanceof ZodError) {
          return handleZodError(error.cause);
        }

        if (error instanceof ZodError) {
          return handleZodError(error);
        }

        return {
          statusCode: 500,
          headers: { 'Content-Type': 'application/json' },
          body: JSON.stringify({ message: 'Internal server error' }),
        };
      }
    };

    return descriptor;
  };
}

class Lambda implements LambdaInterface {
  @httpErrorHandler()
  @tracer.captureLambdaHandler({ captureResponse: true })
  @logger.injectLambdaContext({ logEvent: true })
  @parser({ schema: requestSchema })
  public async handler(event: z.infer<typeof requestSchema>, _context: Context): Promise<APIGatewayProxyResult> {
    logger.info('Processing item', { item: event.queryStringParameters.message });

    return {
      statusCode: 200,
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({ message: 'Hello, world!' }),
    };
  }
}

const myFunction = new Lambda();
export const handler = myFunction.handler.bind(myFunction);

Taking the parser component at version 2.20.0, we can see that the Lambda Layer version is supposed to be 27:

$ aws ssm get-parameter --name /aws/service/powertools/typescript/generic/all/2.20.0 --region eu-west-1
{
    "Parameter": {
...
        "Value": "arn:aws:lambda:eu-west-1:094274105915:layer:AWSLambdaPowertoolsTypeScriptV2:27",
...
    }
}

Taking /test?message=testing126123gds as an example request, different behaviour is observed if different layer versions are used.

Parser Package Version	Layer Version	esbuild externalModules	Result
2.20.0	26	['@aws-sdk/*']	422 Validation Error (correct)
2.20.0	26	['@aws-lambda-powertools/', '@aws-sdk/']	500 Internal Server Error ("wrong")
2.20.0	27	['@aws-sdk/*']	422 Validation Error (correct)
2.20.0	27	['@aws-lambda-powertools/', '@aws-sdk/']	422 Validation Error (correct)

This can become especially difficult to manage when installing components at different times. For example, let's say you install Parser one day with yarn add @aws-lambda-powertools/parser and then at a later date, you install the @aws-lambda-powertools/tracer component. You're now in a position where not only the packages might be misaligned but also the Lambda Layers required.

Is this related to an existing documentation section?

https://docs.powertools.aws.dev/lambda/typescript/latest/getting-started/lambda-layers/#how-to-use-with-infrastructure-as-code

How can we improve?

I was originally going to report this as a bug, however, it's probably expected really. You can't expect to use different versions of packages and get the same result.

Whilst it's possible to write a dirty unit test in my project to make sure the package versions align, fetching the layer version for a particular version requires AWS credentials which I don't want to generate every time I have to run tests locally. Therefore, I'm just going to rely on API tests to pick this up and stick a big notice in the readme. Not ideal but will probably do okay. Maybe it's possible to do this if the parameter was a publicly accessible parameter?

Got a suggestion in mind?

I think the best course of action would be to simply update the documentation where it talks about excluding modules from esbuild output. I think the best result we can expect is a clear notice that all the versions have to be aligned and the Lambda Layer has to align to that version.

Acknowledgment

• I understand the final update might be different from my proposed suggestion, or refused.

[alexbaileyuk]

After a bit more thought, it's also possible to do this (or something similar):

import { ILayerVersion, LayerVersion } from 'aws-cdk-lib/aws-lambda';
import { StringParameter } from 'aws-cdk-lib/aws-ssm';
import { execSync } from 'child_process';
import { Construct } from 'constructs';
import { readFileSync } from 'fs';

const getPackageVersion = (packageName: string): string => {
  const output = execSync(`yarn info ${packageName} -A --json | jq -r ".children.Version"`, { encoding: 'utf-8' });
  return output.trim();
};

const ensurePowertoolsPackagesAreTheSameVersion = (): string => {
  const packageJson = readFileSync('package.json', 'utf8');
  const packageJsonObject = JSON.parse(packageJson);

  // find all packages beginning with @aws-lambda-powertools/
  const powertoolsPackages = Object.keys(packageJsonObject.dependencies).filter((dependency) =>
    dependency.startsWith('@aws-lambda-powertools/'),
  );

  // Error if no powertools packages are found
  if (powertoolsPackages.length === 0) {
    throw new Error('No powertools packages found in package.json');
  }

  // Ensure all packages are the same version
  const versions = powertoolsPackages.map((packageName) => getPackageVersion(packageName));
  if (versions.some((version) => version !== versions[0])) {
    throw new Error('All powertools packages must be the same version');
  }

  return versions[0]!;
};

export const powerToolsLayer = (scope: Construct): ILayerVersion => {
  const version = ensurePowertoolsPackagesAreTheSameVersion();

  const powerToolsLayerArn = StringParameter.valueForStringParameter(
    scope,
    `/aws/service/powertools/typescript/generic/all/${version}`,
  );

  return LayerVersion.fromLayerVersionArn(scope, 'PowerToolsLayer', powerToolsLayerArn);
};

[dreamorosi]

Hi @alexbaileyuk, thanks for the extensive report.

I need a bit more time to think about this, but overall I understand where you're coming from and acknowledge that this is a challenge when using Lambda layers.

The recommendation to exclude these packages is that by not doing that, it'd defeat the purpose of using the Layers. I think from our side the idea of offering these layers is so that you don't need to bundle your dependencies. If you don't exclude them, then why add the layers to the function in the first place?

Keeping Lambda layers and dev environment in sync is however a challenge. In theory there're no intentional changes between minor/patch versions, so it should be OK for versions to drift slightly, but it's true that in some cases we might fix bugs or cause unintended regressions - so from your standpoint I can see why you'd want to keep them in sync.

In the meantime, I'd like to also point out that all our errors are (or at least should be) "branded", meaning that you could avoid using instanceof but rather do if (error.name === 'ParseError)- this wouldn't require you to handle specific errors rather than inheritance chains of errors (i.e.ParseError->DecompressError`), but it could potentially mitigate the issue.

Along the same lines, besides adding notices in the docs, I'd be also open to considering implementing dedicated logic to help you with this instance comparison, similar to how other frameworks are doing.

[alexbaileyuk]

@dreamorosi I really like the idea of adding the instance comparison like some other frameworks. I've not seen that before and it would probably resolve this issue completely. Hopefully this is something that can be added without too much work?

[alexbaileyuk]

@dreamorosi I came back to this again today and noticed another issue. The ZodError doesn't match either :) Assume something funky is going on there as well. I'm going to go with the .name option for now and switch if something changes in the implementation here.

PS. Here is a nice function to debug these issues:

function diagnoseInstanceof(obj: unknown, expectedConstructor: Function) {
  if (typeof obj !== 'object' || obj === null) {
    console.log('❌ Not an object');
    return;
  }

  const objProto = Object.getPrototypeOf(obj);
  const expectedProto = expectedConstructor.prototype;

  console.log(`--- instanceof Diagnosis ---`);
  console.log(`Expected constructor name: ${expectedConstructor.name}`);
  console.log(`Object constructor name: ${obj.constructor?.name}`);
  console.log(`Object.toString(): ${Object.prototype.toString.call(obj)}`);

  const instanceofResult = obj instanceof (expectedConstructor as any);
  console.log(`obj instanceof expectedConstructor: ${instanceofResult}`);

  console.log(`Object.getPrototypeOf(obj) === expectedConstructor.prototype:`, objProto === expectedProto);

  console.log(`obj.constructor === expectedConstructor:`, (obj as any).constructor === expectedConstructor);

  console.log(`Traversing prototype chain of object:`);
  let proto = obj;
  while (proto) {
    console.log(' -', proto.constructor?.name || '[no constructor]');
    proto = Object.getPrototypeOf(proto);
  }

  console.log(`--- End Diagnosis ---`);
}

diagnoseInstanceof(error, ParseError);
diagnoseInstanceof((error as ParseError).cause, ZodError);

[dreamorosi]

Thank you for following up on this.

For now I'll be adding a disclaimer to the docs in the Layers section to warn others that this can be an issue, and perhaps suggest to do comparison based on names.

Regarding adding a custom method, I'll have to think about it a bit longer and understand if there's anything at all that we can do also for 3rd party libraries.

At least when we're throwing these errors - or adding them as cause - we can potentially brand them by adding a global symbol property, but I am unsure of the implications. Before doing that, I'd also want to look more closely at Zod's implementation and see if they already have any unique stable identifier that we could use.

[dreamorosi]

Sorry, these past ~10 days we were busy with the launch of another feature and this issue fell through the cracks.

I'll work on an update to the docs this week.