@aws-cdk/integ-tests-alpha: Integ-test's singleton Function still does not have the proper permission to invoke Lambda Function

[oqian]

Describe the bug

An earlier issue #27865 was reported, and although it was reported that the issue is addressed, it is actually not. The wait provider is now granted the correct permission to invoke a lambda function, but not for the actual singleton function that is in charge of invoking the function-to-be-tested.
The following error is still reported:

AccessDeniedException: User: arn:aws:sts::****:assumed-role/IntegTestdevDefaultTestDe-SingletonFunction76b3e830-4CXwsoDDSbg8/IntegTestdevDefaultTestDe-SingletonFunction76b3e83-PoYjI4o55pQY is not authorized to perform: lambda:InvokeFunction on resource: arn:aws:lambda:us-west-2:*****:function: _function_name_ because no identity-based policy allows the lambda:InvokeFunction action

Expected Behavior

The correct policy should be propagated to all related singleton Functions.

Current Behavior

Integration test stack cannot be deployed due to the permission issue.

Reproduction Steps

lambda_invocation = integ.assertions.invoke_function(    
    function_name=stack_to_be_tested.lambda_function.function_name,
    invocation_type=InvocationType.EVENT,
    payload=json.dumps({
        "days":1
    })
).expect(
    ExpectedResult.object_like(
        {
            "execution_arn": Match.string_like_regexp("arn:aws:states:us-.*")
        }
    )
).wait_for_assertions(
    interval=Duration.seconds(10),
    total_timeout=Duration.minutes(90)
)

Here integ is an aws_cdk.integ_tests_alpha.IntegTest object.

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.118.0

Framework Version

No response

Node.js Version

18.17.1

OS

macOS

Language

Python

Language Version

3.9

Other information

No response

[pahud]

related to #28424

Thanks for the report and we probably need some inputs and verification from @sakurai-ryo .

[sakurai-ryo]

Thanks for reaching out @oqian, and sorry for the confusion.

I ran the same test in TypeScript and could not reproduce the access denied error.
The test code is below.

import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';
import * as lambda from 'aws-cdk-lib/aws-lambda';
import { IntegTest, InvocationType, ExpectedResult } from '@aws-cdk/integ-tests-alpha';

export class Lambda extends cdk.Stack {
  public readonly lambda: lambda.Function;

  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    this.lambda = new lambda.Function(this, 'MyFunction', {
      handler: 'index.handler',
      runtime: lambda.Runtime.NODEJS_18_X,
      code: lambda.Code.fromInline('exports.handler = async (event) => { console.log(event); return { body: "Hello World" }; };'),
    });

    this.lambda.applyRemovalPolicy(cdk.RemovalPolicy.DESTROY);
  }
}

const app = new cdk.App();

const stack = new Lambda(app, 'LambdaStack');

const integ = new IntegTest(app, 'LambdaStackTest', {
  testCases: [stack],
});

integ.assertions.invokeFunction({
  functionName: stack.lambda.functionName,
  invocationType: InvocationType.EVENT,
  payload: JSON.stringify({"days":1}),
}).expect(
  ExpectedResult.objectLike({ StatusCode: 202 }),
).waitForAssertions({
    interval: cdk.Duration.seconds(10),
    totalTimeout: cdk.Duration.minutes(90)
});

$ integ-runner --directory test --update-on-failed --parallel-regions ap-northeast-1

Verifying integration test snapshots...

  NEW        integ.lambda 2.232s

Snapshot Results:

Tests:    1 failed, 1 total
Failed: /Users/hoge/Desktop/lambdaStack/test/integ.lambda.ts

Running integration tests for failed tests...

Running in parallel across regions: ap-northeast-1
Running test /Users/hoge/Desktop/lambdaStack/test/integ.lambda.ts in ap-northeast-1
  SUCCESS    integ.lambda-LambdaStackTest/DefaultTest 129.575s
       AssertionResultsLambdaInvoke2fe9b1bd8e86bbbf668f5daf023e0ef2 - success

Test Results:

Tests:    1 passed, 1 total

Could you tell us the entire test code?
Also, Please ensure you have the latest aws-cdk.integ-tests-alpha (2.118.0a0) module installed.
https://pypi.org/project/aws-cdk.integ-tests-alpha

[oqian]

Hi @sakurai-ryo , thank you for getting back to me. When I was testing the code, I only updated the AWS CDK CLI, and did not update the dependency on the code side. However, after I updated the aws-cdk and the aws-cdk.integ-tests-alpha to the latest version 2.119.0. I still could not get the test through, as I am receiving the following errors:

(.venv) (base) ➜  prism-cdk-templates git:(yqian/codepipeline_test) ✗ pip show aws-cdk-lib
Name: aws-cdk-lib
Version: 2.119.0
Summary: Version 2 of the AWS Cloud Development Kit library
Home-page: https://github.com/aws/aws-cdk
Author: Amazon Web Services
Author-email: 
License: Apache-2.0
Location: /Users/yqian/Desktop/art-ai-lab-louvre/prism-cdk-templates/.venv/lib/python3.9/site-packages
Requires: aws-cdk.asset-awscli-v1, aws-cdk.asset-kubectl-v20, aws-cdk.asset-node-proxy-agent-v6, constructs, jsii, publication, typeguard
Required-by: aws-cdk.integ-tests-alpha
(.venv) (base) ➜  prism-cdk-templates git:(yqian/codepipeline_test) ✗ pip show aws-cdk.integ-tests-alpha
Name: aws-cdk.integ-tests-alpha
Version: 2.119.0a0
Summary: CDK Integration Testing Constructs
Home-page: https://github.com/aws/aws-cdk
Author: Amazon Web Services
Author-email: 
License: Apache-2.0
Location: /Users/yqian/Desktop/art-ai-lab-louvre/prism-cdk-templates/.venv/lib/python3.9/site-packages
Requires: aws-cdk-lib, constructs, jsii, publication, typeguard
Required-by: 
(.venv) (base) ➜  cdk_playground git:(main) ✗ cdk --version
2.119.0 (build 0392e71)
(.venv) (base) ➜  cdk_playground git:(main) ✗ npm run integ-test                

> integ-test
> integ-runner --directory ./tests --language python --verbose --update-on-failed --parallel-regions us-west-2 --profiles Louvre


Verifying integration test snapshots...

  ERROR      integ_python-test 6.845s
      "cdk-integ" can only operate on apps with a single stack.

  If your app has multiple stacks, specify which stack to select by adding this to your test source:

      /// !cdk-integ STACK ...

  Available stacks: cdk-playground-lambda integ-test/DefaultTest/DeployAssert (wildcards are also supported)


Snapshot Results: 

Tests:    1 failed, 1 total
Failed: /Users/yqian/Desktop/cdk_playground/tests/integ_python-test.py

Running integration tests for failed tests...

Running in parallel across profiles Louvre and regions: us-west-2
Running test /Users/yqian/Desktop/cdk_playground/tests/integ_python-test.py in Louvre/us-west-2
  ERROR      /Users/yqian/Desktop/cdk_playground/tests/integ_python-test.py (Louvre/us-west-2) 6.485s
      Error during integration test: Error: "cdk-integ" can only operate on apps with a single stack.

  If your app has multiple stacks, specify which stack to select by adding this to your test source:

      /// !cdk-integ STACK ...

  Available stacks: cdk-playground-lambda integ-test/DefaultTest/DeployAssert (wildcards are also supported)


Test Results: 

Tests:    1 failed, 1 total
   --- Integration test metrics ---
Profile Louvre + Region us-west-2 total time: 6.486
  /Users/yqian/Desktop/cdk_playground/tests/integ_python-test.py: 6.486
Error: Some integration tests failed!
    at main (/Users/yqian/Desktop/cdk_playground/node_modules/@aws-cdk/integ-runner/lib/index.js:10397:15)

The integ test is implemented as:

test_app = cdk.App()

test_stack = CdkPlaygroundStack(test_app, "cdk-playground-lambda")

integ_test = IntegTest(
    test_app,
    "integ-test",
    test_cases = [test_stack]
)

integ_test.assertions.invoke_function(
    function_name = test_stack.lambda_function.function_name,
    invocation_type=InvocationType.EVENT,
    payload=json.dumps({
        "days":1
    })
).expect(
    ExpectedResult.object_like(
        {
            "execution_arn": Match.string_like_regexp("arn:aws:states:us-.*")
        }
    )
).wait_for_assertions(
    interval=Duration.seconds(10),
    total_timeout=Duration.minutes(10)
)

test_app.synth()

And the Lambda Stack class is:

class CdkPlaygroundStack(Stack):

    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        self.lambda_function = _lambda.Function(
            self, "HelloHandler",
            runtime=_lambda.Runtime.NODEJS_18_X,
            code=_lambda.Code.from_inline('exports.handler = async (event) => { console.log(event); return { body: "Hello World" }; };'),
            handler="index.handler",
        )

        self.lambda_function.apply_removal_policy(policy=RemovalPolicy.DESTROY)

I then downgraded my package to 2.90, and no such error was indicated, would you please help me take a look at it? It most likely is a new issue. Thanks in advance.

[sakurai-ryo]

@oqian
Thanks for the reply.

I ran the test with your Python code but could not reproduce the error.
The error is probably caused by the missing integ.json file under the snapshot directory.
Could you delete the snapshot directory, update the integ-runner cli to the latest version (2.119.0-alpha.0), and rerun it?
https://www.npmjs.com/package/@aws-cdk/integ-runner

[oqian]

Hi @sakurai-ryo , I deleted the snapshot library and can confirm that I have both the CLI and cdk dependency in the latest version, and I kept getting the same error. The content of integ.json:

{
  "version": "36.0.0",
  "testCases": {
    "integ-test/DefaultTest": {
      "stacks": [
        "cdk-playground-lambda"
      ],
      "assertionStack": "integ-test/DefaultTest/DeployAssert",
      "assertionStackName": "integtestDefaultTestDeployAssert24D5C536"
    }
  }
}

For your running my provided code, would you please confirm that you have dependencies and node modules that are of same version as mine?
I have tried to run the integ-test on both of my projects and they are reporting the same issue. This is the vanilla project which contains very minimal work, so my guess is that the problem is very likely on the dependencies' side. Let me know if there is anything that you need further on my side, thank you!

[sakurai-ryo]

Thanks @oqian.

Sorry to bother you, but if possible, could you push reproducible code to your GitHub account?
If this is not possible, I would appreciate it if you could tell me the contents of the requirements.txt and package.json.

[oqian]

Thank you for sticking with me. @sakurai-ryo https://github.com/oqian/cdk_playground This repo should contain everything that you've asked for. Hope that it will help you reproduce the issue.

[sakurai-ryo]

@oqian
I appreciate it, thank you.
After cloning the repository, I performed the following steps to run the test.

$ integ-runner --version
2.121.0-alpha.0

$ cdk --version
2.121.0 (build 9f2b78c)

$ python3 --version
Python 3.9.6

$ node -v
v18.17.0

$ python3 -m venv .venv
$ source .venv/bin/activate
$ pip install -r requirements.txt
$ npm run integ-test

The Lambda function did not return execution_arn so the Assertion failed, but the test could run.

I still don't know the cause of the problem so that I will investigate further.

[oqian]

@sakurai-ryo Thank you again for trying to reproduce the error. In this case, if you have time, we can definitely schedule a meeting offline to share my screen and go over the issue together. My email address is yqian@adobe[dot]com.

[sakurai-ryo]

@oqian
Yes, that is definitely one way to go, but I am sorry I may not be able to take the time as I am not on the core team and busy with other work.

If you run the exact steps that I ran the test before, does it still give an error in your environment?

Also, although it should not be necessary, I would like you to try adding /// !cdk-integ * to the beginning of the test file as a workaround to see what happens.
Here is the sample file below.

aws-cdk/packages/@aws-cdk-testing/framework-integ/test/aws-apigateway/test/integ.usage-plan.sharing.ts

Line 1 in e25c5b6

/// !cdk-integ *

[oqian]

Hi @sakurai-ryo, after updating cdk from 2.119 to 2.221, the issue seems to have been resolved, as I am seeing the stack is being created and deployed, and the lambda functions are being triggered by the singleton function. So I think this issue can be closed. Thank you so much for sticking with me along the way and providing very helpful leads and suggestions.
@pahud Please close the issue, it's addressed right now as of version 2.221.

[sakurai-ryo]

@oqian
Awesome!
I'm happy to hear that.

[sakurai-ryo]

@oqian
You can also close this issue, so closing it is no problem!

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.