[sam-beta-cdk] sam build ignores --profile

[rupe120]

Description:

When using sam-beta-cdk build --profile the manifest is not updated with the account and region of the profile

Steps to reproduce:

1. sam-beta-cdk init --project-type cdk
2. sam-beta-cdk build --profile <profile-with-region-config>

Observed result:

The manifest.json reflects an unknown account and region. The environment property for the resource is aws://unknown-account/unknown-region

2021-10-30 09:45:07,619 | IAC Plugin getting project...
2021-10-30 09:45:07,629 | CDK Toolkit found at C:\Program Files\nodejs\cdk.CMD
2021-10-30 09:45:07,630 | Synthesizing CDK App
2021-10-30 09:45:07,630 | command: ['C:\\Program Files\\nodejs\\cdk.CMD', 'synth', '--no-staging', '-o', '.aws-sam/.cdk-out']
2021-10-30 09:45:23,978 | Cloud assembly synthed at .aws-sam/.cdk-out
2021-10-30 09:45:23,978 | _validate_cloud_assembly: .aws-sam/.cdk-out
2021-10-30 09:45:23,982 | Found assets: {'C:\\Temp\\Innovative\\cdk\\sam-app\\hello_world': <samcli.lib.iac.interface.S3Asset object at 0x000002A21ED4A2B0>}
2021-10-30 09:45:23,983 | ca-stack_name: sam-app
2021-10-30 09:45:23,983 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2021-10-30 09:45:23,985 | Using config file: samconfig.toml, config environment: default
2021-10-30 09:45:23,985 | Expand command line arguments to:
2021-10-30 09:45:23,986 | --project_type=CDK --template_file=C:\Temp\Innovative\cdk\sam-app\template.yml --use_container --build_dir=.aws-sam\build --cache_dir=.aws-sam\cache --iac=<samcli.lib.iac.cdk.plugin.CdkPlugin object at 0x000002A21E1B9D90> --project=<samcli.lib.iac.interface.Project object at 0x000002A21ED6DE20>
2021-10-30 09:45:24,243 | 'build' command is called
2021-10-30 09:45:24,244 | Starting Build inside a container
2021-10-30 09:45:24,245 | Collected default values for parameters: {}
2021-10-30 09:45:24,320 | 21 stacks found in the template
2021-10-30 09:45:24,320 | Collected default values for parameters: {}
2021-10-30 09:45:24,374 | 21 resources found in the stack sam-app
2021-10-30 09:45:24,375 | Collected default values for parameters: {}
2021-10-30 09:45:24,424 | Found Lambda function with name='cdkhelloworldlambda1060AF2E' and CodeUri='C:\Temp\Innovative\cdk\sam-app\hello_world'2021-10-30 09:45:24,424 | --base-dir is not presented, adjusting uri C:\Temp\Innovative\cdk\sam-app\hello_world relative to .aws-sam/.cdk-out
2021-10-30 09:45:24,425 | Collected default values for parameters: {}
2021-10-30 09:45:24,612 | Instantiating build definitions
2021-10-30 09:45:24,620 | Same function build definition found, adding function (Previous: BuildDefinition(python3.8, C:\Temp\Innovative\cdk\sam-app\hello_world, Zip, , b2fd66ef-2732-4113-8c6a-438c2427b924, {'aws:cdk:path': 'sam-app/cdk-hello-world-lambda/Resource', 'aws:asset:path': 'C:\\Temp\\Innovative\\cdk\\sam-app\\hello_world', 'aws:asset:property': 'Code'}, {}, []), Current: BuildDefinition(python3.8, C:\Temp\Innovative\cdk\sam-app\hello_world, Zip, , 0e73e948-f7ba-4a15-90a7-2177e0b88335, OrderedDict([('aws:cdk:path', 'sam-app/cdk-hello-world-lambda/Resource'), ('aws:asset:path', 'C:\\Temp\\Innovative\\cdk\\sam-app\\hello_world'), ('aws:asset:property', 'Code')]), {}, []), Function: Function(function_id='cdk-hello-world-lambda', name='cdkhelloworldlambda1060AF2E', functionname='cdk-hello-world-lambda-function', runtime='python3.8', memory=None, 
timeout=None, handler='app.lambda_handler', imageuri=None, packagetype='Zip', imageconfig=None, codeuri='C:\\Temp\\Innovative\\cdk\\sam-app\\hello_world', environment=None, rolearn='arn:aws:lambda:us-east-1:123456789012:function:cdkhelloworldlambdaServiceRole495FD13E', layers=[], events=None, metadata=OrderedDict([('aws:cdk:path', 'sam-app/cdk-hello-world-lambda/Resource'), ('aws:asset:path', 'C:\\Temp\\Innovative\\cdk\\sam-app\\hello_world'), ('aws:asset:property', 'Code')]), inlinecode=None, codesign_config_arn=None, stack_path='sam-app'))
2021-10-30 09:45:24,626 | Building codeuri: C:\Temp\Innovative\cdk\sam-app\hello_world runtime: python3.8 metadata: {'aws:cdk:path': 'sam-app/cdk-hello-world-lambda/Resource', 'aws:asset:path': 'C:\\Temp\\Innovative\\cdk\\sam-app\\hello_world', 'aws:asset:property': 'Code'} functions: ['sam-app/cdk-hello-world-lambda']
2021-10-30 09:45:24,627 | Building to following folder C:\Temp\Innovative\cdk\sam-app\.aws-sam\build\sam-app\cdkhelloworldlambda1060AF2E        

Fetching public.ecr.aws/sam/build-python3.8:latest Docker container image......
2021-10-30 09:45:26,874 | Mounting C:\Temp\Innovative\cdk\sam-app\hello_world as /tmp/samcli/source:ro,delegated inside runtime container       
Using the request object from command line argument
Loading workflow module 'aws_lambda_builders.workflows'
Registering workflow 'PythonPipBuilder' with capability 'Capability(language='python', dependency_manager='pip', application_framework=None)'   
Registering workflow 'NodejsNpmBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)'   
Registering workflow 'RubyBundlerBuilder' with capability 'Capability(language='ruby', dependency_manager='bundler', application_framework=None)'
Registering workflow 'GoDepBuilder' with capability 'Capability(language='go', dependency_manager='dep', application_framework=None)'
Registering workflow 'GoModulesBuilder' with capability 'Capability(language='go', dependency_manager='modules', application_framework=None)'   
Registering workflow 'JavaGradleWorkflow' with capability 'Capability(language='java', dependency_manager='gradle', application_framework=None)'Registering workflow 'JavaMavenWorkflow' with capability 'Capability(language='java', dependency_manager='maven', application_framework=None)'  
Registering workflow 'DotnetCliPackageBuilder' with capability 'Capability(language='dotnet', dependency_manager='cli-package', application_framework=None)'
Registering workflow 'CustomMakeBuilder' with capability 'Capability(language='provided', dependency_manager=None, application_framework=None)' 
Found workflow 'PythonPipBuilder' to support capabilities 'Capability(language='python', dependency_manager='pip', application_framework=None)' 
Running workflow 'PythonPipBuilder'
Running PythonPipBuilder:ResolveDependencies
calling pip download -r /tmp/samcli/source/requirements.txt --dest /tmp/samcli/scratch --exists-action i
Full dependency closure: {requests==2.26.0(wheel), certifi==2021.10.8(wheel), urllib3==1.26.7(wheel), charset-normalizer==2.0.7(wheel), idna==3.3(wheel)}
initial compatible: {requests==2.26.0(wheel), certifi==2021.10.8(wheel), urllib3==1.26.7(wheel), charset-normalizer==2.0.7(wheel), idna==3.3(wheel)}
initial incompatible: set()
Downloading missing wheels: set()
compatible wheels after second download pass: {requests==2.26.0(wheel), certifi==2021.10.8(wheel), urllib3==1.26.7(wheel), charset-normalizer==2.0.7(wheel), idna==3.3(wheel)}
Build missing wheels from sdists (C compiling True): set()
compatible after building wheels (no C compiling): {requests==2.26.0(wheel), certifi==2021.10.8(wheel), urllib3==1.26.7(wheel), charset-normalizer==2.0.7(wheel), idna==3.3(wheel)}
Build missing wheels from sdists (C compiling False): set()
compatible after building wheels (C compiling): {requests==2.26.0(wheel), certifi==2021.10.8(wheel), urllib3==1.26.7(wheel), charset-normalizer==2.0.7(wheel), idna==3.3(wheel)}
Final compatible: {requests==2.26.0(wheel), charset-normalizer==2.0.7(wheel), certifi==2021.10.8(wheel), urllib3==1.26.7(wheel), idna==3.3(wheel)}
Final incompatible: set()
Final missing wheels: set()
PythonPipBuilder:ResolveDependencies succeeded
Running PythonPipBuilder:CopySource
PythonPipBuilder:CopySource succeeded
2021-10-30 09:45:31,052 | Build inside container returned response {"jsonrpc": "2.0", "id": 1, "result": {"artifacts_dir": "/tmp/samcli/artifacts"}}
2021-10-30 09:45:31,058 | Build inside container was successful. Copying artifacts from container to host
2021-10-30 09:45:31,460 | Copying from container: /tmp/samcli/artifacts/. -> C:\Temp\Innovative\cdk\sam-app\.aws-sam\build\sam-app\cdkhelloworldlambda1060AF2E
2021-10-30 09:45:31,805 | Build inside container succeeded

Build Succeeded

Built Artifacts  : .aws-sam\build

Commands you can use next
=========================
[*] Invoke Function: sam local invoke
[*] Deploy: cdk deploy -a .aws-sam/build

2021-10-30 09:45:31,841 | Sending Telemetry: {'metrics': [{'commandRun': {'requestId': '82e486bb-f838-4196-9458-7610e9423ef6', 'installationId': '70ce78ee-79f5-4ef9-867e-02893e68d012', 'sessionId': '0ee5c211-24b8-4f47-8376-da61cd61d194', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.8.8', 'samcliVersion': '1.29.0.dev202108311500', 'metricSpecificAttributes': {'projectType': 'CDK'}, 'awsProfileProvided': True, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam build', 'duration': 7855, 'exitReason': 'success', 'exitCode': 0}}]}
2021-10-30 09:45:32,284 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)

Expected result:

The manifest.json should reflect the account and region of the configured profile in the environment property, instead of aws://unknown-account/unknown-region

The profile should also override or set the CDK_DEFAULT_ACCOUNT and CDK_DEFAULT_REGION in the case where env=core.Environment(account=os.getenv('CDK_DEFAULT_ACCOUNT'), region=os.getenv('CDK_DEFAULT_REGION')) is used to initialize the stack instance. This is actually the biggest problem. The build command sets never sets the environment property region to unknown-region if set from the environment variable. The same is not true for the account number. This has been submitted as it's own issue #3428

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: Windows 10.0.19042 Build 19042
2. sam --version: SAM CLI, version 1.29.0.dev202108311500
3. AWS region: us-west-2

[rupe120]

If anyone is looking for a work around, this is what I used. I build projects for many different accounts, so my default profile is intentionally empty and I am always specifying a profile.

I created a sam_helper.py with the following:

from os import listdir, path
from os.path import isdir, join

def get_lambda_src_path(construct_id, lambda_id, lambda_source_path):
    lambda_folder_starts_with = lambda_id.replace("-", "")
    stack_build_path = f"./.aws-sam/build/{construct_id}"
    lambda_final_source_path = lambda_source_path 

    if(path.exists(stack_build_path)):
        built_functions = [diritem for diritem in listdir(stack_build_path) if isdir(join(stack_build_path, diritem)) and diritem.startswith(lambda_folder_starts_with)]

        if len(built_functions) > 0:
            lambda_final_source_path = join(stack_build_path, built_functions[0])

    print(f"{lambda_id} Source Path")
    print(lambda_final_source_path)

    return lambda_final_source_path

I then import and use it like this.

...
from .sam_helper import get_lambda_src_path
...

        transformLambdaId = "s3-dms-trigger-transform-lambda"

        # Find the folder in .aws-sam/build/data-lake-S3DmsTriggerStack????? that represents the current Lambda, or just return the standard path
        transformLambdaSourcePath = get_lambda_src_path(construct_id, transformLambdaId, "./s3_dms_trigger/s3_dms_trigger_transform")
        
        self.s3_dms_trigger_transform_lambda = _lambda.Function(
            scope=self,
            id=transformLambdaId,
            runtime=_lambda.Runtime.PYTHON_3_8,
            role=lambda_transform_role,
            function_name="s3-dms-trigger-transform-lambda-function",
            code=_lambda.Code.from_asset(transformLambdaSourcePath),

The final piece is that I use a shell script when I run sam-beta-cdk build to clear out the .aws-sam/build folder, because the code above runs before the build folder is removed by the build command.

Remove-Item -Recurse -Force .\.aws-sam\build
sam-beta-cdk build --project-type cdk --use-container 

Now the deploy can be a standard cdk deploy --profile my-profile-name and the Lambda will use the compiled folder from the SAM build.

I also delete the build folder just before running a local invoke.

[CoshUS]

@rupe120 Thanks for providing the workaround! We will look into this issue.

[moelasmar]

Thanks for raising this issue, and the workaround. The issue is sam-beta-cdk ignores the --profile option and does not pass it to cdk synth.

Also as a work around, you can run cdk synth first with whatever parameters you want, then you can run sam-beta-cdk build --cdk-app <path to the directory that contains the output cloud asembly like ./cdk.out> ... this should help you to fix this issue.

I see from your comments that you was using sam-beta-cdk build before running cdk deploy, can I ask are you using this just for testing purposes, or you have a specific use case that it is not covered by CDK build features (Assets bundling option, and Runtime Lambda Function Constructs Python, nodeJs, go)

[rupe120]

I see from your comments that you was using sam-beta-cdk build before running cdk deploy, can I ask are you using this just for testing purposes, or you have a specific use case that it is not covered by CDK build features (Assets bundling option, and Runtime Lambda Function Constructs Python, nodeJs, go)

I'm mainly running the sam-beta-cdk build for dependency handling (requirements.txt in my Python based Lambdas)

[rupe120]

Thanks for raising this issue, and the workaround. The issue is sam-beta-cdk ignores the --profile option and does not pass it to cdk synth.

Also as a work around, you can run cdk synth first with whatever parameters you want, then you can run sam-beta-cdk build --cdk-app <path to the directory that contains the output cloud asembly like ./cdk.out> ... this should help you to fix this issue.

I see from your comments that you was using sam-beta-cdk build before running cdk deploy, can I ask are you using this just for testing purposes, or you have a specific use case that it is not covered by CDK build features (Assets bundling option, and Runtime Lambda Function Constructs Python, nodeJs, go)

I tried the --cdk-app switch for sam-beta-cdk build and for some reason it made the build command take almost an hour, while my work around runs in just over 3 minutes. There seems to be a bug with that switch.

[ssenchenko]

duplicate of #3428

[rupe120]

This is a different issue than #3428. The --profile switch should still be fixed if it is going to be included as an option.