aws cloudformation package should put artifacts in a local bucket

[mariano-calandra-xp]

Hi, I would like to deploy a SAM application to localstack. I have essentially 3 commands to run a sam build followed by a sam package and sam deploy. The second command upload artifacts to S3 and I would like to have this bucket simulated locally.

So I created it aws s3api create-bucket --bucket my-bucket --endpoint-url http://localhost:4572 and then I executed the sam build command. It worked. Now I have to run sam package that is essentually an alias for aws cloudformation package so I execute.

aws cloudformation package \
    --output-template-file packaged.yml \
    --endpoint-url http://localhost:4581 \ 
    --profile localstack \
    --s3-bucket my-bucket \
    --template-file template.yaml

but I get:

Unable to upload artifact XXX referenced by CodeUri parameter of XXX resource.
An error occurred (InvalidAccessKeyId) when calling the PutObject operation: The AWS Access Key Id you provided does not exist in our records.

before to execute this command I executed aws configure --profile localstack and configured access and secret as dummy and us-est-1as defult region.

Can I use local s3 for this or do I need a real S3 bucket?

┆Issue is synchronized with this Jira Bug by Unito

[miron4dev]

I'm receiving the same error. Did anyone find a solution?

[mariano-calandra-xp]

I hope the new sam CLI version may help in resolving this issue

[whummer]

Thanks for reporting @mariano-calandra-xp @miron4dev , good catch. Would you mind creating an upstream issue against sam CLI and/or aws CLI for this issue? Thanks!

[MatteoGioioso]

Any update on this? Is there any workaround?

[newtondev]

I am still experiencing this issue, any workaround?

[whummer]

Hi @newtondev @MatteoGioioso , I believe this should be working with the awslocal command line (https://github.com/localstack/awscli-local) - can you please give it a try and let us know? Thanks

[deepalakshmmic]

@whummer I am experiencing a different error when tried with localstack CLI

awslocal cloudformation package --template-file database.yaml --s3-bucket artifact-bucket --output-template-file database_out.yaml --profile localstack

Unknown options: --s3-endpoint-url=http://localhost:4566

[whummer]

Thanks for the update @deepalakshmmic . Can you please try upgrading to the latest version and confirm if the problem persists?

pip install --upgrade awscli-local

Also, can you please report the version of the underlying awscli module:

awslocal --version

Thanks

[gsteckman]

I'm getting the same error.

awslocal --version
aws-cli/2.0.46 Python/3.7.7 Windows/10 exe/AMD64

I also tried using the aws client with the --endpoint-url on both my Windows 10 host and in WSL2. I get the same error as in the original issue report, in both environments.

Versions below:

aws --version
aws-cli/1.18.69 Python/3.8.5 Linux/4.19.104-microsoft-standard botocore/1.16.19

aws --version
aws-cli/2.0.46 Python/3.7.7 Windows/10 exe/AMD64

[vincentippon]

Same problem here

[ben-elsen]

Same here, any update?

[jordan-cote]

+1
Same here

[paul-uz]

Encountered the same issue.

Any work arounds for deploying Cloudformation stacks?

We have a fairly monolithic main template.yml with a load of nested stacks.

[vincentippon]

If you guys are trying to package a SAM template with AWS CLI, it's simply not possible (sam package is not a simple alias). AWS CLI packages Cloudformation stack, SAM CLI packages SAM stack, that's it.

To deploy a SAM stack to Localstack, you need to have a Cloudformation stack, which can be the output of sam package.

If you want your SAM CLI to be able to package through Localstack (not the AWS cloud), it's not officially supported yet. In fact, SAM CLI won't have this feature like AWS CLI does, but, boto3 has an open PR which can provide this feature (SAM CLI uses boto3 under the hood).

Doing a manual build of SAM CLI with the PR's fork of boto3 allowed me to package and deploy a SAM stack to Localstack without any interaction with the AWS cloud.

[paul-uz]

@vincentippon I'm trying to package an AWS template (using aws cloudformation package ...)

[vincentippon]

@paul-uz If you're working with Cloudformation stack only, your problem is different. The author mentionned a "SAM application", which is not the same use case (SAM is not really Cloudformation).

[paul-uz]

@vincentippon ah fair enough. Still, I'm totally unable to use Localstack at rpesent due to the issues with the cloudformation commands not working proeprly.

Is there another way I can test my Lambda functions? I need to use s3, dynamodb and API Gateway (for REST API requests)

[vincentippon]

@paul-uz Writing as much tests as you can (unit tests and integration tests) with mocks is the first step. You can manually deploy a Dynamo/s3 to Localstack with AWS CLI, use the Localstack endpoint in your lambda and invoke your lambda locally, it's not perfect but can help you a bit more.

However, you should be able to deploy your Cloudformation stack to Localstack, you may want to open an issue or ask on Stackoverflow to get some help on your case.

[paul-uz]

@vincentippon I have opened some issues, just awaiting responses :)

[paul-uz]

@adamarla that might have to be the solution I look into until these issues are fixed.

Can you clairfy what you mean by "the same docker network that localstack is on" ? Do you mean if i run localstack on my mac to run on localhost I just run sam on my mac as well?

[whummer]

Thanks for the update @adamarla @paul-uz @vincentippon @jordan-cote @ben-elsen @gsteckman .

Unfortunately, the cloudformation deploy/package commands only work when using the AWS CLI v1 - for v2, we're depending on the following PR to get merged: aws/aws-cli#3309 . Unfortunately, there is no easy workaround at this point, as the v2 CLI is packaged as a proprietary binary instead of a source package (see also this discussion).

We have just released a new version 0.15 of the awscli-local pip package. It now contains a warning message to inform the user that these particular commands are only available when using the v1 AWS CLI (i.e., installed via pip).

!NOTE! The cloudformation deploy/package commands currently do not work with the AWS CLI v2. Please run "pip install awscli" to install version 1 of the AWS CLI

Hope that helps - we'll keep an eye on the upstream PR and add support for v2 of the AWS CLI, once the --s3-endpoint-url parameter becomes available there. Thanks!

[paul-uz]

@whummer you are required as a reviewer to get aws/aws-cli#3309 merged in. Can you please approve it and get it merged?

[whummer]

@paul-uz Done - however, I'm not a maintainer of that repo (LocalStack has no affiliation with AWS whatsoever), so not sure how much my approval will change in this case... :/

[paul-uz]

@whummer sorry, I saw you as a reviewer so thought you needed to approve. Looks like it still needs someone to approve it, but the PR is not assigned to a maintainer or anything :| hopefully pinged the right person now 🤞