sam package fails to include node_modules when --template-file is specified

[greensopinion]

Description

Using a function created using sam init --runtime nodejs --name hello-world, the deployed artifact on s3 does not include the node_modules folder. This results in a runtime error:

{
    "errorMessage": "Cannot find module 'axios'",
    "errorType": "Error",
    "stackTrace": [
        "Module.require (module.js:596:17)",
        "require (internal/module.js:11:18)",
        "Object.<anonymous> (/var/task/app.js:2:15)",
        "Module._compile (module.js:652:30)",
        "Object.Module._extensions..js (module.js:663:10)",
        "Module.load (module.js:565:32)",
        "tryModuleLoad (module.js:505:12)",
        "Function.Module._load (module.js:497:3)"
    ]
}

Steps to reproduce

Run the following commands in order (per the README.md):

1. sam init --runtime nodejs --name hello-world-example
2. sam build
3. sam package --template-file template.yaml --output-template-file packaged.yaml --s3-bucket getting-started2

Observe that the s3 file created, when unzipped, does not contain a node_modules folder. The .aws-sam/build/HelloWorldFunction/node_modules folder is present.

If step 3 is replaced with sam package --s3-bucket getting-started2 the s3 file created does include a node_modules folder with all of the expected files/folders.

Observed result

$ sam package --template-file template.yaml --output-template-file packaged.yaml --s3-bucket getting-started2

Successfully packaged artifacts and wrote output template to file packaged.yaml.
Execute the following command to deploy the packaged template
aws cloudformation deploy --template-file /Users/dgreen/git/sam-hello/hello-world-example/packaged.yaml --stack-name <YOUR STACK NAME>

Expected result

The file uploaded to s3 should include the node_modules folder.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: Mac
2. sam --version:
   SAM CLI, version 0.9.0

Add --debug flag to command you are running

[jfuss]

@greensopinion Thanks for creating the issue but what you are seeing is expected.

When passing in --template-file option, you are telling the command to use that template. So if the CodeUri location for a function points at a directory that does not contain the node_modules folder then it will not be packaged. When using package with build, you can omit the --template-file from the command. This will tell sam to look for default locations for templates (ones produced by sam build and the default template.[yaml|yml]).

Closing since this is the expected behavior.

[greensopinion]

@jfuss thanks for the clarification, that helps a lot.

Unless I'm mistaken, it looks like the template README might need updating, since the steps provided there lead to this problem.

Would it make sense to update places where --template-file and --output-template-file are mentioned to prefix the file paths with .aws-sam/build/? If so I could issue a PR to that effect.

[jfuss]

@greensopinion We should just remove the --template-file from the package command. We did this in the python example but looks like we missed this for node and ruby.

A PR with the update would be great!

[greensopinion]

@jfuss anything left to do here? It looks to me as though the CI test failure is unrelated to this change:

========================== 772 passed in 9.16 seconds ==========================
The command "make pr" exited with 0.
1200.06s$ SAM_CLI_DEV=1 travis_wait pytest -vv tests/integration
Still running (20 of 20): pytest -vv tests/integration
Timeout (20 minutes) reached. Terminating "pytest -vv tests/integration"
The command pytest -vv tests/integration exited with 137.

[jfuss]

@greensopinion This was addressed in #849. I didn't realize that previously, so sorry about that.

The the integ tests that fails is flaky on Travis. We have it captured in #857 but you can safely ignore this failure.

[greensopinion]

No problem @jfuss thanks.

[jfuss]

Released in v0.11.0

Closing

[sdole]

I stumbled upon this when looking up another issue. I just wanted to share that the sam package --template-file (-t) option is useful when there are no lambdas to be built, and the template contains only an api gateway. (lambdas are deployed separately).

[anandarpit]

Just adding that also check for .gitignore file. If it contains node_modules written in it then delete that line. This helped me

[ahmednkhan24]

"When using package with build, you can omit the --template-file from the command. This will tell sam to look for default locations for templates (ones produced by sam build and the default template.[yaml|yml])."

What if I need to use a file that isn't the default template.yaml?
I have a pre-process script that will replace a few variables within the template.yaml file and output the contents to a new file:

sed -e 's/{MY_VARIABLE}/test/g' template.yaml > processed.yaml

When I run a sam build && sam package --template-file processed.yaml each subdirectory has a node_modules folder, ie .aws-sam/build/*/node_modules/, but the root build directory does not.

[vishwasnavadak]

Not sure if this helps, If you are still facing this issue, I upgraded my SAM CLI from 1.37.0 to 1.38.1 and then the node_modules files were uploaded to Lambda.