Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Should the sam build during deploy include "--use-container"? #749

Closed
seaders opened this issue Jan 15, 2019 · 9 comments · Fixed by #797
Closed

Should the sam build during deploy include "--use-container"? #749

seaders opened this issue Jan 15, 2019 · 9 comments · Fixed by #797
Labels
feature-request New feature or enhancement. May require GitHub community feedback.
Milestone
Sprint 7

Comments

@seaders
Copy link

seaders commented Jan 15, 2019 •
edited

Or at least an option to include it?

aws-toolkit-jetbrains/jetbrains-core/src/software/aws/toolkits/jetbrains/services/lambda/deploy/SamDeployDialog.kt

Line 94 in 0a1728d

val command = createBaseCommand()

I'm running my lambda function in a virtual environment, and it has 2 dependencies, praw and pycrypto. My machine python install is under anaconda. That install has pycrypto installed, but not praw.

When I run the default deploy (right-click, Deploy Serverless Application), and it runs the sam build --template .../template.yaml --build-dir .../.aws-sam/build command, I get an error telling me my module can't be imported because pycrypto can't be imported.

When I check the build folder, I see that praw has been installed by the pip build process, but pycrypto hasn't (because it's already installed on that system python install).

I've actually gone and manually added --use-container in to the sam script,

import re
import sys

from samcli.cli.main import cli

if __name__ == '__main__':
    if ('build' in sys.argv) and ('--use-container' not in sys.argv):
        sys.argv.append('--use-container')

    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
    sys.exit(cli())

And now everything works correctly.

So yeah, should --use-container be there by default? Or at least give us an option to include it? Because it doesn't seem like the built will build correctly without it.
@abrooksv
Copy link
Contributor

Where is your requirements.txt and also its contents?

@seaders
Copy link
Author

seaders commented Jan 15, 2019

It's location is within my code folder (in this case reddiseaders). If I remove it or blank it, praw isn't installed, so it's definitely being read.

Contents:

praw==6.0.0
pycrypto==2.6.1

@abrooksv
Copy link
Contributor

Can you also include the full sam build logs from sam build --template .../template.yaml --build-dir .../.aws-sam/build --debug?

@seaders
Copy link
Author

seaders commented Jan 15, 2019

(starting from where I think is relevant, I'll include the full lot if you want)

2019-01-15 18:09:23 Running workflow 'PythonPipBuilder'
2019-01-15 18:09:23 Running PythonPipBuilder:ResolveDependencies
2019-01-15 18:09:23 calling pip download -r /Users/seaders/Desktop/reddiseaders-testhttp/reddiseaders/requirements.txt --dest /var/folders/pj/t75m622168d0rs54tt7_grg80000gp/T/tmpozieugt3
2019-01-15 18:09:26 Full dependency closure: {update-checker==0.16(wheel), prawcore==1.0.0(wheel), urllib3==1.24.1(wheel), idna==2.8(wheel), certifi==2018.11.29(wheel), chardet==3.0.4(wheel), requests==2.21.0(wheel), pycrypto==2.6.1(sdist), praw==6.0.0(wheel)}
2019-01-15 18:09:26 initial compatible: {update-checker==0.16(wheel), prawcore==1.0.0(wheel), urllib3==1.24.1(wheel), idna==2.8(wheel), chardet==3.0.4(wheel), requests==2.21.0(wheel), certifi==2018.11.29(wheel), praw==6.0.0(wheel)}
2019-01-15 18:09:26 initial incompatible: {pycrypto==2.6.1(sdist)}
2019-01-15 18:09:26 Downloading missing wheels: {pycrypto==2.6.1(sdist)}
2019-01-15 18:09:26 calling pip download --only-binary=:all: --no-deps --platform manylinux1_x86_64 --implementation cp --abi cp36m --dest /var/folders/pj/t75m622168d0rs54tt7_grg80000gp/T/tmpozieugt3 pycrypto==2.6.1
2019-01-15 18:09:28 compatible wheels after second download pass: {update-checker==0.16(wheel), prawcore==1.0.0(wheel), urllib3==1.24.1(wheel), idna==2.8(wheel), chardet==3.0.4(wheel), requests==2.21.0(wheel), certifi==2018.11.29(wheel), praw==6.0.0(wheel)}
2019-01-15 18:09:28 Build missing wheels from sdists (C compiling True): {pycrypto==2.6.1(sdist)}
2019-01-15 18:09:28 calling pip wheel --no-deps --wheel-dir /var/folders/pj/t75m622168d0rs54tt7_grg80000gp/T/tmpozieugt3 /var/folders/pj/t75m622168d0rs54tt7_grg80000gp/T/tmpozieugt3/pycrypto-2.6.1.tar.gz
2019-01-15 18:09:44 compatible after building wheels (no C compiling): {update-checker==0.16(wheel), prawcore==1.0.0(wheel), urllib3==1.24.1(wheel), idna==2.8(wheel), chardet==3.0.4(wheel), requests==2.21.0(wheel), certifi==2018.11.29(wheel), praw==6.0.0(wheel)}
2019-01-15 18:09:44 Build missing wheels from sdists (C compiling False): {pycrypto==2.6.1(sdist)}
2019-01-15 18:09:44 calling pip wheel --no-deps --wheel-dir /var/folders/pj/t75m622168d0rs54tt7_grg80000gp/T/tmpozieugt3 /var/folders/pj/t75m622168d0rs54tt7_grg80000gp/T/tmpozieugt3/pycrypto-2.6.1.tar.gz
2019-01-15 18:09:52 compatible after building wheels (C compiling): {update-checker==0.16(wheel), prawcore==1.0.0(wheel), urllib3==1.24.1(wheel), idna==2.8(wheel), chardet==3.0.4(wheel), requests==2.21.0(wheel), certifi==2018.11.29(wheel), praw==6.0.0(wheel)}
2019-01-15 18:09:52 Final compatible: {update-checker==0.16(wheel), prawcore==1.0.0(wheel), urllib3==1.24.1(wheel), idna==2.8(wheel), chardet==3.0.4(wheel), requests==2.21.0(wheel), certifi==2018.11.29(wheel), praw==6.0.0(wheel)}
2019-01-15 18:09:52 Final incompatible: {pycrypto==2.6.1(wheel)}
2019-01-15 18:09:52 Final missing wheels: {pycrypto==2.6.1(sdist)}

and then the stacktrace bit,

2019-01-15 18:09:52 PythonPipBuilder:ResolveDependencies failed
Traceback (most recent call last):
  File "/Users/seaders/.local/lib/python3.6/site-packages/aws_lambda_builders/workflows/python_pip/actions.py", line 27, in execute
    self.scratch_dir
  File "/Users/seaders/.local/lib/python3.6/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 143, in build_dependencies
    requirements_path, artifacts_dir_path, scratch_dir_path)
  File "/Users/seaders/.local/lib/python3.6/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 212, in build_site_packages
    raise MissingDependencyError(packages_without_wheels)
aws_lambda_builders.workflows.python_pip.packager.MissingDependencyError: {pycrypto==2.6.1(sdist)}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/seaders/.local/lib/python3.6/site-packages/aws_lambda_builders/workflow.py", line 164, in run
    action.execute()
  File "/Users/seaders/.local/lib/python3.6/site-packages/aws_lambda_builders/workflows/python_pip/actions.py", line 30, in execute
    raise ActionFailedError(str(ex))
aws_lambda_builders.actions.ActionFailedError: {pycrypto==2.6.1(sdist)}
Build Failed
Error: PythonPipBuilder:ResolveDependencies - {pycrypto==2.6.1(sdist)}

Obviously there was a bit more going on here than I knew about.

@seaders
Copy link
Author

seaders commented Jan 15, 2019

And here's the debug log of a successful run (with --use-container flag),

Found workflow 'PythonPipBuilder' to support capabilities 'Capability(language='python', dependency_manager='pip', application_framework=None)'
Running workflow 'PythonPipBuilder'
Running PythonPipBuilder:ResolveDependencies
calling pip download -r /tmp/samcli/source/requirements.txt --dest /tmp/samcli/scratch
Full dependency closure: {urllib3==1.24.1(wheel), pycrypto==2.6.1(sdist), praw==6.0.0(wheel), requests==2.21.0(wheel), idna==2.8(wheel), certifi==2018.11.29(wheel), update-checker==0.16(wheel), prawcore==1.0.0(wheel), chardet==3.0.4(wheel)}
initial compatible: {urllib3==1.24.1(wheel), praw==6.0.0(wheel), requests==2.21.0(wheel), idna==2.8(wheel), certifi==2018.11.29(wheel), update-checker==0.16(wheel), prawcore==1.0.0(wheel), chardet==3.0.4(wheel)}
initial incompatible: {pycrypto==2.6.1(sdist)}
Downloading missing wheels: {pycrypto==2.6.1(sdist)}
calling pip download --only-binary=:all: --no-deps --platform manylinux1_x86_64 --implementation cp --abi cp36m --dest /tmp/samcli/scratch pycrypto==2.6.1
compatible wheels after second download pass: {urllib3==1.24.1(wheel), praw==6.0.0(wheel), requests==2.21.0(wheel), idna==2.8(wheel), certifi==2018.11.29(wheel), update-checker==0.16(wheel), prawcore==1.0.0(wheel), chardet==3.0.4(wheel)}
Build missing wheels from sdists (C compiling True): {pycrypto==2.6.1(sdist)}
calling pip wheel --no-deps --wheel-dir /tmp/samcli/scratch /tmp/samcli/scratch/pycrypto-2.6.1.tar.gz
compatible after building wheels (no C compiling): {urllib3==1.24.1(wheel), pycrypto==2.6.1(wheel), praw==6.0.0(wheel), requests==2.21.0(wheel), idna==2.8(wheel), certifi==2018.11.29(wheel), update-checker==0.16(wheel), prawcore==1.0.0(wheel), chardet==3.0.4(wheel)}
Build missing wheels from sdists (C compiling False): set()
compatible after building wheels (C compiling): {urllib3==1.24.1(wheel), pycrypto==2.6.1(wheel), praw==6.0.0(wheel), requests==2.21.0(wheel), idna==2.8(wheel), certifi==2018.11.29(wheel), update-checker==0.16(wheel), prawcore==1.0.0(wheel), chardet==3.0.4(wheel)}
Final compatible: {praw==6.0.0(wheel), idna==2.8(wheel), certifi==2018.11.29(wheel), update-checker==0.16(wheel), chardet==3.0.4(wheel), urllib3==1.24.1(wheel), pycrypto==2.6.1(wheel), requests==2.21.0(wheel), prawcore==1.0.0(wheel)}
Final incompatible: set()
Final missing wheels: set()
PythonPipBuilder:ResolveDependencies succeeded
Running PythonPipBuilder:CopySource
PythonPipBuilder:CopySource succeeded

Looks like I've a quite specific situation because of the weird-ass pycrypto anaconda installed?

@abrooksv
Copy link
Contributor

Thanks, the error log shows that the error handling is not working correctly. There is supposed to be a nice messaging saying that it could not package a certain package.

Going to split this into 2 issues:

  1. Feature request for --use-container
  2. Fix the error handling in Lambda builders

@abrooksv abrooksv mentioned this issue Jan 15, 2019
Open
@abrooksv
Copy link
Contributor

Lambda builders issue: aws/aws-lambda-builders#71

@abrooksv abrooksv added the feature-request New feature or enhancement. May require GitHub community feedback. label Jan 15, 2019
@seaders
Copy link
Author

seaders commented Jan 15, 2019

👍

On the --use-container bit, might you expand that to a few things? Like even here, --debug to be able to see the rest of the output? Or a generic, pass other commands to the build command?

I understand that's fairly awkward here because there's multiple commands happening one after the other, but just thought I'd suggest.

@abrooksv abrooksv added this to the Sprint 7 milestone Feb 11, 2019
@rli rli mentioned this issue Feb 12, 2019
Merged
9 tasks
@Rubyj
Copy link

Rubyj commented Oct 9, 2019

@abrooksv How do we get this issue to work? My build is failing due to a whl not being found for a package scikit-allel

Final incompatible: set()
Final missing wheels: {scikit-allel==1.2.1(sdist)}
PythonPipBuilder:ResolveDependencies failed
Traceback (most recent call last):
  File "/var/lang/lib/python3.7/site-packages/aws_lambda_builders/workflows/python_pip/actions.py", line 39, in execute
    requirements_path=self.manifest_path
  File "/var/lang/lib/python3.7/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 143, in build_dependencies
    requirements_path, artifacts_dir_path, scratch_dir_path)
  File "/var/lang/lib/python3.7/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 212, in build_site_packages
    raise MissingDependencyError(packages_without_wheels)
aws_lambda_builders.workflows.python_pip.packager.MissingDependencyError: {scikit-allel==1.2.1(sdist)}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lang/lib/python3.7/site-packages/aws_lambda_builders/workflow.py", line 248, in run
    action.execute()
  File "/var/lang/lib/python3.7/site-packages/aws_lambda_builders/workflows/python_pip/actions.py", line 42, in execute
    raise ActionFailedError(str(ex))
aws_lambda_builders.actions.ActionFailedError: {scikit-allel==1.2.1(sdist)}
Builder workflow failed
Traceback (most recent call last):
  File "/var/lang/lib/python3.7/site-packages/aws_lambda_builders/workflows/python_pip/actions.py", line 39, in execute
    requirements_path=self.manifest_path
  File "/var/lang/lib/python3.7/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 143, in build_dependencies
    requirements_path, artifacts_dir_path, scratch_dir_path)
  File "/var/lang/lib/python3.7/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 212, in build_site_packages
    raise MissingDependencyError(packages_without_wheels)
aws_lambda_builders.workflows.python_pip.packager.MissingDependencyError: {scikit-allel==1.2.1(sdist)}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lang/lib/python3.7/site-packages/aws_lambda_builders/workflow.py", line 248, in run
    action.execute()
  File "/var/lang/lib/python3.7/site-packages/aws_lambda_builders/workflows/python_pip/actions.py", line 42, in execute
    raise ActionFailedError(str(ex))
aws_lambda_builders.actions.ActionFailedError: {scikit-allel==1.2.1(sdist)}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lang/lib/python3.7/site-packages/aws_lambda_builders/__main__.py", line 137, in main
    mode=params.get('mode', None))
  File "/var/lang/lib/python3.7/site-packages/aws_lambda_builders/builder.py", line 115, in build
    return workflow.run()
  File "/var/lang/lib/python3.7/site-packages/aws_lambda_builders/workflow.py", line 65, in wrapper
    func(self, *args, **kwargs)
  File "/var/lang/lib/python3.7/site-packages/aws_lambda_builders/workflow.py", line 257, in run
    reason=str(ex))
aws_lambda_builders.exceptions.WorkflowFailedError: PythonPipBuilder:ResolveDependencies - {scikit-allel==1.2.1(sdist)}
Build inside container returned response {"jsonrpc": "2.0", "id": 1, "error": {"code": 400, "message": "PythonPipBuilder:ResolveDependencies - {scikit-allel==1.2.1(sdist)}"}}

Build Failed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
feature-request New feature or enhancement. May require GitHub community feedback.
Projects
None yet
Milestone
Sprint 7
Development

Successfully merging a pull request may close this issue.

Add option to --use-container during serverless app deployment aws/aws-toolkit-jetbrains
3 participants
@seaders @Rubyj @abrooksv