terraform apply failing with pip_requirements and python3.6 because of pip options

[edsenabr]

I have a very simple setup:

module "lambda_function" {
  source = "terraform-aws-modules/lambda/aws"
  function_name = "sample-function"
  handler       = "sample.lambda_handler"
  runtime       = "python3.6"
  lambda_role   = aws_iam_role.lambda.arn
  source_path = {
    path        = "lambda",
    pip_requirements = true
  }
}

... and when I try to run an terraform apply, I receive the following error:

Error: Error running command 'builds/ae0f6c889fb359294fdeabfbefc9046b5694c660221cd580ce3fd730401a9f08.plan.json': exit status 1. Output: zip: creating 'builds/ae0f6c889fb359294fdeabfbefc9046b5694c660221cd580ce3fd730401a9f08.zip' archive
Installing python requirements: lambda/requirements.txt
> mktemp -d terraform-aws-lambda-XXXXXXXX # /tmp/terraform-aws-lambda-d2oqcfi9
> cd /tmp/terraform-aws-lambda-d2oqcfi9
> python3.6 -m pip install --no-compile --prefix= --target=. --requirement=requirements.txt
Collecting boto3>=1.16.32 (from -r requirements.txt (line 1))
  Using cached https://files.pythonhosted.org/packages/07/f6/b12ccd6484c15a6d22aaeda6d7aac4278d923a327f63eda6e61072b02f26/boto3-1.16.32.tar.gz
Collecting botocore<1.20.0,>=1.19.32 (from boto3>=1.16.32->-r requirements.txt (line 1))
  Using cached https://files.pythonhosted.org/packages/40/c3/5fd80e8fc843fe8596eeee430375a81716e91ebed6a0607614c80df12800/botocore-1.19.32.tar.gz
Collecting jmespath<1.0.0,>=0.7.1 (from boto3>=1.16.32->-r requirements.txt (line 1))
  Using cached https://files.pythonhosted.org/packages/3c/56/3f325b1eef9791759784aa5046a8f6a1aff8f7c898a2e34506771d3b99d8/jmespath-0.10.0.tar.gz
Collecting s3transfer<0.4.0,>=0.3.0 (from boto3>=1.16.32->-r requirements.txt (line 1))
  Using cached https://files.pythonhosted.org/packages/50/de/2b688c062107942486c81a739383b1432a72717d9a85a6a1a692f003c70c/s3transfer-0.3.3.tar.gz
Collecting python-dateutil<3.0.0,>=2.1 (from botocore<1.20.0,>=1.19.32->boto3>=1.16.32->-r requirements.txt (line 1))
  Using cached https://files.pythonhosted.org/packages/be/ed/5bbc91f03fa4c839c4c7360375da77f9659af5f7086b7a7bdda65771c8e0/python-dateutil-2.8.1.tar.gz
Collecting urllib3<1.27,>=1.25.4 (from botocore<1.20.0,>=1.19.32->boto3>=1.16.32->-r requirements.txt (line 1))
  Using cached https://files.pythonhosted.org/packages/29/e6/d1a1d78c439cad688757b70f26c50a53332167c364edb0134cadd280e234/urllib3-1.26.2.tar.gz
Collecting six>=1.5 (from python-dateutil<3.0.0,>=2.1->botocore<1.20.0,>=1.19.32->boto3>=1.16.32->-r requirements.txt (line 1))
  Using cached https://files.pythonhosted.org/packages/6b/34/415834bfdafca3c5f451532e8a8d9ba89a21c9743a0c59fbd0205c7f9426/six-1.15.0.tar.gz
Skipping bdist_wheel for boto3, due to binaries being disabled for it.
Skipping bdist_wheel for botocore, due to binaries being disabled for it.
Skipping bdist_wheel for jmespath, due to binaries being disabled for it.
Skipping bdist_wheel for s3transfer, due to binaries being disabled for it.
Skipping bdist_wheel for python-dateutil, due to binaries being disabled for it.
Skipping bdist_wheel for urllib3, due to binaries being disabled for it.
Skipping bdist_wheel for six, due to binaries being disabled for it.
Installing collected packages: jmespath, six, python-dateutil, urllib3, botocore, s3transfer, boto3
  Running setup.py install for jmespath: started
    Running setup.py install for jmespath: finished with status 'error'
    Complete output from command /usr/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-jnmeomaa/jmespath/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-pwjzp8ah-record/install-record.txt --single-version-externally-managed --prefix  --no-compile --user --prefix= --home=/tmp/tmp0etmy0xb:
    running install
    error: can't combine user with prefix, exec_prefix/home, or install_(plat)base
    
    ----------------------------------------
/usr/lib/python3/dist-packages/pip/req/req_file.py:150: UserWarning: Disabling all use of wheels due to the use of --build-options / --global-options / --install-options.
  cmdoptions.check_install_build_global(options, opts)
Command "/usr/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-jnmeomaa/jmespath/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-pwjzp8ah-record/install-record.txt --single-version-externally-managed --prefix  --no-compile --user --prefix= --home=/tmp/tmp0etmy0xb" failed with error code 1 in /tmp/pip-build-jnmeomaa/jmespath/
zip: Error during zip archive creation
Traceback (most recent call last):
  File ".terraform/modules/lambda_function/package.py", line 1134, in build_command
    bpm.execute(build_plan, zs, query)
  File ".terraform/modules/lambda_function/package.py", line 764, in execute
    with install_pip_requirements(query, pip_requirements) as rd:
  File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File ".terraform/modules/lambda_function/package.py", line 900, in install_pip_requirements
    check_call(pip_command, env=subproc_env)
  File "/usr/lib/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['python3.6', '-m', 'pip', 'install', '--no-compile', '--prefix=', '--target=.', '--requirement=requirements.txt']' returned non-zero exit status 1.

when running pip manually, one has to add a --system flag to it, to bypass this issue. Is there a way to configure this module to use --system instead of --user ?

[edsenabr]

just figured a solution: One has to run terraform apply inside of a virtualenv sandbox.

[antonbabenko]

Great, I will close this issue.

[edsenabr]

hi @antonbabenko , it would be nice if we have a self-contained solution within the module; Probably by internally using virtualenv during the build.

[antonbabenko]

It is going to be a lot of work for us to do and to maintain.

Since last week announcement that AWS Lambdas can be launched from container images, I have released v1.31.0 where the complete support for that feature was implemented (see #80 for details and example for the complete end-to-end implementation with building image, pushing it to ECR, and deploying as AWS Lambda function).

I think it is the best way to go and I am not a big fan of inflating package.py with even more features.

[edsenabr]

But one would need to pip install, even if packaging on containers... I don't see how it would solve this issue

[antonbabenko]

Have you seen examples here - https://github.com/terraform-aws-modules/terraform-aws-lambda/blob/master/examples/build-package/main.tf ?

There are many combinations of parameters specifically for Python.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.