Lacking examples of managing/deploying Python dependencies

[alukach]

❓ Guidance Question

The Question

As mentioned in aws/aws-cdk#3660, we are lacking a good example of a more realistic Lambda which includes dependencies. How should a developer ensure that handlers are packaged with their required dependencies? The current examples/boilerplate code only seems to demonstrate managing deployment code (i.e. cdk dependencies) but does not demonstrate best-practice for handling runtime dependencies.

Perhaps the correct solution is to utilize Lambda Layers to package dependencies? If so, it would be helpful to have that demonstrated in an example project.

Whatever the solution, it should demonstrate:

• separating runtime dependencies from build-time dependencies

Additionally, it would be nice if we could demonstrate:

• supporting the concept of packaging different handlers with only the dependencies that they need

[benfarr]

@alukach Layers is a good way to go, depending on what dependencies you're trying to include. I just submitted this example project that includes a requests layer used by one of the Lambdas. Hope it helps: #134

[JamieMcKernanKaizen]

Yeah I'm also needing to do this (and I'm surprised it isn't well documented). It seems like it would be something that people would want to be doing often!

[alukach]

@JamieMcKernanKaizen I ended up doing something like the following:

import os
import subprocess
from typing import List

from aws_cdk import (
    core,
    aws_lambda,
)

class MyStack(core.Stack):
    def __init__(
        self,
        scope: core.Construct,
        id: str,
        requirements_path: str = "./requirements.txt",
        layer_dir: str = "./.layer",
        **kwargs,
    ) -> None:
        super().__init__(scope, id, **kwargs)

        dependenciesLayer = self.create_dependencies_layer(
            requirements_path=requirements_path, output_dir=layer_dir
        )
        self.create_handler(
            layers=[dependenciesLayer],
        )

    def create_dependencies_layer(
        self, requirements_path: str, output_dir: str
    ) -> aws_lambda.LayerVersion:
        # Install requirements for layer
        if not os.environ.get("SKIP_PIP"):
            subprocess.check_call( 
                # Note: Pip will create the output dir if it does not exist
                f"pip install -r {requirements_path} -t {output_dir}/python".split()
            )
        return aws_lambda.LayerVersion(
            self, "HandlerDependencies", code=aws_lambda.Code.from_asset(output_dir)
        )
    
    def create_handler(
        self,
        layers: List[aws_lambda.LayerVersion],
    ):
        return aws_lambda.Function(
            self,
            "Some Function",
            code=aws_lambda.Code.from_asset('path/to/code'),
            handler='handlers.my_handler',
            runtime=aws_lambda.Runtime.PYTHON_3_7,
            layers=layers,
        )

[JamieMcKernanKaizen]

@alukach Thanks! I found that installing the packages locally to the lambda code asset folder worked too!

[kjpgit]

I wrote an example here: https://github.com/kjpgit/techdemo/tree/master/cdk_python_lambda
That's based on our real-world usage, on 3 projects.

[rhboyd]

@alukach thank you for that great example. I would like to send you some CDK stickers to thank you for your contribution. Please feel free to email me at rhboyd@amazon.com with a preferred mailing address and I will send them out to you.

[gmiretti]

Making an aws-samples/aws-cdk-examples for python using bundling would be great.

Mainly because the example code in section "Bundling asset code" in CDK Python docs (v 1.47.1) doesn't work without some tweaking ( as published in https://docs.aws.amazon.com/cdk/api/latest/python/aws_cdk.aws_lambda.README.html#id3 )

[alukach]

doesn't work without some tweaking

@gmiretti Can you expand on this? What tweaking did you need to make this function as expected?

[gmiretti]

Mostly the usual missing translation from typescript to Python (lambda underscored, no __dirname, correct multiline string, etc.)

_lambda.Function(self, "Function",
    code=_lambda.Code.from_asset("./lambda",
        bundling={
            "image": _lambda.Runtime.PYTHON_3_6.bundling_docker_image,
            "command": ["bash", "-c", 
        """
        pip install -r requirements.txt -t /asset-output &&
        rsync -r . /asset-output
        """
            ]
        }
    ),
    runtime=_lambda.Runtime.PYTHON_3_6,
    handler="handler.handler"
)

where the dir layout is:

/
- lambda/
-- handler.py
-- requirements.txt

Also to test it locally you have to do chmod -R 777 .cdk.staging after cdk synth --no-staging > template.yaml because of aws/aws-cdk#8707

[esteban-uo]

I just published a simple example to deal with it, hope it helps someone else https://github.com/esteban-uo/aws-cdk-python-application-dependencies (based on https://stackoverflow.com/a/61248003)

[alex9311]

I ran into this issue as well. I used a solution like @alukach just fine when I was working on my ubuntu machine. However, I ran into issue when working on MacOS. Apparently some (all?) python packages don't work on lambda (linux env) if they are pip installeded on a different os (see stackoverflow post)

My solution was to run the pip install inside a docker container. This allowed me to cdk deploy from my macbook and not run into issues with my python packages in lambda.

suppose you have a dir lambda_layers/python in your cdk project that will house your python packages for the lambda layer.

lambda_name = 'image-pipeline-image-processor'
current_path = str(pathlib.Path(__file__).parent.absolute())
pip_install_command = ("docker run --rm --entrypoint /bin/bash -v "
            + current_path
            + "/lambda_layers:/lambda_layers python:3.8 -c "
            + "'pip3 install Pillow==8.1.0 -t /lambda_layers/python'")
subprocess.run(pip_install_command, shell=True)
lambda_layer = aws_lambda.LayerVersion(
    self,
    lambda_name+"-layer",
    compatible_runtimes=[aws_lambda.Runtime.PYTHON_3_8],
    code=aws_lambda.Code.asset("lambda_layers"))

[alex9311]

Actually, there does appear to be new functionality for preparing python dependencies for lambda functions built into CDK now

https://docs.aws.amazon.com/cdk/api/latest/python/aws_cdk.aws_lambda_python/PythonFunction.html

[JamieMcKernanKaizen]

Actually, there does appear to be new functionality for preparing python dependencies for lambda functions built into CDK now

https://docs.aws.amazon.com/cdk/api/latest/python/aws_cdk.aws_lambda_python/PythonFunction.html

Yup, I recently updated an SO answer of mine because of this.

[kaiz-io]

CDK Python Docs has an example showing how to package dependencies. https://docs.aws.amazon.com/cdk/api/v2/python/aws_cdk.aws_lambda_python_alpha/README.html Though I will wait for it to go GA before closing this.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.