Support for container images

[davidolmo]

Is it currently possible or is it planned to support custom containers?

Lambda announcement

One of the limitations that seems to appear again and again is the 250MB limit of lambda package. This new feature seems to solve that! (in addition to other use cases)

[jamesls]

Yeah this is great! There are some additional requirements needed to make this work (ECR repo along with pushing updated images) so we'd need to figure out the best way to make this work without requiring a lot of management overhead. But I'm marking this as a feature request, and if anyone else would like this, please feel free to vote (:+1:) for this issue.

[sionsmith]

I would be interested in help out with this. A couple of things which should be considered:

• configurable ECR, we should be able to pass in existing but have Chalice create one if there is a Dockerfile present or something.
• Permission on running / configuring container workloads.

[sionsmith]

Also how are people doing this currently? Terraform?

[eliyaba]

Are there any news on this? I am also struggling with the 250MB limit restriction

[denis-ryzhkov]

I managed to patch chalice==1.23.0 to support Lambda Image,
it should be compatible with latest version of Chalice too

It works, replacing 250 MB limit of ZipFile with 10 GB limit of ImageUri

Not sure I find time to create a PR from this patch,
so please either create this PR yourself,
or wait for someone to do it,
or just use this patch as is right now

• lambda-image/Dockerfile:

FROM public.ecr.aws/lambda/python:3.8

# Sequence of commands creates layers of cache,
# so that updating app code does not rebuild requirements

RUN pip3 install pip==21.1.1 wheel==0.36.2

COPY requirements.txt ${LAMBDA_TASK_ROOT}/
RUN pip3 install -r requirements.txt

COPY app.py python ${LAMBDA_TASK_ROOT}/
CMD ["app.app"]

• lambda-image/awsclient-append.py:

# patch:


def _get_image(function_name):
    # You may return `None` to use `ZipFile` depending on `function_name`,
    # or per-project by `CHALICE_LAMBDA_IMAGE=YES chalice deploy --stage $STAGE`
    if os.environ.get("CHALICE_LAMBDA_IMAGE") != "YES":
        return None

    parts = function_name.split("-")
    if len(parts) == 2:
        app_name, stage = parts
        handler_name = "app"
    elif len(parts) == 3:
        app_name, stage, handler_name = parts
    else:
        app_name = stage = handler_name = None

    AWS_ACCOUNT_ID = os.environ["AWS_ACCOUNT_ID"]
    AWS_REGION = os.environ["AWS_REGION"]
    image_name = f"{app_name}-{stage}" if app_name and stage else function_name

    uri = f"{AWS_ACCOUNT_ID}.dkr.ecr.{AWS_REGION}.amazonaws.com/{image_name}:latest"
    cmd = [f"app.{handler_name}"] if handler_name else None
    return {"uri": uri, "cmd": cmd}


def _create_lambda_function(self, api_args):
    # type: (Dict[str, Any]) -> Tuple[str, str]

    image = _get_image(api_args["FunctionName"])
    if image:
        del api_args["Code"]["ZipFile"]
        del api_args["Handler"]
        del api_args["Runtime"]
        api_args["Code"]["ImageUri"] = image["uri"]
        api_args["PackageType"] = "Image"
        if image["cmd"]:
            api_args["ImageConfig"] = {"Command": image["cmd"]}

    try:
        result = self._call_client_method_with_retries(
            self._client("lambda").create_function,
            api_args,
            max_attempts=self.LAMBDA_CREATE_ATTEMPTS,
        )
        return result["FunctionArn"], result["State"]
    except _REMOTE_CALL_ERRORS as e:
        context = LambdaErrorContext(
            api_args["FunctionName"],
            "create_function",
            0 if image else len(api_args["Code"]["ZipFile"]),
        )
        raise self._get_lambda_code_deployment_error(e, context)


def _update_function_code(self, function_name, zip_contents):
    # type: (str, str) -> Dict[str, Any]

    image = _get_image(function_name)
    api_args = {"ImageUri": image["uri"]} if image else {"ZipFile": zip_contents}

    lambda_client = self._client("lambda")
    try:
        result = lambda_client.update_function_code(
            FunctionName=function_name, **api_args
        )
    except _REMOTE_CALL_ERRORS as e:
        context = LambdaErrorContext(
            function_name,
            "update_function_code",
            0 if image else len(zip_contents),
        )
        raise self._get_lambda_code_deployment_error(e, context)
    if result["LastUpdateStatus"] != "Successful":
        self._wait_for_function_update(function_name)
    return result


def _do_update_function_config(self, function_name, kwargs):
    # type: (str, Dict[str, Any]) -> None

    if _get_image(function_name):
        del kwargs["Runtime"]

    _old_do_update_function_config(self, function_name, kwargs)


_old_do_update_function_config = TypedAWSClient._do_update_function_config

TypedAWSClient._create_lambda_function = _create_lambda_function
TypedAWSClient._update_function_code = _update_function_code
TypedAWSClient._do_update_function_config = _do_update_function_config

• Before other imports in app.py:

#
# fix import
#

# See https://github.com/aws/aws-lambda-base-images/issues/8
# resulting in /var/lang/lib/python3.8/site-packages/requests/__init__.py:89:
# RequestsDependencyWarning: urllib3 (1.26.2) or chardet (3.0.4)
# doesn't match a supported version!

import sys

import pkg_resources

site_packages = "/var/lang/lib/python3.8/site-packages"
try:
    sys.path.remove(site_packages)
except ValueError:
    pass
else:
    sys.path.insert(0, site_packages)
    for dist in pkg_resources.find_distributions(site_packages, True):
        pkg_resources.working_set.add(dist, site_packages, False, replace=True)

• If you need a writable directory:

DATA_DIR = "/tmp" if ENV == LOCAL else os.environ.get("LAMBDA_TASK_ROOT", "/var/task")

• Relevant part of the deploy script:

#!/bin/bash
set -euf
IFS=$'\n\t'

# ...
IMAGE_NAME=${PROJECT_ID}_chalice-$ENV

echo "Building lambda image $IMAGE_NAME ..."
CHALICE_PACKAGE_DIR=$(python -c 'import chalice; print(chalice.__path__[0])')
CHALICE_APP_DIR=$(pwd)
IMAGE_DIR=$CHALICE_APP_DIR/lambda-image
IMAGE_BUILD_DIR=$IMAGE_DIR/build
rm -Rf $IMAGE_BUILD_DIR
mkdir $IMAGE_BUILD_DIR
cp $IMAGE_DIR/Dockerfile $IMAGE_BUILD_DIR/
pushd $IMAGE_BUILD_DIR
mkdir -p python/chalice
cp $CHALICE_PACKAGE_DIR/{__init__.py,app.py} python/chalice/
cp -R $CHALICE_APP_DIR/chalicelib python/
cp -R $CHALICE_APP_DIR/{app.py,requirements.txt} ./
docker build -t $IMAGE_NAME .
popd

echo "Reading ECR repo $IMAGE_NAME ..."
aws ecr describe-repositories --repository-names $IMAGE_NAME &>/dev/null || {
    echo "Creating ECR repo $IMAGE_NAME ..."
    aws ecr create-repository --repository-name $IMAGE_NAME >/dev/null
}

echo "Refreshing ECR token of docker..."
ECR_DOMAIN=$AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com
aws ecr get-login-password | docker login --username AWS --password-stdin $ECR_DOMAIN

echo "Tagging and pushing the image..."
IMAGE_TAG=$IMAGE_NAME:latest
ECR_TAG=$ECR_DOMAIN/$IMAGE_TAG
docker tag $IMAGE_TAG $ECR_TAG
docker push $ECR_TAG

echo "Reading Chalice lambda-image patch..."
FILE_NAME_TO_PATCH=$CHALICE_PACKAGE_DIR/awsclient.py
grep -Fq 'TypedAWSClient._create_lambda_function' $FILE_NAME_TO_PATCH || {
    echo "Patching $FILE_NAME_TO_PATCH ..."
    cat lambda-image/awsclient-append.py >> $FILE_NAME_TO_PATCH
}

echo "Temporary updating requirements.txt ..."
mv requirements.txt requirements-backup.txt
# We just need minimal requirements to satisfy Chalice that is importing app on build:
grep -E "boto3|sentry" requirements-backup.txt > requirements.txt
# `try-finally` to always restore from `requirements-backup.txt`:
DEPLOY_FAILED=NO

echo "Deploying Chalice to ENV=$ENV..."
CHALICE_LAMBDA_IMAGE=YES chalice deploy --stage $ENV || DEPLOY_FAILED=YES

echo "Restoring requirements.txt ..."
mv requirements-backup.txt requirements.txt
rm -Rf $IMAGE_BUILD_DIR
[[ $DEPLOY_FAILED == YES ]] && exit 1

• Lambdas created as ZipFile can not be switched to ImageUri,
  so before running the deploy script above, please delete old lambdas from:

  • "Lambda triggers" tab in each SQS, etc
  • Lambda functions list
  • resources in .chalice/deployed/$ENV.json

• Make sure automatic_layer is disabled or absent in .chalice/config.json,
  as it is neither compatible nor needed with Lambda Image

[areeves87]

Still a newbie with aws's sam but I'm leaning toward using it for deploying chalice containers. Obviously official support of containers from chalice itself would be preferred but until then this approach doesn't seem too bad. Here's a minimal example for deploying chalice containers via sam, which I like for its local container testing, management of ecr, patching config files rather than source code:

chalice new-project hello-world && cd hello-world

# Dockerfile for sam build
cat << EOF > ./Dockerfile
FROM public.ecr.aws/lambda/python:3.8

COPY requirements.txt ./

RUN python3.8 -m pip install -r requirements.txt -t .

COPY deployment/ ./

CMD ["app.app"]
EOF

# generate a deployment folder; its contents copied to docker image
chalice package .
rm -r deployment
mkdir deployment
unzip deployment.zip -d deployment

# patch the sam.json to use image-based fields rather than zipfile-based fields
# in the api handler lambda
cat sam.json | jq '
  del(.Resources.APIHandler.Properties.Runtime) 
| del(.Resources.APIHandler.Properties.Handler) 
| del(.Resources.APIHandler.Properties.CodeUri) 
| .Resources.APIHandler.Properties += {PackageType : "Image"} 
| .Resources.APIHandler.Metadata.Dockerfile += "Dockerfile" 
| .Resources.APIHandler.Metadata.DockerContext += "." 
| .Resources.APIHandler.Metadata.DockerTag += "python3.8-v1"' > sam.img.json
sam validate -t sam.img.json

sam build -t sam.img.json
sam local start-api # verify behavior locally
sam deploy --guided

@jamesls looks like container support is about ready. What else is needed?

[gerritjvv]

@jamesls Any idea how long this feature is going to take to be released? I can't count the amount of times I had to migrate a chalice lambda to something else just because I needed a native lib or using pandas and then needs a docker image due to size. Please please please we need this feature!!.

[r0lodex]

Still a newbie with aws's sam but I'm leaning toward using it for deploying chalice containers. Obviously official support of containers from chalice itself would be preferred but until then this approach doesn't seem too bad. Here's a minimal example for deploying chalice containers via sam, which I like for its local container testing, management of ecr, patching config files rather than source code:

chalice new-project hello-world && cd hello-world

# Dockerfile for sam build
cat << EOF > ./Dockerfile
FROM public.ecr.aws/lambda/python:3.8

COPY requirements.txt ./

RUN python3.8 -m pip install -r requirements.txt -t .

COPY deployment/ ./

CMD ["app.app"]
EOF

# generate a deployment folder; its contents copied to docker image
chalice package .
rm -r deployment
mkdir deployment
unzip deployment.zip -d deployment

# patch the sam.json to use image-based fields rather than zipfile-based fields
# in the api handler lambda
cat sam.json | jq '
  del(.Resources.APIHandler.Properties.Runtime) 
| del(.Resources.APIHandler.Properties.Handler) 
| del(.Resources.APIHandler.Properties.CodeUri) 
| .Resources.APIHandler.Properties += {PackageType : "Image"} 
| .Resources.APIHandler.Metadata.Dockerfile += "Dockerfile" 
| .Resources.APIHandler.Metadata.DockerContext += "." 
| .Resources.APIHandler.Metadata.DockerTag += "python3.8-v1"' > sam.img.json
sam validate -t sam.img.json

sam build -t sam.img.json
sam local start-api # verify behavior locally
sam deploy --guided

You missed Resources.APIHandler.Properties.ImageUri

[areeves87]

@r0lodex ImageUri gets ignored if you're using the metadata fields like I am. The metadata specifies the docker file location and builds the image locally. It somehow maintains a reference to the image built and deploys it to ecr.

If you want to use imageuri to specify the image directly instead make sure to remove the metadata fields since they take precedence over imageuri.

[imanebosch]

Any updates on this issue?

Here's an update to @areeves87 POC:

cat sam.json | jq '
  del(.Resources.APIHandler.Properties.Runtime)
| del(.Resources.APIHandler.Properties.Handler)
| del(.Resources.APIHandler.Properties.CodeUri)
| .Resources.APIHandler.Properties += {PackageType : "Image"}
| .Resources.APIHandler.Metadata.Dockerfile += "Dockerfile"
| .Resources.APIHandler.Metadata.DockerContext += "."
| .Resources.APIHandler.Metadata.DockerTag += "python3.9-v1"
| .Resources.RestAPI.Properties.EndpointConfiguration = {"Type": .Resources.RestAPI.Properties.EndpointConfiguration}' > sam.img.json