Setting @mock_aws(config={"lambda": {"use_docker": False}}) on a pytest test causes Lambdas to disappear

[andymadge]

I'm trying to use Moto and pytest to test a Python script which invokes an AWS Lambda Function.

In my testing, script_to_test.py won't complete successfully because it tries to invoke a Lambda Function. I'm using Moto to create a simple dummy Lambda Function that the main script can invoke. As long as that returns status:200 then script_to_test.py should complete successfully, so the "Simple Lambda happy path OK" response will be ok.

(script_to_test.py is in fact itself a Lambda Function, but here I'm testing by simply importing it and calling a function from it.)

I have Moto working fine for other tests in the suite, but I can't get it to work for this one.

I can create create the dummy Lambda function in a pytest fixture, and I can do list_functions() to confirm it exists at the end of the fixture.

However, when I then include that fixture in a test, the function is not available in the test.

I'm using:

• python v3.12
• pytest v8.1.1
• moto v5.0.3
• boto3 v1.34.65

This is my cut-down test code which illustrates the issue:

import zipfile
import io
import pytest
import os
import boto3
from botocore.exceptions import ClientError

from moto import mock_aws

LAMBDA_REGION = "eu-west-2"


@pytest.fixture(scope='function')
def aws_credentials():
    os.environ['AWS_ACCESS_KEY_ID'] = 'fake-access-key'
    os.environ['AWS_SECRET_ACCESS_KEY'] = 'fake-secret-key'
    os.environ['AWS_SECURITY_TOKEN'] = 'fake-security-token'
    os.environ['AWS_SESSION_TOKEN'] = 'fake-session-token'


@pytest.fixture
def dummy_lambda(aws_credentials):
    with mock_aws():
        client = boto3.client("lambda", LAMBDA_REGION)
        zip_content = get_test_zip_file1()
        role = get_role_name()
        client.create_function(
            FunctionName="LambdaFunctionDummyName",
            Runtime="3.12",
            Role=role,
            Handler="lambda_function.lambda_handler",
            Code={"ZipFile": zip_content},
            Description="test lambda function",
            Timeout=3,
            MemorySize=128,
            Publish=True,
        )
        # print(client.list_functions()['Functions'], "\n")
        print(boto3.client('lambda').list_functions()['Functions'], "\n")
        yield client


@mock_aws(config={"lambda": {"use_docker": False},})
def test_lambda(dummy_lambda):
    # print(dummy_lambda.list_functions()['Functions'], "\n")
    print(boto3.client('lambda').list_functions()['Functions'], "\n")
    # print(boto3.client('sts').get_caller_identity())
    assert True



### Utilities ###

def get_test_zip_file1():
    pfunc = """
def lambda_handler(event, context):
    print("custom log event")
    return event
"""
    zip_output = io.BytesIO()
    zip_file = zipfile.ZipFile(zip_output, "w", zipfile.ZIP_DEFLATED)
    zip_file.writestr("lambda_function.py", pfunc)
    zip_file.close()
    zip_output.seek(0)
    return zip_output.read()


def get_role_name():
    '''Create a role for the lambda function to assume and return the ARN. If the role already exists, return the ARN.'''
    with mock_aws():
        iam = boto3.client("iam", region_name=LAMBDA_REGION)
        while True:
            try:
                return iam.get_role(RoleName="my-role")["Role"]["Arn"]
            except ClientError:
                try:
                    return iam.create_role(
                        RoleName="my-role",
                        AssumeRolePolicyDocument="some policy",
                        Path="/my-path/",
                    )["Role"]["Arn"]
                except ClientError:
                    pass

I'm running it with pytest -rP to see print output.

This is the output of the test:

pytest -rP
============================================================ test session starts =============================================================
platform darwin -- Python 3.12.2, pytest-8.1.1, pluggy-1.4.0
rootdir: /Users/andym/Dropbox/_home/_Dev/aws-resource-scheduler/modules/_python/testing123
plugins: socket-0.7.0
collected 1 item                                                                                                                             

test_lambda.py .                                                                                                                       [100%]

============================================================== warnings summary ==============================================================
test_lambda.py::test_lambda
test_lambda.py::test_lambda
test_lambda.py::test_lambda
test_lambda.py::test_lambda
test_lambda.py::test_lambda
  /Users/andym/.local/share/virtualenvs/_python-7ct6g_05/lib/python3.12/site-packages/botocore/auth.py:419: DeprecationWarning: datetime.datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.now(datetime.UTC).
    datetime_now = datetime.datetime.utcnow()

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=================================================================== PASSES ===================================================================
________________________________________________________________ test_lambda _________________________________________________________________
----------------------------------------------------------- Captured stdout setup ------------------------------------------------------------
[{'FunctionName': 'LambdaFunctionDummyName', 'FunctionArn': 'arn:aws:lambda:eu-west-2:123456789012:function:LambdaFunctionDummyName', 'Runtime': '3.12', 'Role': 'arn:aws:iam::123456789012:role/my-path/my-role', 'Handler': 'lambda_function.lambda_handler', 'CodeSize': 208, 'Description': 'test lambda function', 'Timeout': 3, 'MemorySize': 128, 'LastModified': '2024-03-21T18:51:16.377378000Z', 'CodeSha256': 'K4LH8osVR22xLvTSlPkxoj7CKPf/kx8BQPBJtTAHiG0=', 'Version': '$LATEST', 'VpcConfig': {'SubnetIds': [], 'SecurityGroupIds': []}, 'TracingConfig': {'Mode': 'PassThrough'}, 'Layers': [], 'State': 'Active', 'LastUpdateStatus': 'Successful', 'PackageType': 'Zip', 'Architectures': ['x86_64'], 'EphemeralStorage': {'Size': 512}, 'SnapStart': {'ApplyOn': 'None', 'OptimizationStatus': 'Off'}}] 

------------------------------------------------------------ Captured stdout call ------------------------------------------------------------
[] 

======================================================= 1 passed, 5 warnings in 0.33s ========================================================

You can see the "Captured stdout setup" shows that print(boto3.client('lambda').list_functions()['Functions'], "\n") does list the Lambda function, however the same line in "Captured stdout call" just shows an empty list.

If I comment out the line @mock_aws(config={"lambda": {"use_docker": False},}) then it works and the function is visible in the test, however this doesn't help me since I want to avoid running the lambda in Docker

pytest -rP
============================================================ test session starts =============================================================
platform darwin -- Python 3.12.2, pytest-8.1.1, pluggy-1.4.0
rootdir: /Users/andym/Dropbox/_home/_Dev/aws-resource-scheduler/modules/_python/testing123
plugins: socket-0.7.0
collected 1 item                                                                                                                             

test_lambda.py .                                                                                                                       [100%]

============================================================== warnings summary ==============================================================
test_lambda.py::test_lambda
test_lambda.py::test_lambda
test_lambda.py::test_lambda
test_lambda.py::test_lambda
test_lambda.py::test_lambda
  /Users/andym/.local/share/virtualenvs/_python-7ct6g_05/lib/python3.12/site-packages/botocore/auth.py:419: DeprecationWarning: datetime.datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.now(datetime.UTC).
    datetime_now = datetime.datetime.utcnow()

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=================================================================== PASSES ===================================================================
________________________________________________________________ test_lambda _________________________________________________________________
----------------------------------------------------------- Captured stdout setup ------------------------------------------------------------
[{'FunctionName': 'LambdaFunctionDummyName', 'FunctionArn': 'arn:aws:lambda:eu-west-2:123456789012:function:LambdaFunctionDummyName', 'Runtime': '3.12', 'Role': 'arn:aws:iam::123456789012:role/my-path/my-role', 'Handler': 'lambda_function.lambda_handler', 'CodeSize': 208, 'Description': 'test lambda function', 'Timeout': 3, 'MemorySize': 128, 'LastModified': '2024-03-21T18:52:14.759583000Z', 'CodeSha256': '0E+b62YkMYiIbKE4h/D2otRUbi8ukeYwGVrUlpjmToo=', 'Version': '$LATEST', 'VpcConfig': {'SubnetIds': [], 'SecurityGroupIds': []}, 'TracingConfig': {'Mode': 'PassThrough'}, 'Layers': [], 'State': 'Active', 'LastUpdateStatus': 'Successful', 'PackageType': 'Zip', 'Architectures': ['x86_64'], 'EphemeralStorage': {'Size': 512}, 'SnapStart': {'ApplyOn': 'None', 'OptimizationStatus': 'Off'}}] 

------------------------------------------------------------ Captured stdout call ------------------------------------------------------------
[{'FunctionName': 'LambdaFunctionDummyName', 'FunctionArn': 'arn:aws:lambda:eu-west-2:123456789012:function:LambdaFunctionDummyName', 'Runtime': '3.12', 'Role': 'arn:aws:iam::123456789012:role/my-path/my-role', 'Handler': 'lambda_function.lambda_handler', 'CodeSize': 208, 'Description': 'test lambda function', 'Timeout': 3, 'MemorySize': 128, 'LastModified': '2024-03-21T18:52:14.759583000Z', 'CodeSha256': '0E+b62YkMYiIbKE4h/D2otRUbi8ukeYwGVrUlpjmToo=', 'Version': '$LATEST', 'VpcConfig': {'SubnetIds': [], 'SecurityGroupIds': []}, 'TracingConfig': {'Mode': 'PassThrough'}, 'Layers': [], 'State': 'Active', 'LastUpdateStatus': 'Successful', 'PackageType': 'Zip', 'Architectures': ['x86_64'], 'EphemeralStorage': {'Size': 512}, 'SnapStart': {'ApplyOn': 'None', 'OptimizationStatus': 'Off'}}] 

======================================================= 1 passed, 5 warnings in 0.35s ========================================================

Now we see the print(boto3.client('lambda').list_functions()['Functions'], "\n") works correctly in both places.

What am I doing wrong here? Or is it a bug?

[bblommers]

Hi @andymadge! There are two things at play here:

• The mock_aws-decorator overwrites the config everytime it starts.
• Functions created in a mock without docker are stored separately from functions in a mock where docker is enabled

Your example effectively does the following:

# The fixture is executed first by pytest, starting a mock with docker enabled (lambda: {use_docker: True} is the default)
mock_aws(config=None)

# Create function in the lambda with Docker enabled
docker_lambda.create_function()
docker_lambda.list_function()

# Now the test-decorator starts
# Start a Mock without docker
@mock_aws(config={"lambda": {"use_docker": False},})

# We now list the functions inside the docker-less lambda
dockerless_lambda.list_functions == []

The easiest way to fix this would be to only have a single decorator inside your test case, so it's always obvious which configuration is active at any given point. One way to do this is to have a single fixture:

@pytest.fixture(scope='function')
def aws_credentials():
    ...


@pytest.fixture
def dockerless_fake_aws(aws_credentials):
    with mock_aws(config={"lambda": {"use_docker": False}}):
        yield

@pytest.fixture
def dummy_lambda(dockerless_fake_aws):
    client = boto3.client("lambda", LAMBDA_REGION)
    zip_content = get_test_zip_file1()
    role = get_role_name()
    client.create_function(
        FunctionName="LambdaFunctionDummyName",
        Runtime="3.12",
        Role=role,
        Handler="lambda_function.lambda_handler",
        Code={"ZipFile": zip_content},
        Description="test lambda function",
        Timeout=3,
        MemorySize=128,
        Publish=True,
    )
    # print(client.list_functions()['Functions'], "\n")
    print(boto3.client('lambda', LAMBDA_REGION).list_functions()['Functions'], "\n")
    yield client



def test_lambda(dummy_lambda):
    # print(dummy_lambda.list_functions()['Functions'], "\n")
    print(boto3.client('lambda', LAMBDA_REGION).list_functions()['Functions'], "\n")
    # print(boto3.client('sts').get_caller_identity())
    assert True



### Utilities ###

def get_test_zip_file1():
    ...


def get_role_name():
    '''Create a role for the lambda function to assume and return the ARN. If the role already exists, return the ARN.'''
    iam = boto3.client("iam", region_name=LAMBDA_REGION)
    while True:
        try:
            return iam.get_role(RoleName="my-role")["Role"]["Arn"]
        except ClientError:
            try:
                return iam.create_role(
                    RoleName="my-role",
                    AssumeRolePolicyDocument="some policy",
                    Path="/my-path/",
                )["Role"]["Arn"]
            except ClientError:
                pass

Note that I've also removed the decorator from the get_role_name - as long as this function is always executed within the context of another function with mock_aws active, there's no reason to apply it again.

I also added the LAMBDA_REGION explicitly to every boto3.client-initialization, just to ensure that they are not executed within my system-default region.

[andymadge]

Thanks, that makes sense. I had wondered if it needed configuring earlier, but the only way I could see to configure it differently was by adding the decorator to fixtures, but that didn't seem to do anything.

I can see now that I needed to add the config to the original call to mock_aws() rather than as a decorator.

So just to check my understanding...

• If you want to configure mock_aws(), then you need to do it when it is instantiated (obvious really, but I just wasn't thinking of it in those terms)
• Therefore if you are instantiating it manually, with mock = mock_aws() or a context manager, then you include the config in the call to mock_aws()
• If you use the decorator it will implicitly create the mock, so you include the config in the decorator

These creation options are covered in https://docs.getmoto.org/en/latest/docs/getting_started.html#moto-usage but I had failed to grasp the fact that this is also the place to configure it.

That all seems completely obvious now that I know. Of course you need to configure the object instance when you create it, but my understanding of Moto was lacking and I wasn't piecing together the various ways of instantiating Moto to understand each of them.

Thanks for taking the time to explain.

[bblommers]

No problem @andymadge! Glad I could help. I'll close this, but let us know if you have any other questions.