Fix failing ECS task tests

[desertaxle]

Tests are failing in CI with the following error:

__________________________ test_launch_types[FARGATE] __________________________

aws_credentials = AwsCredentials(aws_access_key_id='access_key_id', aws_secret_access_key=SecretStr('**********'), aws_session_token=None, profile_name=None, region_name='us-east-1')
launch_type = 'FARGATE'

    @pytest.mark.usefixtures("ecs_mocks")
    @pytest.mark.parametrize("launch_type", ["EC2", "FARGATE", "FARGATE_SPOT"])
    async def test_launch_types(aws_credentials, launch_type: str):
        task = ECSTask(
            aws_credentials=aws_credentials,
            auto_deregister_task_definition=False,
            command=["prefect", "version"],
            launch_type=launch_type,
        )
        print(task.preview())
    
        session = aws_credentials.get_boto3_session()
        ecs_client = session.client("ecs")
    
>       task_arn = await run_then_stop_task(task)

...
self = <moto.ecs.models.Task object at 0x7fce6af6cf10>
cluster = <moto.ecs.models.Cluster object at 0x7fce65f7d190>
task_definition = <moto.ecs.models.TaskDefinition object at 0x7fce691a6e10>
container_instance_arn = 'arn:aws:ecs:us-east-1:123456789012:container-instance/default/e2f46c39-c747-4a71-af62-7426d6d1c898'
resource_requirements = {'CPU': 0, 'MEMORY': 0, 'PORTS': [], 'PORTS_UDP': []}
backend = <moto.ecs.models.EC2ContainerServiceBackend object at 0x7fce6aaa9610>
launch_type = 'FARGATE'
overrides = {'containerOverrides': [{'command': ['prefect', 'version'], 'environment': [{'name': 'PREFECT_ORION_DATABASE_CONNECTION_URL', 'value': 'sqlite+aiosqlite:////tmp/tmp9xu65h5q/orion.db'}], 'name': 'prefect'}]}
started_by = '', tags = []
networking_configuration = {'awsvpcConfiguration': {'assignPublicIp': 'ENABLED', 'subnets': ['subnet-5b085f7a', 'subnet-57e3fb65', 'subnet-544405f4', 'subnet-7be12aee', 'subnet-c47aeb59', 'subnet-0b5523d7']}}

    def __init__(
        self,
        cluster,
        task_definition,
        container_instance_arn,
        resource_requirements,
        backend,
        launch_type="",
        overrides=None,
        started_by="",
        tags=None,
        networking_configuration=None,
    ):
        self.id = str(mock_random.uuid4())
        self.cluster_name = cluster.name
        self.cluster_arn = cluster.arn
        self.container_instance_arn = container_instance_arn
        self.last_status = "RUNNING"
        self.desired_status = "RUNNING"
        self.task_definition_arn = task_definition.arn
        self.overrides = overrides or {}
        self.containers = []
        self.started_by = started_by
        self.tags = tags or []
        self.launch_type = launch_type
        self.stopped_reason = ""
        self.resource_requirements = resource_requirements
        self.region_name = cluster.region_name
        self._account_id = backend.account_id
        self._backend = backend
        self.attachments = []
    
        if task_definition.network_mode == "awsvpc":
            if not networking_configuration:
    
                raise InvalidParameterException(
                    "Network Configuration must be provided when networkMode 'awsvpc' is specified."
                )
    
            self.network_configuration = networking_configuration
            net_conf = networking_configuration["awsvpcConfiguration"]
            ec2_backend = ec2_backends[self._account_id][self.region_name]
    
            eni = ec2_backend.create_network_interface(
                subnet=net_conf["subnets"][0],
                private_ip_address=random_private_ip(),
>               group_ids=net_conf["securityGroups"],
                description="moto ECS",
            )
E           KeyError: 'securityGroups'

Originally posted by @ahuang11 in #125 (comment)

[zanieb]

Is this after a version bump or what?

[ahuang11]

Here's the differences:
{'apprise-1.0.0',
'apprise-1.1.0',
'boto3-1.24.88',
'boto3-1.24.89',
'boto3-stubs-1.24.88',
'boto3-stubs-1.24.89',
'botocore-1.27.88',
'botocore-1.27.89',
'botocore-stubs-1.27.88',
'botocore-stubs-1.27.89',
'greenlet-1.1.3',
'greenlet-1.1.3.post0',
'moto-4.0.6',
'moto-4.0.7',
'pillow-9.2.0',
'prefect-aws-0+untagged.1.gb42b1ff',
'prefect-aws-0+untagged.1.ge6e5b88',
'responses-0.21.0',
'responses-0.22.0',
'types-toml-0.10.8'}

From:

x = set("Jinja2-3.1.2 Mako-1.2.3 Markdown-3.3.7 MarkupSafe-2.1.1 aiofiles-22.1.0 aiohttp-3.8.3 aiosignal-1.2.0 aiosqlite-0.17.0 alembic-1.8.1 anyio-3.6.1 apprise-1.1.0 asgi-lifespan-1.0.1 async-timeout-4.0.2 asyncpg-0.26.0 attrs-22.1.0 black-22.10.0 boto3-1.24.89 boto3-stubs-1.24.89 botocore-1.27.89 botocore-stubs-1.27.89 cachetools-5.2.0 certifi-2022.9.24 cffi-1.15.1 cfgv-3.3.1 charset-normalizer-2.1.1 click-8.1.3 cloudpickle-2.2.0 colorama-0.4.5 commonmark-0.9.1 coolname-1.1.0 coverage-6.5.0 croniter-1.3.7 cryptography-38.0.1 distlib-0.3.6 docker-6.0.0 fastapi-0.85.0 filelock-3.8.0 flake8-5.0.4 frozenlist-1.3.1 fsspec-2022.8.2 ghp-import-2.1.0 google-auth-2.12.0 greenlet-1.1.3.post0 griffe-0.22.2 h11-0.12.0 httpcore-0.15.0 httpx-0.23.0 identify-2.5.6 idna-3.4 iniconfig-1.1.1 interrogate-1.5.0 isort-5.10.1 jmespath-1.0.1 jsonpatch-1.32 jsonpointer-2.3 kubernetes-24.2.0 mccabe-0.7.0 mergedeep-1.3.4 mkdocs-1.4.0 mkdocs-autorefs-0.4.1 mkdocs-gen-files-0.4.0 mkdocs-material-8.5.6 mkdocs-material-extensions-1.0.3 mkdocstrings-0.19.0 mkdocstrings-python-legacy-0.2.3 moto-4.0.7 multidict-6.0.2 mypy-0.982 mypy-extensions-0.4.3 nodeenv-1.7.0 oauthlib-3.2.1 orjson-3.8.0 packaging-21.3 pathspec-0.10.1 pendulum-2.1.2 pillow-9.2.0 platformdirs-2.5.2 pluggy-1.0.0 pre-commit-2.20.0 prefect-2.5.0 prefect-aws-0+untagged.1.ge6e5b88 py-1.11.0 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycodestyle-2.9.1 pycparser-2.21 pydantic-1.10.2 pyflakes-2.5.0 pygments-2.13.0 pymdown-extensions-9.6 pyparsing-3.0.9 pytest-7.1.3 pytest-asyncio-0.19.0 pytest-lazy-fixture-0.6.3 python-dateutil-2.8.2 python-slugify-6.1.2 pytkdocs-0.16.1 pytz-2022.4 pytzdata-2020.1 pyyaml-6.0 pyyaml-env-tag-0.1 readchar-4.0.3 requests-2.28.1 requests-oauthlib-1.3.1 responses-0.22.0 rfc3986-1.5.0 rich-12.6.0 rsa-4.9 s3transfer-0.6.0 setuptools-65.4.1 six-1.16.0 sniffio-1.3.0 sqlalchemy-1.4.41 starlette-0.20.4 tabulate-0.9.0 text-unidecode-1.3 toml-0.10.2 tomli-2.0.1 typer-0.6.1 types-awscrt-0.14.7 types-boto3-1.0.2 types-s3transfer-0.6.0.post4 types-toml-0.10.8 typing-extensions-4.4.0 urllib3-1.26.12 uvicorn-0.18.3 virtualenv-20.16.5 watchdog-2.1.9 websocket-client-1.4.1 werkzeug-2.1.2 xmltodict-0.13.0 yarl-1.8.1".split())
y = set("Jinja2-3.1.2 Mako-1.2.3 Markdown-3.3.7 MarkupSafe-2.1.1 aiofiles-22.1.0 aiohttp-3.8.3 aiosignal-1.2.0 aiosqlite-0.17.0 alembic-1.8.1 anyio-3.6.1 apprise-1.0.0 asgi-lifespan-1.0.1 async-timeout-4.0.2 asyncpg-0.26.0 attrs-22.1.0 black-22.10.0 boto3-1.24.88 boto3-stubs-1.24.88 botocore-1.27.88 botocore-stubs-1.27.88 cachetools-5.2.0 certifi-2022.9.24 cffi-1.15.1 cfgv-3.3.1 charset-normalizer-2.1.1 click-8.1.3 cloudpickle-2.2.0 colorama-0.4.5 commonmark-0.9.1 coolname-1.1.0 coverage-6.5.0 croniter-1.3.7 cryptography-38.0.1 distlib-0.3.6 docker-6.0.0 fastapi-0.85.0 filelock-3.8.0 flake8-5.0.4 frozenlist-1.3.1 fsspec-2022.8.2 ghp-import-2.1.0 google-auth-2.12.0 greenlet-1.1.3 griffe-0.22.2 h11-0.12.0 httpcore-0.15.0 httpx-0.23.0 identify-2.5.6 idna-3.4 iniconfig-1.1.1 interrogate-1.5.0 isort-5.10.1 jmespath-1.0.1 jsonpatch-1.32 jsonpointer-2.3 kubernetes-24.2.0 mccabe-0.7.0 mergedeep-1.3.4 mkdocs-1.4.0 mkdocs-autorefs-0.4.1 mkdocs-gen-files-0.4.0 mkdocs-material-8.5.6 mkdocs-material-extensions-1.0.3 mkdocstrings-0.19.0 mkdocstrings-python-legacy-0.2.3 moto-4.0.6 multidict-6.0.2 mypy-0.982 mypy-extensions-0.4.3 nodeenv-1.7.0 oauthlib-3.2.1 orjson-3.8.0 packaging-21.3 pathspec-0.10.1 pendulum-2.1.2 platformdirs-2.5.2 pluggy-1.0.0 pre-commit-2.20.0 prefect-2.5.0 prefect-aws-0+untagged.1.gb42b1ff py-1.11.0 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycodestyle-2.9.1 pycparser-2.21 pydantic-1.10.2 pyflakes-2.5.0 pygments-2.13.0 pymdown-extensions-9.6 pyparsing-3.0.9 pytest-7.1.3 pytest-asyncio-0.19.0 pytest-lazy-fixture-0.6.3 python-dateutil-2.8.2 python-slugify-6.1.2 pytkdocs-0.16.1 pytz-2022.4 pytzdata-2020.1 pyyaml-6.0 pyyaml-env-tag-0.1 readchar-4.0.3 requests-2.28.1 requests-oauthlib-1.3.1 responses-0.21.0 rfc3986-1.5.0 rich-12.6.0 rsa-4.9 s3transfer-0.6.0 setuptools-65.4.1 six-1.16.0 sniffio-1.3.0 sqlalchemy-1.4.41 starlette-0.20.4 tabulate-0.9.0 text-unidecode-1.3 toml-0.10.2 tomli-2.0.1 typer-0.6.1 types-awscrt-0.14.7 types-boto3-1.0.2 types-s3transfer-0.6.0.post4 typing-extensions-4.4.0 urllib3-1.26.12 uvicorn-0.18.3 virtualenv-20.16.5 watchdog-2.1.9 websocket-client-1.4.1 werkzeug-2.1.2 xmltodict-0.13.0 yarl-1.8.1".split())
x ^ y

[ahuang11]

Locally, I tested using moto 4.0.7 and 4.0.6. 4.0.7 breaks the test while 4.0.6 does not. Not sure if we want to pin the version or do group_ids=net_conf.get("securityGroups", [])

[desertaxle]

I'd say go with .get

[ahuang11]

Actually, this isn't our code; it's from moto models. https://github.com/rafcio19/moto/blob/05738df4cfa886e256fe91e0732502e759e52690/moto/ecs/models.py#L326-L359

[zanieb]

Let's pin and open an issue upstream? (unless we are misusing)

[ahuang11]

This was committed 11 days ago: https://github.com/spulec/moto/pull/5520/files

Would you like to open an issue upstream? I don't think I understand this enough to report this.

[desertaxle]

It looks like we don't add any security groups to the awsvpcConfiguration that we generate in _load_vpc_network_config which might be the root of this issue. Should we attempt to infer security groups like we do with subnets?

[zanieb]

Apparently security groups are distinct and we can't infer them from the VPC — I asked the same question in #112 (see also #115) :)

[zanieb]

Separately, you don't need to pass security groups to use ECS in this mode so this moto code is just incorrect.