'Unknown Parameter' Error in ECS when a Task Definition is Registered

[anthony-desmier-axomic]

When running a deployment that uses an ECSTask block the Prefect agent is able to successfully retrieve the associated task definition, however upon registering the task definition using the returned json object a ParamValidationError is thrown from the underlying botocore library for the following fields in the task definition json: requiresAttributes, registeredAt, registeredBy.

Stack trace from the Prefect 2 agent:

13/10/2022, 09:33:19 | 08:33:19.822 \| INFO \| prefect.agent - Completed submission of flow run 'ad1f930d-875a-4878-99ed-36641a16f788' | prefect-2-agent
13/10/2022, 09:33:19 | 08:33:19.821 \| ERROR \| prefect.agent - Failed to submit flow run 'ad1f930d-875a-4878-99ed-36641a16f788' to infrastructure. | prefect-2-agent
13/10/2022, 09:33:19 | Traceback (most recent call last): | prefect-2-agent
13/10/2022, 09:33:19 | File "/usr/local/lib/python3.9/site-packages/prefect/agent.py", line 233, in _submit_run_and_capture_errors | prefect-2-agent
13/10/2022, 09:33:19 | result = await infrastructure.run(task_status=task_status) | prefect-2-agent
13/10/2022, 09:33:19 | File "/usr/local/lib/python3.9/site-packages/prefect_aws/ecs.py", line 362, in run | prefect-2-agent
13/10/2022, 09:33:19 | ) = await run_sync_in_worker_thread( | prefect-2-agent
13/10/2022, 09:33:19 | File "/usr/local/lib/python3.9/site-packages/prefect/utilities/asyncutils.py", line 57, in run_sync_in_worker_thread | prefect-2-agent
13/10/2022, 09:33:19 | return await anyio.to_thread.run_sync(call, cancellable=True) | prefect-2-agent
13/10/2022, 09:33:19 | File "/usr/local/lib/python3.9/site-packages/anyio/to_thread.py", line 31, in run_sync | prefect-2-agent
13/10/2022, 09:33:19 | return await get_asynclib().run_sync_in_worker_thread( | prefect-2-agent
13/10/2022, 09:33:19 | File "/usr/local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread | prefect-2-agent
13/10/2022, 09:33:19 | return await future | prefect-2-agent
13/10/2022, 09:33:19 | File "/usr/local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 867, in run | prefect-2-agent
13/10/2022, 09:33:19 | result = context.run(func, *args) | prefect-2-agent
13/10/2022, 09:33:19 | File "/usr/local/lib/python3.9/site-packages/prefect_aws/ecs.py", line 441, in _create_task_and_wait_for_start | prefect-2-agent
13/10/2022, 09:33:19 | task_definition_arn = self._register_task_definition( | prefect-2-agent
13/10/2022, 09:33:19 | File "/usr/local/lib/python3.9/site-packages/prefect_aws/ecs.py", line 818, in _register_task_definition | prefect-2-agent
13/10/2022, 09:33:19 | response = ecs_client.register_task_definition(**task_definition_request) | prefect-2-agent
13/10/2022, 09:33:19 | File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 508, in _api_call | prefect-2-agent
13/10/2022, 09:33:19 | return self._make_api_call(operation_name, kwargs) | prefect-2-agent
13/10/2022, 09:33:19 | File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 878, in _make_api_call | prefect-2-agent
13/10/2022, 09:33:19 | request_dict = self._convert_to_request_dict( | prefect-2-agent
13/10/2022, 09:33:19 | File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 939, in _convert_to_request_dict | prefect-2-agent
13/10/2022, 09:33:19 | request_dict = self._serializer.serialize_to_request( | prefect-2-agent
13/10/2022, 09:33:19 | File "/usr/local/lib/python3.9/site-packages/botocore/validate.py", line 381, in serialize_to_request | prefect-2-agent
13/10/2022, 09:33:19 | raise ParamValidationError(report=report.generate_report()) | prefect-2-agent
13/10/2022, 09:33:19 | botocore.exceptions.ParamValidationError: Parameter validation failed: | prefect-2-agent
13/10/2022, 09:33:19 | Unknown parameter in input: "requiresAttributes", must be one of: family, taskRoleArn, executionRoleArn, networkMode, containerDefinitions, volumes, placementConstraints, requiresCompatibilities, cpu, memory, tags, pidMode, ipcMode, proxyConfiguration, inferenceAccelerators, ephemeralStorage, runtimePlatform | prefect-2-agent
13/10/2022, 09:33:19 | Unknown parameter in input: "registeredAt", must be one of: family, taskRoleArn, executionRoleArn, networkMode, containerDefinitions, volumes, placementConstraints, requiresCompatibilities, cpu, memory, tags, pidMode, ipcMode, proxyConfiguration, inferenceAccelerators, ephemeralStorage, runtimePlatform | prefect-2-agent
13/10/2022, 09:33:19 | Unknown parameter in input: "registeredBy", must be one of: family, taskRoleArn, executionRoleArn, networkMode, containerDefinitions, volumes, placementConstraints, requiresCompatibilities, cpu, memory, tags, pidMode, ipcMode, proxyConfiguration, inferenceAccelerators, ephemeralStorage, runtimePlatform | prefect-2-agent
13/10/2022, 09:33:19 | 08:33:19.819 \| DEBUG \| prefect.infrastructure.ecs-task - Task definition payload | prefect-2-agent
...
13/10/2022, 09:33:19 | 08:33:19.813 \| INFO \| prefect.infrastructure.ecs-task - ECSTask 'cocky-bulldog': Registering task definition... | prefect-2-agent
13/10/2022, 09:33:19 | 08:33:19.765 \| INFO \| prefect.infrastructure.ecs-task - ECSTask 'cocky-bulldog': Retrieving task definition 'manual_hello_world_test'... | prefect-2-agent

It appears to be an issue on the AWS SDK side where some fields in the response from describe-task-definition are not compatible in a call to register-task-definition (aws/aws-sdk#38, aws/aws-sdk#406). The current workaround appears to be removing these fields when registering the task.

Steps to Reproduce:
We are currently defining our ECS task definitions within terraform, however I have manually created a new ECS task from within the ECS GUI. Viewing the Overview for this task in the GUI and clicking on the JSON tab you can see that the requiresAttributes, registeredAt and registeredBy are all defined.

Our ECSTask block is defined with the name of our ECS cluster and we use what AWS call the 'Task definition family' for the 'task_definition_arn' argument which is correctly translated to the full arn.

There doesn't appear to a fix coming from the AWS SDK side, so this might need to be something that is handled on the Prefect side.

[ahuang11]

If I'm not mistaken, maybe the invalid keys can be dropped here:
https://github.com/PrefectHQ/prefect-aws/blob/main/prefect_aws/ecs.py#L900-L918

Would you like to provide a PR to fix this?

[anthony-desmier-axomic]

If I'm not mistaken, maybe the invalid keys can be dropped here: https://github.com/PrefectHQ/prefect-aws/blob/main/prefect_aws/ecs.py#L900-L918

Would you like to provide a PR to fix this?

Sure, happy to add a PR for this 👍