`ECSRun` memory and CPU settings are not set in the task definition

[marvin-robot]

CPU and Memory set on an ECSRun do not update the task definition, but instead passed as overrides to the specific task run. The AWS console does not display these settings unless you look at the task JSON which can lead to confusion, but CPU and memory are passed correctly to your task run. Since you can define your own task definition we must pass things like this as overrides per run rather than on the task definition. You can use aws ecs describe-tasks to take a look at your task and see that the settings are correct, see https://docs.aws.amazon.com/cli/latest/reference/ecs/describe-tasks.html for details on usage.

Archived from the Prefect Public Slack Community

nikola.lusic: Hey guys, so I'm setting up the Prefect environment that will be running tasks on AWS ECS via ECSRun configuration. Everything is working except for one parameter: setting the cpu and memory parameters of the ECSRun doesn't seem to have any effect. It seems that the agent always creates a task with 1CPU and 2GB RAM.
Do you maybe have some pointers for this issue?

kevin701: Hi <@U020BC7KNQ5>, could you show how you’re setting them?

nikola.lusic:

run_config = ECSRun(
    image=ec.get("DOCKER_IMAGE"),
    env=env,
    cpu=512,
    memory=2048)
storage = S3(bucket=ec.get("S3_BUCKET"))
result = S3Result(bucket=ec.get("S3_BUCKET"))
with Flow(
        name="flow_name",
        run_config=run_config,
        storage=storage,
        result=result
) as flow:
    flow_steps()

nikola.lusic: As I've mentioned, the flow works, and the logic is being executed, but the ECS task CPU and memory are always 1vcpu/2GB RAM, no matter what parameters I pass to ECSRun.

kevin701: Have you tried something like?

flow.run_config = ECSRun(cpu="2 vcpu", memory="4 GB")

kevin701: Actually what you have should work. Let me take a look some more

nikola.lusic: I have tried both int values and the string "2 vcpu"/"4 GB"

kevin701: You’re not supplying a task definition right?

nikola.lusic: nope

kevin701: I’m digging through the agent code.

nikola.lusic: I appreciate it. I'm combing through the docs, but there are not too many mentions of the CPU/RAM parameters for ECS tasks.

nikola.lusic: <@U01QEJ9PP53> the agent that I've deployed has no special configuration, other than the connection to Prefect Sever.

kevin701: Yeah I’ll get back to this in a bit. Just have a meeting first.

nikola.lusic: Just an update on my testing:
If I define illegal cpu/ram arguments for an ECS run (for example, 256/256), the ECS task fails, altough, on ECS task definition it's again defined as 1CPU/2GB RAM task.

kevin701: What version of Prefect are you on?

nikola.lusic: 0.14.17

michael054: Hey <@U020BC7KNQ5> -- just to clarify, are you trying to set the CPU and memory for the flow container in your ECS task?

nikola.lusic: Indeed. We have tasks that require variable amounts of resources, but no matter when parameters of cpu and memory I set in the ECSRun, the result is always an ECS task that has 1 CPU and 2 GB RAM.

michael054: It looks like it can be set at the container level and at the task definition level and we're only overriding the task definition value

michael054: Would you be able to share a redacted task definition from one of your runs?

nikola.lusic: This is the task definition created by Prefect that is running the ECS container:

{
  "ipcMode": null,
  "executionRoleArn": "<AWS ROLE>",
  "containerDefinitions": [
    {
      "dnsSearchDomains": null,
      "environmentFiles": null,
      "logConfiguration": null,
      "entryPoint": null,
      "portMappings": [],
      "command": null,
      "linuxParameters": null,
      "cpu": 0,
      "environment": [
        {
          "name": "PREFECT__CONTEXT__IMAGE",
          "value": "<DOCKER IMAGE>""
        }
      ],
      "resourceRequirements": null,
      "ulimits": null,
      "dnsServers": null,
      "mountPoints": [],
      "workingDirectory": null,
      "secrets": null,
      "dockerSecurityOptions": null,
      "memory": null,
      "memoryReservation": null,
      "volumesFrom": [],
      "stopTimeout": null,
      "image": "<DOCKER IMAGE>",
      "startTimeout": null,
      "firelensConfiguration": null,
      "dependsOn": null,
      "disableNetworking": null,
      "interactive": null,
      "healthCheck": null,
      "essential": true,
      "links": null,
      "hostname": null,
      "extraHosts": null,
      "pseudoTerminal": null,
      "user": null,
      "readonlyRootFilesystem": null,
      "dockerLabels": null,
      "systemControls": null,
      "privileged": null,
      "name": "flow"
    }
  ],
  "placementConstraints": [],
  "memory": "2048",
  "taskRoleArn": null,
  "compatibilities": [
    "EC2",
    "FARGATE"
  ],
  "taskDefinitionArn": "<TASK DEFINITION ARN>",
  "family": "prefect-opal-copy",
  "requiresAttributes": [
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.ecr-auth"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "ecs.capability.execution-role-ecr-pull"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.docker-remote-api.1.18"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "ecs.capability.task-eni"
    }
  ],
  "pidMode": null,
  "requiresCompatibilities": [
    "FARGATE"
  ],
  "networkMode": "awsvpc",
  "cpu": "1024",
  "revision": 22,
  "status": "INACTIVE",
  "inferenceAccelerators": null,
  "proxyConfiguration": null,
  "volumes": [],
  "statusString": "(INACTIVE)"
}

nikola.lusic: This task definition is the product of the following ECSRun configuration:

ECSRun(labels=["<LABEL>"],
       image=ec.get("DOCKER_IMAGE"),
       env=env,
       cpu=2048,
       memory=4096)

michael054: So I'm not entirely sure how this is reflected in AWS -- but we are passing the CPU and memory as "Overrides" for the specific task run, not in the task definition

michael054: ie we are using https://boto3.amazonaws.com/v1/documentation/api/1.9.88/reference/services/ecs.html#ECS.Client.run_task with an overrides dictionary

michael054: Can you confirm that your actual task run does not have the correct CPU/memory allocated?

nikola.lusic: I can check, although I am not sure that a ECS container could have more resources allocated to it that it's task definition specifies.

michael054: We are
• Registering a task definition
• Creating a task with overrides for some fields of the task definition

michael054: The CPU/memory overrides are for the task level resources, not the container resources

nikola.lusic: Alright. I'm not sure how to check a Fargate task allocated resources (I can't SSH into the container), and the AWS doesn't offer obvious source of this information. Do you maybe know how to check this?

michael054: I'll ping someone who's worked with ECS recently on our team.

michael054: Hey <@U020BC7KNQ5> -- we have confirmed that the CPU and memory are passed correctly to your task run. Since you define your own task definition we must pass things like this as overrides per run rather than on the task definition. You can use aws ecs describe-tasks to take a look at your task and see that the settings are correct https://docs.aws.amazon.com/cli/latest/reference/ecs/describe-tasks.html

michael054: <@ULVA73B9P> archive "ECSRun memory and CPU settings are not set in the task definition"

Original thread can be found here.