"Network Configuration is not valid for the given networkMode of this task definition" error when using the EcsRunLauncher

[dagsir[bot]]

Hello folks. 👋

I'm trying to move from DefaultRunLauncher to EcsRunLauncher. Deamon, Ui, grpc's are running on ECS.
I'm stucked with the below error when trying to launch a run:
botocore.errorfactory.InvalidParameterException: An error occurred (InvalidParameterException) when calling the RunTask operation: Network Configuration is not valid for the given networkMode of this task definition.

I added a env var to indicate each grpc server which task definition to use and not use the deamon task def:
DAGSTER_CONTAINER_CONTEXT = "{"ecs":{"task_definition_arn":"task_def_arn", "container_name":"name"}}"

Even tough we are working with a vpc, my task definitions have no network mode indicated and are all set to default. According to https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ecs/client/run_task.html|aws doc, the networkconfiguration should be provided only for task definition with network mode: "awsvpc".
Is anyone facing this issue or already resolved it ?
dagster_version: 1.2.7

Thanks for your help, cheers!

---

The question was originally asked in Dagster Slack.

[gibsondan]

The EcsRunLauncher will pull certain fields to pass into the boto3 run_task call using the ECS task that is launching the run. You can see how it sets the default networkConfiguration values here: https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-aws/dagster_aws/ecs/tasks.py#L386-L392

It seems like that default networkConfiguration is incompatible with your task definition. You can set the "run_task_kwargs" key on your run launcher in your dagster.yaml to override the networkConfiguration value to something different, following the example here: https://docs.dagster.io/deployment/guides/aws#customizing-the-launched-runs-task

[alexis-smartway]

Thanks @gibsondan I managed to make it work by upgrading service and task definition network mode to awsvpc. The task is correctly launched but I don't get any feedback from the task and dagster stays at "Starting" status. I'll check logs and figure it out. It might be a network issue. Thanks for your help, cheers

[whillas-yabble]

what exactly did you set it to? I'm getting the same error.

[alexis-smartway]

@whillas-yabble I took some time to investigate.
Finally i figured it out. The issue wasn't a network issue. The task definition that I used had a entrypoint defined. When dagster launch the new task it concatenates the entrypoint of the task tefinition with a command "dagster api execute_run".
Removing the entrypoint fixed the issue (i created a specific task definition for launcher and put the task_definition_arn in DAGSTER_CONTEXT env.
Below you can fin the dagster.yaml extract for run launcher:

run_launcher:
  module: dagster_aws.ecs
  class: EcsRunLauncher
  config:
    include_sidecars: True
    use_current_ecs_task_config: False
    run_task_kwargs:
      cluster: cluster_arn
      capacityProviderStrategy: 
        - capacityProvider: capacity_provider

Do not hesitate if you any further question,
Cheers

[whillas-yabble]

@gibsondan so maybe unrelated but I'm looking for a way to define a different task definition from that of the user code definition and the main daemon/webserver, which I'm inferring from above i can just set the env. var. DAGSTER_CONTAINER_CONTEXT = "{"ecs":{"task_definition_arn":"some_task_def_arn"}}" on the user container and it will be picked up when jobs are run? (which is totally undocumented as far as I can tell, only defining it in the dagster.yaml and copying the user code container, right?)

[gibsondan]

There's a discussion that somewhat covers that here: #12621 - agree that it is under-documented.

[whillas-yabble]

Looks like its not getting deserialized from a string to a dict? Getting this error?

dagster._check.ParameterCheckError: Param "container_context" is not a dict. Got '{"ecs":{"task_definition_arn": "arn:aws:ecs:ap-southeast-2:554694190286:task-definition/devau-yabble-dagster-scribe-pipeline-worker-task:1", "container_name": "devau-yabble-dagster-scribe-pipeline-worker-container"}}' which is type <class 'str'>.

[gibsondan]

Can you share the exact repro steps you're taking / a full stack trace of the error that you're seeing? It's deserialized into a dictionary here: https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_cli/api.py#L675

I get an error like that if I set the env var to a JSON string, e.g.

DAGSTER_CONTAINER_CONTEXT='"FOO"' dagster api grpc -p 4000 -m dagster_test.toys.repo

but setting it to a JSON dictionary works fine for me

 DAGSTER_CONTAINER_CONTEXT='{"ecs":{"task_definition_arn":"some_task_def_arn"}}' dagster api grpc -p 4000 -m dagster_test.toys.repo

[whillas-yabble]

sorry, my bad. just a parsing error