Internal error when running ExecuteCommand

[WhyNotHugo]

I almost always get this error when attempting to use ExecuteCommand:

An error occurred (TargetNotConnectedException) when calling the ExecuteCommand operation: The execute command failed due to an internal error. Try again later.

It would seem that it does not occur when attempting to ExecuteCommand on containers with a read-write root filesystem. Is is possible that it always fails with a read-only root?

$ aws --version
aws-cli/1.19.33 Python/3.9.2 Linux/5.11.11-arch1-1 botocore/1.20.33

Let me know if there's any additional information that might be useful.

[kdaily]

Hi @WhyNotHugo, sorry to hear you're having trouble. To best determine what the issue could be, please use our template when opening up a new issue, as it contains the information that could be useful. I see you've provided the architecture and CLI version, but having the full command and debug logs will be needed to assist you further. Thanks!

Confirm by changing [ ] to [x] below to ensure that it's a bug:

• I've gone though the User Guide and the API reference
• I've searched for previous similar issues and didn't find any solution

Describe the bug
A clear and concise description of what the bug is.

SDK version number

Platform/OS/Hardware/Device
What are you running the cli on?

To Reproduce (observed behavior)
Steps to reproduce the behavior

Expected behavior
A clear and concise description of what you expected to happen.

Logs/output
Get full traceback and error logs by adding --debug to the command.

Additional context
Add any other context about the problem here.

[WhyNotHugo]

Sure! Keep in mind that you can add ISSUE_TEMPLATE.md file with the template to the root of the repository so that it shows up when someone opens a new issue.

---

Describe the bug
Attempting to run execute-command onto a task with a read-only root volume always fails.

SDK version number

$ aws --version
aws-cli/1.19.33 Python/3.9.2 Linux/5.11.11-arch1-1 botocore/1.20.33

Platform/OS/Hardware/Device

$ uname -srmo
Linux 5.11.11-arch1-1 x86_64 GNU/Linux

To Reproduce (observed behavior)

Run aws ecs execute-command --cluster production --task 38b79c36e14843dd9b479592f6f42cfd --container enviatufoto --command bash --interactive.

Expected behavior

It should open an interactive session.

Actual behaviour

The Session Manager plugin was installed successfully. Use the AWS CLI to start a session.


An error occurred (TargetNotConnectedException) when calling the ExecuteCommand operation: The execute command failed due to an internal error. Try again later.

Logs/output

2021-04-07 19:24:17,761 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/1.19.33 Python/3.9.2 Linux/5.11.11-arch1-1 botocore/1.20.33
2021-04-07 19:24:17,761 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['--debug', 'ecs', 'execute-command', '--cluster', 'production', '--task', '38b79c36e14843dd9b479592f6f42cfd', '--container', 'enviatufoto', '--command', 'bash', '--interactive']
2021-04-07 19:24:17,761 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_scalar_parsers at 0x7fcd196940d0>
2021-04-07 19:24:17,761 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function register_uri_param_handler at 0x7fcd19bb28b0>
2021-04-07 19:24:17,761 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_assume_role_provider_cache at 0x7fcd19b93940>
2021-04-07 19:24:17,762 - MainThread - botocore.utils - DEBUG - IMDS ENDPOINT: http://169.254.169.254/
2021-04-07 19:24:17,763 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function attach_history_handler at 0x7fcd197cd790>
2021-04-07 19:24:17,764 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/lib/python3.9/site-packages/botocore/data/ecs/2014-11-13/service-2.json
2021-04-07 19:24:17,769 - MainThread - botocore.hooks - DEBUG - Event building-command-table.ecs: calling handler <function inject_commands at 0x7fcd196fcee0>
2021-04-07 19:24:17,769 - MainThread - botocore.hooks - DEBUG - Event building-command-table.ecs: calling handler <function add_waiters at 0x7fcd1969b940>
2021-04-07 19:24:17,774 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/lib/python3.9/site-packages/botocore/data/ecs/2014-11-13/waiters-2.json
2021-04-07 19:24:17,775 - MainThread - awscli.clidriver - DEBUG - OrderedDict([('cluster', <awscli.arguments.CLIArgument object at 0x7fcd194a5940>), ('container', <awscli.arguments.CLIArgument object at 0x7fcd194a5880>), ('command', <awscli.arguments.CLIArgument object at 0x7fcd194a5850>), ('interactive', <awscli.arguments.BooleanArgument object at 0x7fcd194a57f0>), ('no-interactive', <awscli.arguments.BooleanArgument object at 0x7fcd194a57c0>), ('task', <awscli.arguments.CLIArgument object at 0x7fcd194a5790>)])
2021-04-07 19:24:17,775 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.ecs.execute-command: calling handler <function add_streaming_output_arg at 0x7fcd19694430>
2021-04-07 19:24:17,775 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.ecs.execute-command: calling handler <function rename_arg.<locals>._rename_arg at 0x7fcd1965c430>
2021-04-07 19:24:17,775 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.ecs.execute-command: calling handler <function rename_arg.<locals>._rename_arg at 0x7fcd1965c4c0>
2021-04-07 19:24:17,775 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.ecs.execute-command: calling handler <function add_cli_input_json at 0x7fcd19b96310>
2021-04-07 19:24:17,775 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.ecs.execute-command: calling handler <function unify_paging_params at 0x7fcd19720820>
2021-04-07 19:24:17,780 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/lib/python3.9/site-packages/botocore/data/ecs/2014-11-13/paginators-1.json
2021-04-07 19:24:17,780 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.ecs.execute-command: calling handler <function add_generate_skeleton at 0x7fcd196ff9d0>
2021-04-07 19:24:17,780 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.ecs.execute-command: calling handler <bound method OverrideRequiredArgsArgument.override_required_args of <awscli.customizations.cliinputjson.CliInputJSONArgument object at 0x7fcd194a5a90>>
2021-04-07 19:24:17,780 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.ecs.execute-command: calling handler <bound method GenerateCliSkeletonArgument.override_required_args of <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x7fcd194a5c70>>
2021-04-07 19:24:17,781 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.ecs.execute-command.cluster: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fcd195dffd0>
2021-04-07 19:24:17,781 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.ecs.execute-command: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7fcd19bd6bb0>
2021-04-07 19:24:17,781 - MainThread - awscli.arguments - DEBUG - Unpacked value of 'production' for parameter "cluster": 'production'
2021-04-07 19:24:17,782 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.ecs.execute-command.container: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fcd195dffd0>
2021-04-07 19:24:17,782 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.ecs.execute-command: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7fcd19bd6bb0>
2021-04-07 19:24:17,782 - MainThread - awscli.arguments - DEBUG - Unpacked value of 'enviatufoto' for parameter "container": 'enviatufoto'
2021-04-07 19:24:17,782 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.ecs.execute-command.command: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fcd195dffd0>
2021-04-07 19:24:17,782 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.ecs.execute-command: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7fcd19bd6bb0>
2021-04-07 19:24:17,782 - MainThread - awscli.arguments - DEBUG - Unpacked value of 'bash' for parameter "command": 'bash'
2021-04-07 19:24:17,782 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.ecs.execute-command.interactive: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fcd195dffd0>
2021-04-07 19:24:17,783 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.ecs.execute-command.task: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fcd195dffd0>
2021-04-07 19:24:17,783 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.ecs.execute-command: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7fcd19bd6bb0>
2021-04-07 19:24:17,783 - MainThread - awscli.arguments - DEBUG - Unpacked value of '38b79c36e14843dd9b479592f6f42cfd' for parameter "task": '38b79c36e14843dd9b479592f6f42cfd'
2021-04-07 19:24:17,783 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.ecs.execute-command.cli-input-json: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fcd195dffd0>
2021-04-07 19:24:17,783 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.ecs.execute-command.generate-cli-skeleton: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fcd195dffd0>
2021-04-07 19:24:17,783 - MainThread - botocore.hooks - DEBUG - Event calling-command.ecs.execute-command: calling handler <bound method CliInputJSONArgument.add_to_call_parameters of <awscli.customizations.cliinputjson.CliInputJSONArgument object at 0x7fcd194a5a90>>
2021-04-07 19:24:17,783 - MainThread - botocore.hooks - DEBUG - Event calling-command.ecs.execute-command: calling handler <bound method GenerateCliSkeletonArgument.generate_json_skeleton of <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x7fcd194a5c70>>

The Session Manager plugin was installed successfully. Use the AWS CLI to start a session.

2021-04-07 19:24:17,791 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: env
2021-04-07 19:24:17,791 - MainThread - botocore.credentials - INFO - Found credentials in environment variables.
2021-04-07 19:24:17,791 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/lib/python3.9/site-packages/botocore/data/endpoints.json
2021-04-07 19:24:17,796 - MainThread - botocore.hooks - DEBUG - Event choose-service-name: calling handler <function handle_service_name_alias at 0x7fcd1a1f4d30>
2021-04-07 19:24:17,797 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.ecs: calling handler <function add_generate_presigned_url at 0x7fcd1a2134c0>
2021-04-07 19:24:17,802 - MainThread - botocore.endpoint - DEBUG - Setting ecs timeout as (60, 60)
2021-04-07 19:24:17,803 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/lib/python3.9/site-packages/botocore/data/_retry.json
2021-04-07 19:24:17,804 - MainThread - botocore.client - DEBUG - Registering retry handlers for service: ecs
2021-04-07 19:24:17,804 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.ecs.ExecuteCommand: calling handler <function generate_idempotent_uuid at 0x7fcd1a1a4040>
2021-04-07 19:24:17,805 - MainThread - botocore.hooks - DEBUG - Event before-call.ecs.ExecuteCommand: calling handler <function inject_api_version_header_if_needed at 0x7fcd1a1a58b0>
2021-04-07 19:24:17,805 - MainThread - botocore.endpoint - DEBUG - Making request for OperationModel(name=ExecuteCommand) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'X-Amz-Target': 'AmazonEC2ContainerServiceV20141113.ExecuteCommand', 'Content-Type': 'application/x-amz-json-1.1', 'User-Agent': 'aws-cli/1.19.33 Python/3.9.2 Linux/5.11.11-arch1-1 botocore/1.20.33'}, 'body': b'{"cluster": "production", "container": "enviatufoto", "command": "bash", "interactive": true, "task": "38b79c36e14843dd9b479592f6f42cfd"}', 'url': 'https://ecs.us-west-2.amazonaws.com/', 'context': {'client_region': 'us-west-2', 'client_config': <botocore.config.Config object at 0x7fcd195d3700>, 'has_streaming_input': False, 'auth_type': None}}
2021-04-07 19:24:17,805 - MainThread - botocore.hooks - DEBUG - Event request-created.ecs.ExecuteCommand: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7fcd19212e50>>
2021-04-07 19:24:17,805 - MainThread - botocore.hooks - DEBUG - Event choose-signer.ecs.ExecuteCommand: calling handler <function set_operation_specific_signer at 0x7fcd1a19dee0>
2021-04-07 19:24:17,805 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth.
2021-04-07 19:24:17,805 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
POST
/

content-type:application/x-amz-json-1.1
host:ecs.us-west-2.amazonaws.com
x-amz-date:20210407T172417Z
x-amz-target:AmazonEC2ContainerServiceV20141113.ExecuteCommand

content-type;host;x-amz-date;x-amz-target
aa53f0e990306d56779f36a4459726c3e274bb4641ba811474831929d63aa890
2021-04-07 19:24:17,806 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20210407T172417Z
20210407/us-west-2/ecs/aws4_request
ff3a243fffe76265a7630a6c6f5bd1938db0a781c6e6ec4bdb6ecf159af212d0
2021-04-07 19:24:17,806 - MainThread - botocore.auth - DEBUG - Signature:
1b32a472ca83ceb97900a9681d8f93e014178f81e1f22225e25c7d6eb0402990
2021-04-07 19:24:17,806 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=https://ecs.us-west-2.amazonaws.com/, headers={'X-Amz-Target': b'AmazonEC2ContainerServiceV20141113.ExecuteCommand', 'Content-Type': b'application/x-amz-json-1.1', 'User-Agent': b'aws-cli/1.19.33 Python/3.9.2 Linux/5.11.11-arch1-1 botocore/1.20.33', 'X-Amz-Date': b'20210407T172417Z', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIA37EBIUN4ADPD6Z4Y/20210407/us-west-2/ecs/aws4_request, SignedHeaders=content-type;host;x-amz-date;x-amz-target, Signature=1b32a472ca83ceb97900a9681d8f93e014178f81e1f22225e25c7d6eb0402990', 'Content-Length': '137'}>
2021-04-07 19:24:17,807 - MainThread - botocore.httpsession - DEBUG - Certificate path: /usr/lib/python3.9/site-packages/certifi/cacert.pem
2021-04-07 19:24:17,807 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): ecs.us-west-2.amazonaws.com:443
2021-04-07 19:24:18,989 - MainThread - urllib3.connectionpool - DEBUG - https://ecs.us-west-2.amazonaws.com:443 "POST / HTTP/1.1" 400 122
2021-04-07 19:24:18,991 - MainThread - botocore.parsers - DEBUG - Response headers: {'x-amzn-RequestId': 'd0315d17-9d78-4714-9e24-16aba018d42a', 'Content-Type': 'application/x-amz-json-1.1', 'Content-Length': '122', 'Date': 'Wed, 07 Apr 2021 17:24:18 GMT', 'Connection': 'close'}
2021-04-07 19:24:18,992 - MainThread - botocore.parsers - DEBUG - Response body:
b'{"__type":"TargetNotConnectedException","message":"The execute command failed due to an internal error. Try again later."}'
2021-04-07 19:24:18,996 - MainThread - botocore.parsers - DEBUG - Response headers: {'x-amzn-RequestId': 'd0315d17-9d78-4714-9e24-16aba018d42a', 'Content-Type': 'application/x-amz-json-1.1', 'Content-Length': '122', 'Date': 'Wed, 07 Apr 2021 17:24:18 GMT', 'Connection': 'close'}
2021-04-07 19:24:18,996 - MainThread - botocore.parsers - DEBUG - Response body:
b'{"__type":"TargetNotConnectedException","message":"The execute command failed due to an internal error. Try again later."}'
2021-04-07 19:24:18,997 - MainThread - botocore.hooks - DEBUG - Event needs-retry.ecs.ExecuteCommand: calling handler <botocore.retryhandler.RetryHandler object at 0x7fcd19219970>
2021-04-07 19:24:18,997 - MainThread - botocore.retryhandler - DEBUG - No retry needed.
2021-04-07 19:24:18,999 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/awscli/clidriver.py", line 217, in main
    return command_table[parsed_args.command](remaining, parsed_args)
  File "/usr/lib/python3.9/site-packages/awscli/clidriver.py", line 358, in __call__
    return command_table[parsed_args.operation](remaining, parsed_globals)
  File "/usr/lib/python3.9/site-packages/awscli/clidriver.py", line 527, in __call__
    return self._operation_caller.invoke(
  File "/usr/lib/python3.9/site-packages/awscli/customizations/ecs/executecommand.py", line 91, in invoke
    response = client.execute_command(**parameters)
  File "/usr/lib/python3.9/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/lib/python3.9/site-packages/botocore/client.py", line 676, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.TargetNotConnectedException: An error occurred (TargetNotConnectedException) when calling the ExecuteCommand operation: The execute command failed due to an internal error. Try again later.
2021-04-07 19:24:19,002 - MainThread - awscli.clidriver - DEBUG - Exiting with rc 255

An error occurred (TargetNotConnectedException) when calling the ExecuteCommand operation: The execute command failed due to an internal error. Try again later.

Additional context

The only difference I'm seeing between tasks that fail and tasks that don't is the readonlyRootFilesystem flag for the container, so I'm strongly suspecting that is the culprit.

---

Sorry if I messed up the formatting, it seems that GitHub strips formatting when copy-pasting for some reason.

[kdaily]

Thanks for providing the details, @WhyNotHugo!

I found this thread on the AWS Containers Roadmap repository of users running into this problem:

aws/containers-roadmap#187

See here specifically:

• How to run "docker exec... " command in ECS containers-roadmap#187 (comment)
• How to run "docker exec... " command in ECS containers-roadmap#187 (comment)

Otherwise we may need to get you in touch with AWS support to determine more completely what the issue could be. Having the request ID from your response will be helpful (it's d0315d17-9d78-4714-9e24-16aba018d42a).

Separately, I'm not sure why you didn't get prompted with an issue template! Ours live here, and we disabled the ability to open an issue without a template. If you can let me know how you got in that state it would help me improve the developer experience here!

[WhyNotHugo]

Those comments indicate that their issue lied elsewhere (missing IAM role).

Like I said, if I turn off readonlyRootFilesystem it works, so there's clearly no IAM roles missing. The problem is that ExecuteCommand fails if the container has a readonly filesystem.

[WhyNotHugo]

Oh, that issue does point to aws-containers/amazon-ecs-exec-checker#21, which also mentions my problem. It would seem (so far), that it's an issue with the current implementation on the AWS side.

[kdaily]

Thanks for finding that @WhyNotHugo - I think I can close this in lieu of that issue:

aws-containers/amazon-ecs-exec-checker#21 (comment)

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.