readonlyRootFilesystem should be false (or, should not be configured) in task definition to use ECS Exec

[toricls]

The managed agent (= SSM agent) requires to create directories and files as described in this proposal.

See the details about readonlyRootFilesystem in the ECS official documentation.

This check should be implemented as a red item.

[WhyNotHugo]

Does this imply that ExecuteCommand only works if the filesystem is not mounted readonly?

Will this never be supported in future? This is a hard dealbreaker for me, since I cannot downgrade security in my production cluster and make the root read-write.

Might it be possible to mount a tmpfs in the right place as a workaround?

[toricls]

Hi @WhyNotHugo, thanks for the comment! As you pointed out ECS Exec requires the container storage writable at this moment. I also suggested the team explicitly mention on this in the ECS official documentation but it'll take some time I think.

Regarding workaround, I need to do some additional tests to comment on it but I think I can think of some ways to deal with. I will note that in the README or somewhere in this repository once we implemented this check.

[toricls]

Update - The following note has been added in the official ECS documentation.

The SSM agent requires that the container file system is able to be written to in order to create the required directories and files. Therefore, making the root file system read-only using the readonlyRootFilesystem task definition parameter, or any other method, isn't supported.

---

Dropping an update - we're going to clarify in the ECS doc with awsdocs/amazon-ecs-developer-guide#157 that ECS Exec doesn't support read-only containers at this moment. This change will be included in the next batch of the ECS doc release.

[sd65]

@toricls I've managed to get it working with readonlyRootFilesystem: true by mounting /managed-agents, /var/lib/amazon/ssm and /var/log/amazon/ssm as writable volumes inside. Works fine for me.

But there is still one undocumented limitation : some Linux capabilities are needed and it's too hard/slow to reverse engineer this... If you got any info on this, it would be highly appreciated !

[toricls]

@sd65 Thanks for the info!

Actually I was thinking very similar approach to make it work in read-only containers. There were two ways in my mind, by making those directories writable by 1) creating them in Dockerfile, or 2) creating them in ECS task definition using bind mount.

It's reasonable to me that the ssm-agent requires some of the default allowed linux caps. I assume you took one of the workarounds I described above with dropping some linux caps from your containers, but could you possibly elaborate more on your approach?

[L1Cafe]

@toricls I was talking to @sd65 in our correspondence. I was ready to open a new issue for the Linux capabilities situation on behalf of them on https://github.com/aws/containers-roadmap, do you want me to?

Also: Should I open an issue on https://github.com/aws/containers-roadmap in regards to this readonly matter? Hopefully we can either update the documentation or the way ECS Exec works.

Let me know your thoughts (reach me on Slack "Leon Castillejos" for more details).

[sd65]

@toricls You're completely right, the default allowed Linux capabilities are enough for the ECS exec agent to work. We have a drop all caps then whitelist approach on our containers, so I guess I will have to remove them one by one and see if it's still working 😛

@L1Cafe Glad to see you here, hope we can update the documentation soon to prevent more headaches.

[toricls]

@sd65 I would appreciate if you could create feature requests with your use cases in our containers roadmap repository as a user.
I think the requests will be two items:

1.ECS Exec feature request - Support readonlyRootFilesystem containers

This would be a feature request since we, the ECS team doesn't officially support read-only containers as described in the documentation here at this moment.

2.ECS Exex documentation request - Document on required Linux caps

This would be another request to doc update for explicitly stating the required Linux caps for ECS Exec on the ECS official doc or on the SSM agent doc.

[toricls]

Thank you @sd65!

[toricls]

Write a small article for working around this limitation - https://toris.io/2021/06/using-ecs-exec-with-readonlyrootfilesystem-enabled-containers/