[ECS, Fargate] Support awslogs-endpoint task configuration option

[copumpkin]

Which service(s) is this request for?

Fargate, ECS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?

After @clareliguori helpfully pointed out that my issue #48 was mistaken and the issue was with the awslogs driver, I looked into the CloudWatch Logs VPCE support. Currently the only way to make it work is to update the VPC's DNS to point at the PrivateLink endpoint, which while fine for some situations, can be a bit coarse-grained for others (especially if there are nontrivial policies on the endpoint). As of a recent Docker (this commit adds it), the awslogs driver now supports the awslogs-endpoint configuration option, which would allow me to point my task directly at the logs VPCE that's appropriate for it. If I try to pass that into my task definition today, it tells me immediately that the option is invalid.

Are you currently working around this issue?

Just using DNS for the VPCE.

Edit: a broader way to phrase this might have been "Support Docker 18.09", as long as the front-end validation for the task definition schema notices that the new option is present 😄

[mailjunze]

By default "Private DNS" is not enabled for Cloudwatch endpoint, In order to use Cloudwatch with fargate You would need to "Enable Private DNS Name" for com.amazonaws.eu-west-1.logs. Go to the Cloudwatch Endpoint in VPC console > Actions > Modify Private DNS name. Also, make sure you're using the latest Fargate platform Version.

[pjelar]

Any update on this? I have a vpc endpoint with private DNS enable but I'm still unable to get logs.

[clareliguori]

The latest ECS AMIs now include Docker 18.09.9-ce:
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-ami-versions.html

[clareliguori]

Re-opening, my mistake: the task definition log driver options are validated in ECS, and do not yet allow awslogs-endpoint

[rdtunnicliffe]

Is there any way the awslogs-endpoint could be exposed in the APIs? I'm ssking because it would really help with migrating ECS Fargate Tasks to IPv6. Right now if you are using the awslogs driver in a mixed IPv4/IPv6 environment, tasks won't start if there isn't an IPv4 route to the outside world because they are trying to use the old, non-dualstack Cloudwatch logs endpoint. If this option was allowed in the API calls, then it would be trivial to force usage of the dualstack endpoint instead.