Document suggested/example remote logging facility

[otterley]

We need a way to send system log streams for the kernel and all user-space components to external log collectors in real time.

For the EC2 AMI, we need to make it easy to stream logs to CloudWatch Logs.

Support for other log collectors can be added as well - it would be useful for interested users to indicate their preferences here and vote on them.

[zmrow]

We have no current plans to add a logging agent to the host OS.

When talking with many EKS customers, we found that a common pattern is to use Kubernetes’ facilities for log streaming, even for system level logs. Another method is to use Fluent Bit as covered in this blog post. These are our suggested methods for customers to get both container logs as well as other logs off the box.

[otterley]

@zmrow I agree that it doesn't necessarily need to be in the host OS per se. Consider this a recommendation to add some documentation/examples to enable the functionality for those who need it.

[gabegorelick]

I think the particulars about remote logging are less important than documenting what to log. In other words, it would be useful to have a reference of where notable log files live. Something that says "the journal is available at /var/log/journal, /var/log/messages doesn't exist, container logs are available at /var/log/containers/*.log, etc."

Especially with host containers, it can be hard to figure out where something like the admin container's ssh logs live (is it mounted on the host somewhere, or do we have to enter the container to get it?).

[stmcginnis]

@stockholmux can we get this issue transferred to the project-website repo to track the work there?

I think it would be useful to document how to create and deploy a logging and metrics host container as part of the overall docs.

Another option would be to have a blog post similar to https://aws.amazon.com/blogs/opensource/centralized-container-logging-fluent-bit/, but I think this comes up often enough that it would be useful to have it right in the docs.

[stockholmux]

Works for me @stmcginnis