Document log aggregation via Loki

[BrianJKoopman]

Description

This PR adds a page in the docs describing the installation and setup of Loki, a log aggregation system made by Grafana Labs. It has nice integration with Grafana and allows viewing realtime logs in the Explore menu and configuring "Logs" visualizations in Dashboard panels.

Motivation and Context

Log visibility can often be cumbersome, especially on multi-node systems. This allows log aggregation across nodes in a centralized location and provides an easy viewing alternative to running docker logs -f <container> on each container you want to view the logs for.

How Has This Been Tested?

I have tested the configuration example in an isolated VM. I'm waiting to setup on the Yale system for our cooldown to be completed (tomorrow), since this requires remaking each container you want to forward logs.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist:

• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• Unless I am preparing a release, I have opened this PR onto the develop branch.

[jlashner]

Just tested this on the UCSD system and documentation worked pretty much flawlessly. One thing to maybe make a note about is that if you try to look at logs through docker-compose, you will get an error that looks like this:

ocs-aggregator_1         | WARNING: no logs are available with the 'loki' log driver

even though the container might be running successfully. You can still view logs with docker logs just not docker-compose logs for some reason, not that this will affect very many people... Thanks Brian!

[BrianJKoopman]

Great, thanks for the feedback! It's strange to me that they're still available to simple docker logs, but not docker-compose logs. Anyway, I added a small note about the availability of logs in these systems. Will merge once checks are completed.