Infrastructure Dashboard

We need a single location to organize our infrastructure monitoring.

Step 2: Make sure we can provide a community view

for each stack (aka aws, radiant, aws-dev):

• scheduler view
• what sources are loading
  • ways to look at the logging for those source. Gleaner pull logs from Minio. Others pull info from minio
• what sources are queued
• portainer
  • what sources are running
• simply show headless container logs to be sure it's not just failing).
• quick html overview of the reports
  • look in report bucket, list all and repos, let user click down to load an info page (which renders files from s3)
• what tenants are associated with sources
• check if there is a long running query in each graphstore.

This should use streamlit, since it's quick to develop. Secrets for each stack should be stored as secrets.

Portainer:

secrets:

PORTAINER_ENDPOINT FOR STACK: PORTAINER_ READ ONLY API TOKEN FOR STACK use docker.py, and the portainer docker endpoint. Example using a portainer api key in scheduler.

What sources are loading/queued: read services, and look for sch_ services. Look at tasks properties in services with a running|queued|failed (may not need this filter).

Headless: look for a container with headless in name: provide ability to query for logs.

GleanerIO:

MAY WANT MINIO KEYS, but read only minio keys

sources: list of sources from a gleaner config stored in s3

tenant: list of sources in a tenant config stored in s3

Graph

parse the performance api for queries that are running after 10 minutes.