Infrastructure Dashboard

We need a single location to organize our infrastructure monitoring.

Step 2: Make sure we can provide a community view

for each stack (aka aws, radiant, aws-dev):

• scheduler view
  • what sources are loading
    • ways to look at the logging for those source. Gleaner pull logs from Minio. Others pull info from minio
  • what sources are queued
• portainer
  • what sources are running
  • simply show headless container logs to be sure it's not just failing).
• quick html overview of the reports
  • look in report bucket, list all and repos, let user click down to load an info page (which renders files from s3)
  • what tenants are associated with sources
• Triplestore
  • check if there is a long running query in each graphstore.
• overall minio/s3
  • what are the file counts for each source in path: summon
    • Ages of files (1 month or less), older than 1 month less than 6. older than 6

This should use streamlit, since it's quick to develop. Secrets for each stack should be stored as secrets.

Portainer:

secrets:

• PORTAINER_ENDPOINT FOR STACK:
• PORTAINER_ READ ONLY API TOKEN FOR STACK

use docker.py, and the portainer docker endpoint. Example using a portainer api key in scheduler.

logic for What sources are loading/queued:

read services, and look for sch_ services. Look at tasks properties in services with a running|queued|failed (may not need this filter).

Headless:

look for a container with headless in name: provide ability to query for logs.

GleanerIO:

• MAY WANT MINIO KEYS, but read only minio keys

• sources: list of sources from a gleaner config stored in s3

• tenant: list of sources in a tenant config stored in s3

Graph

parse the performance api for queries that are running after 10 minutes.