Infrastructure Dashboard

We need a single location to organize our infrastructure monitoring.

Step 2: Make sure we can provide a community view

for each stack (aka aws, radiant, aws-dev):

scheduler view
- what sources are loading
  - ways to look at the logging for those source. Gleaner pull logs from Minio. Others pull info from minio
- what sources are queued
portainer
- what sources are running
- simply show headless container logs to be sure it's not just failing).
quick html overview of the reports
- look in report bucket, list all and repos, let user click down to load an info page (which renders files from s3)
- what tenants are associated with sources
Triplestore
- check if there is a long running query in each graphstore.
overall minio/s3
- what are the file counts for each source in path: summon
  - Ages of files (1 month or less), older than 1 month less than 6. older than 6
sources view
- tab (sources/tenants) * sources list of sources from gleanerconfig.yaml * ability to click on a source and see source details page * tenant * name of tenant, description * link to client * sources with link to sources page * link to open a sparqlgui page pointing to that graph
source details page (link to from sources view)
- name
- used in tenant
- good html rendering of the last s3/graph load report
- count of files in s3/summon
- file size for release and summarize
- good html rendering of some stats from reports

This should use streamlit, since it's quick to develop. Secrets for each stack should be stored as secrets.

Portainer:

use docker.py, and the portainer docker endpoint. Example using a portainer api key in scheduler.

read services, and look for sch_ services. Look at tasks properties in services with a running|queued|failed (may not need this filter).

look for a container with headless in name: provide ability to query for logs.

parse the performance api for queries that are running after 10 minutes.