Vigil

Vigil is a web-based network and systems monitor for Linux systems, homelabs, and small networks. Inspired by Uptime Kuma, Prometheus, Grafana, and Loki, it provides a centralized dashboard to configure and manage diverse infrastructure from a single pane of glass — without requiring agents on remote hosts.

Unlike most network and system monitors, Vigil is designed to be highly extensible and capable of performing actions on monitored targets, not just observing them.

Features

Pull-Based & Agentless: Uses a pull-based design over SSH, HTTP, DNS, and ICMP to collect events, logs, and metrics — no software needed on target nodes.
Web Dashboard: Real-time interactive visualizations built with NiceGUI and ECharts, featuring latency history, status distribution, and log views.
Alerting & Notifications: Sends alerts to various channels when events or metric thresholds are detected. (WIP)
Target Control: Trigger actions on monitored targets (e.g. restarting systemd services) directly from the UI.
Plugin Architecture: Core features are implemented through plugins called "monitors" which handle specific domains (systemd services, host uptime, hardware parameters, etc.).
Hierarchical Organization: Organize monitors into nested groups by location, service, or environment.
Lightweight: Minimal dependencies and low resource footprint.
Easy Development: Fully written in Python.

Architecture

Vigil is built around pure plugins and a Coordination Engine that owns all IO and persistence. A plugin is constructed with only its (name, config); it declares what to collect and how to interpret the results as pure functions, and the engine performs every side effect — SSH, HTTP/DNS/ICMP, database writes, thread offloading. This keeps plugins small, side-effect-free, and testable without mocks.

A plugin's contract is:

Declare — requests() returns a list of connector requests (Command / HttpRequest / DnsQuery / PingRequest); an SSH-only plugin overrides the commands() shorthand instead. Both are pure.
Interpret — parse_results() (or parse() for the SSH shorthand) turns the connector results into a single CollectResult describing everything to persist (metrics, logs, status, an optional snapshot). Pure — no IO.
Act (optional) — plan_action() / interpret_action() describe control actions (restart a service, force a backup) the same declarative way.
Present (optional) — a declarative UI_SPEC dict renders the plugin's dashboard page; only genuinely bespoke plugins hand-write render_ui().

The engine executes the declared IO, persists the returned CollectResult, and drives one independent polling loop per monitor.

Data Flow

Initialization: vigil/__main__.py builds one VigilEngine (core/coordination/engine.py), which loads config.yaml and instantiates plugins (setup_modules). Group plugins act as containers for nested monitors.
Wiring: for each plugin the engine builds its engine-owned SSH handle and injects a read-only PluginDataView as plugin.data. The plugin holds no database or connection object.
Polling: each monitor runs its own async loop at its own interval. Per cycle the engine runs the plugin's declared requests through the Connector Engine, then calls the plugin's pure parse_results().
Persistence: the engine writes the resulting CollectResult via db.apply_result(...) into SQLite (Peewee ORM). A background writer thread batches commits off the event loop.
Visualization: the NiceGUI dashboard polls the database on one shared per-client timer and renders the sidebar tree plus each plugin's detail page. It reads only through plugin.data / the database — never through a plugin's IO.
Export: metrics are exposed to Prometheus (pull, /metrics) and optionally pushed to InfluxDB.

Project Structure

vigil/
├── __main__.py              # Entry point: build engine, load plugins, start GUI
├── core/
│   ├── coordination/        # VigilEngine (Coordination Engine) + PluginDataView
│   ├── connectors/          # All IO: SSH + HTTP/DNS/ICMP sub-connectors, request types
│   ├── database/            # DatabaseManager (SQLite/Peewee), models, read-result types
│   ├── settings/            # config.yaml loader + typed schema
│   ├── exporters/           # Prometheus pull + InfluxDB push
│   └── ui/                  # NiceGUI dashboard, declarative UI_SPEC renderer
└── plugins/
    ├── base/                # Plugin ABC + shared config/helper mixins
    └── *.py                 # One module per monitor type (uptime, systemd_service, …)

See DEVELOP.md for the architectural rationale — the pure-plugin contract, the collection lifecycle, the SQLite writer/reader model, and the declarative UI spec.

Technical Stack

Concern	Technology
Language	Python 3.9+
Connectivity	AsyncSSH (SSH), `requests` (HTTP), dnspython (DNS)
Configuration	YAML
Concurrency	`asyncio`
Storage	SQLite via Peewee ORM
Frontend	NiceGUI + ECharts

Theme

All colors used in the dashboard can be overridden in the theme: section of config.yaml. All fields are optional — omit any field to keep its default.

Field	Default	Description
`primary`	`#00ACFF`	Header, links, and primary accents
`accent`	`#FF5500`	Secondary accent color
`background`	`#FFFFFF`	Sidebar and card backgrounds
`background_muted`	`#FAFAFA`	Page body background
`text`	`#111827`	Primary text
`text_muted`	`#6B7280`	Labels and secondary text
`status_online`	`lime`	Color shown when a monitor is online
`status_warning`	`gold`	Color shown when a monitor is in warning
`status_failed`	`red`	Color shown when a monitor has failed
`status_offline`	`lightgray`	Color shown when a monitor is offline

theme:
  primary: "#7C3AED"
  status_online: "limegreen"
  status_warning: "orange"

Plugin Types

Summary

Type	Monitors	Collection	Key metrics	Actions
`uptime`	Host reachability	ICMP ping	`up`, `latency_ms`	—
`push`	External heartbeat (dead man's switch)	REST API (caller pushes in)	`last_push_epoch`, `reported_up`, `value`	—
`dns_record`	DNS record resolution	DNS query (via dnspython, in-process)	`resolved`, `ttl`, `matches_expected`	—
`ddns_updater`	Dynamic DNS record kept current	Public IP lookup + DNS query (in-process)	`in_sync`, `last_update_epoch`	Force Update
`systemd_service`	systemd unit state / last run	SSH (`systemctl`)	`active` or `last_run_epoch`, `last_run_success`	Restart, Stop, Enable, Disable
`service_list`	Systemd unit browser and control	SSH (`systemctl`)	`services_total`, `services_active`, `services_failed`	Start, Stop, Restart, Enable, Disable, View Status
`smart_disk`	Physical disk SMART health	SSH (`smartctl`)	`disks_total`, `disks_ok`, `disks_failed`	—
`zfs_health`	ZFS pool health state	SSH (`zpool list`)	`pools_total`, `pools_ok`, `pools_degraded`	—
`zfs_pool`	ZFS pool capacity	SSH (`zpool list`)	`usage_pct`	—
`disk_space`	Filesystem usage for a path	SSH (`df`)	`used_pct`, `size_gb`, `used_gb`, `avail_gb`	—
`cpu_usage`	CPU utilization	SSH (`/proc/stat`, 2-sample)	`cpu_pct`	—
`memory_usage`	RAM usage	SSH (`/proc/meminfo`)	`memory_pct`, `memory_used_gb`, `memory_total_gb`	—
`temperature`	Max thermal-zone temperature	SSH (`/sys/class/thermal`)	`temp_c`	—
`load_average`	Load average (normalized by cores)	SSH (`/proc/loadavg`, `nproc`)	`load_pct_1m`, `load_pct_5m`, `load_pct_15m`	—
`processes`	Running processes by CPU	SSH (`ps`)	`process_count`, `top_cpu_pct` (ephemeral)	SIGTERM, SIGKILL
`network_usage`	Network interface throughput	SSH (`/proc/net/dev`, 2-sample)	`rx_kbps`, `tx_kbps`	—
`diskio`	Per-disk read/write throughput	SSH (`/proc/diskstats`, 2-sample)	`read_kbps`, `write_kbps`	—
`interrupts`	Interrupt & context-switch rates	SSH (`/proc/stat`, 2-sample)	`irq_per_sec`, `ctxt_per_sec`	—
`connections`	TCP connection counts by state	SSH (`/proc/net/tcp`)	`total` + per-state (`established`, `listen`, …)	—
`wifi`	WiFi link quality & signal	SSH (`/proc/net/wireless`)	`link_quality`, `signal_dbm`	—
`ports`	TCP port / URL reachability	SSH (`/dev/tcp`, `curl`)	`<check>_up`, `<check>_latency_ms`	—
`borg`	Borg backup freshness	SSH (`borg list`)	`archive_count`, `last_backup_epoch`	—
`gpu`	NVIDIA GPU util / VRAM / temperature	SSH (`nvidia-smi`)	`gpu_util`, `gpu_mem_pct`, `gpu_temp` (+ per-GPU)	—
`containers`	Docker / Podman container states	SSH (`docker`/`podman ps`)	`containers_total`, `containers_running`, `containers_stopped`	Restart (per expected container)
`raid`	Linux software RAID (mdadm) health	SSH (`/proc/mdstat`)	`arrays_total`, `arrays_ok`, `arrays_degraded`	—
`command`	Arbitrary command (generic check)	SSH (any command)	`exit_code` (+ `value` in pattern mode)	—
`filesystems`	All mounted filesystems (auto-discovered)	SSH (`df`)	`worst_used_pct`, `fs_<mount>_used_pct`	—
`folders`	Sizes of arbitrary directories	SSH (`du`)	`worst_folder_gb`, `folder_<path>_gb`	—
`vms`	libvirt/KVM virtual machines	SSH (`virsh`)	`vms_total`, `vms_running`, `vms_stopped`	Start, Shutdown (per expected VM)
`cloud`	Cloud instance metadata (AWS/GCP/Azure)	SSH (metadata endpoint)	`on_cloud`	—
`group`	Container for nested monitors	— (aggregates children)	—	—

All plugin types share these common fields:

Field	Description
`name`	Display name shown in the sidebar and dashboard
`id`	Unique identifier used internally (defaults to `name` if omitted)
`type`	Plugin type — one of `uptime`, `push`, `dns_record`, `ddns_updater`, `systemd_service`, `service_list`, `smart_disk`, `zfs_health`, `zfs_pool`, `disk_space`, `network_usage`, `diskio`, `interrupts`, `connections`, `wifi`, `ports`, `cpu_usage`, `memory_usage`, `temperature`, `load_average`, `processes`, `borg`, `gpu`, `containers`, `raid`, `command`, `filesystems`, `folders`, `vms`, `cloud`, `group`
`interval`	Polling frequency in seconds (default: 60)

`uptime`

Checks host availability using ICMP ping.

Option	Description
`target_host`	IP address or hostname to ping
`interval`	Polling frequency in seconds (default: `60`)

Metrics: up (1/0), latency_ms

- name: "Core Gateway"
  id: "gateway-ping"
  type: "uptime"
  target_host: "192.168.1.1"
  interval: 30

`push`

The inverse of every other monitor: instead of Vigil reaching out to a target, an external script, cron job, or task with no fixed host calls Vigil's REST API to say "I'm alive." Vigil reports failed once a heartbeat hasn't arrived within max_age — a dead man's switch, not a poll.

interval controls how often Vigil checks for staleness, not how often heartbeats are expected — that's max_age's job. A missing max_age defaults to twice the interval, tolerating one missed beat before alarming.

Option	Description
`max_age`	Seconds since the last heartbeat before reporting `failed` (default: `interval * 2`)
`token`	Shared secret the caller must present when pushing. Required — without one, anyone who can reach the API could mark this monitor healthy. Generate with `openssl rand -hex 20`.
`interval`	How often the staleness check itself runs (default: `60`)

Metrics: last_push_epoch (Unix timestamp of the last heartbeat), reported_up (1/0, the caller's own status), value (optional, if the caller supplies one)

To push a heartbeat, hit GET or POST /api/push/{id}/{token}, optionally with status (up or down, default up), msg, and value query parameters. This endpoint is not covered by the dashboard's HTTP Basic Auth (see Authentication) — the per-monitor token is its credential instead.

- name: "Nightly Backup Job"
  id: "nightly-backup"
  type: "push"
  interval: 1h
  max_age: 26h   # daily job, tolerate one slipped run
  token: "a1b2c3d4e5f6..."   # openssl rand -hex 20

# At the end of the cron job:
curl "https://vigil.example.com/api/push/nightly-backup/a1b2c3d4e5f6...?status=up"

# Or report a failure the job detected itself, while still checking in on time:
curl "https://vigil.example.com/api/push/nightly-backup/a1b2c3d4e5f6...?status=down&msg=disk+full"

`dns_record`

Resolves a DNS record and reports failed on NXDOMAIN, no answer, a timeout, or (when expected is set) an answer outside the accepted values — catching a stale record, a botched migration, or a DNS provider outage.

Runs in-process via dnspython rather than over SSH: there is no target host, only a domain to ask about. Query the system resolver by default, or point resolver at a specific one (public or internal) — doing so doubles as a liveness probe for that resolver, distinct from unbound's SERVFAIL-rate monitoring of one resolver's own stats.

Option	Description
`domain`	Domain name to query (required)
`record_type`	One of `A`, `AAAA`, `CNAME`, `MX`, `TXT`, `NS`, `SOA` (default: `A`)
`resolver`	Resolver IP to query directly (default: system resolver)
`port`	Resolver port (default: `53`)
`timeout`	Query timeout in seconds (default: `5`)
`expected`	(Optional) List of acceptable answer values. Any answer outside this list fails the monitor. Order-independent — only presence in the answer set is checked.

Metrics: resolved (1/0), ttl (seconds), matches_expected (1/0, only when expected is set)

# Pin an A record to known IPs — fails if it ever points elsewhere
- name: "Website A Record"
  id: "website-a-record"
  type: "dns_record"
  domain: "example.com"
  record_type: "A"
  expected:
    - "93.184.216.34"
  interval: 5m

# Confirm MX still points at the expected mail provider
- name: "Mail Routing"
  id: "example-mx"
  type: "dns_record"
  domain: "example.com"
  record_type: "MX"
  expected:
    - "10 mail.example.com"
  interval: 1h

# Query a specific resolver directly, e.g. to check an internal DNS server
- name: "Internal DNS"
  id: "internal-dns-check"
  type: "dns_record"
  domain: "heimdall.technet"
  resolver: "10.0.0.1"
  interval: 1m

`ddns_updater`

Keeps a DNS record pointed at this network's current public IP, and reports on it while doing so — a built-in replacement for standalone dynamic-DNS-updater services. Each cycle: look up the current public IP, resolve what the domain currently answers publicly, and push an update to the provider only when the two differ. Because a provider update is a real side effect (and most providers rate-limit or ban accounts that call too often), it is never fired on a fixed schedule — only on detected drift, and even then no more often than min_interval.

Currently speaks FreeDNS's (afraid.org, including *.mooo.com and its other free subdomains) per-host dynamic update URL convention: a plain HTTPS GET to a secret, account-specific URL that updates the record to the caller's apparent IP, responding good <ip> or nochg <ip> on success. Other providers using the same "secret update URL" convention work too; anything returning JSON or requiring a signed request does not, yet.

Resolves the public record against an explicit resolver (default 8.8.8.8) rather than the local/default resolver — a local network commonly has a hosts-file override pinning this exact hostname to a LAN IP (so internal clients don't route out to the internet and back for it), which would mask real DDNS drift by always answering with that LAN IP instead.

Option	Description
`domain`	Domain whose public record is kept current (required)
`update_url`	Provider's per-host dynamic update URL, including its own secret token
`update_url_file`	Path to a file containing the update URL (keeps the token out of config.yaml)
`update_url_command`	Shell command whose stdout is the update URL
`resolver`	Resolver IP to check the current public record against (default: `8.8.8.8`)
`record_type`	Record type being kept current (default: `A`)
`timeout`	Timeout in seconds for both the IP lookup and the update request (default: `10`)
`min_interval`	Minimum seconds between update attempts regardless of how often `interval` ticks (default: `300`)

Precedence when more than one update-URL source is set: update_url > update_url_file > update_url_command.

Metrics: in_sync (1/0), last_update_epoch (Unix timestamp of the last successful push)

Actions: Force Update — pushes an update immediately regardless of detected drift, bypassing min_interval.

- name: "DDNS"
  id: "ddns-bltechnet"
  type: "ddns_updater"
  domain: "bltechnet.mooo.com"
  update_url_file: "/run/secrets/freedns_update_url"
  interval: 5m

`systemd_service`

Monitors systemd units over SSH. Operates in two modes depending on whether max_age is set.

Continuous mode (default) — for long-running daemons. Checks systemctl is-active each cycle and reports online/warning/failed.

Oneshot mode (max_age set) — for timer-driven services that run and exit (e.g. nixos-upgrade, backup jobs). Checks the result and timestamp of the last completed run via systemctl show. Reports failed if the last run did not succeed or completed more than max_age seconds ago.

Option	Description
`service_name`	Name of the systemd unit (e.g. `nginx.service`)
`lines`	Number of `journalctl` log lines to fetch per cycle (default: `10`)
`interval`	Polling frequency in seconds (default: `60`)
`max_age`	(Oneshot mode) Max seconds since last successful run before reporting `failed`
`ssh_config`	SSH connection details — see SSH Config below
`allow_unit_file_edit`	Enable UI editing of the target service unit file. Defaults to `false`.
`allowed_write_paths`	Optional list of absolute paths where unit file writes are permitted. Defaults to standard systemd unit paths.

Continuous metrics: active (1/0)

Oneshot metrics: last_run_epoch (Unix timestamp), last_run_success (1/0)

Actions: Restart Service, Stop Service, Enable on Boot, Disable on Boot

The service detail UI also includes:

View Unit File for the configured service
Reload Daemon to run systemctl daemon-reload
Edit Unit File when allow_unit_file_edit: true is enabled

Note: Remote editing requires passwordless sudo access for the service commands and the configured write helper (currently python3 on the target). Restrict allowed_write_paths carefully.

# Continuous — long-running daemon
- name: "Nginx"
  id: "nginx-service"
  type: "systemd_service"
  service_name: "nginx.service"
  interval: 60
  ssh_config:
    host: "web-01.example.com"

# Oneshot — weekly timer-driven service
- name: "NixOS Upgrade"
  id: "myhost-nixos-upgrade"
  type: "systemd_service"
  service_name: "nixos-upgrade.service"
  interval: 3600
  max_age: 604800  # 1 week
  ssh_config:
    host: "myhost.example.com"

`service_list`

Lists all systemd services on a host and provides a sortable service browser with control actions.

Option	Description
`lines`	Number of recent log lines to preserve for this plugin's internal event log (default: `10`)
`interval`	Polling frequency in seconds (default: `60`)
`ssh_config`	SSH connection details — see SSH Config below
`allow_unit_file_edit`	Allow editing of unit files from the UI (disabled by default).
`allowed_write_paths`	Optional list of absolute paths where unit file writes are permitted. Defaults to standard systemd unit paths.

Metrics: services_total, services_active, services_failed

Actions: Reload Daemon

The service browser renders a sortable table of all units, and offers per-unit actions for:

Start Service
Stop Service
Restart Service
Enable on Boot
Disable on Boot
View Status
View Unit File
Edit Unit File (when enabled)

- name: "Systemd Service Browser"
  id: "systemd-service-browser"
  type: "service_list"
  interval: 60
  ssh_config:
    host: "web-01.example.com"
  allow_unit_file_edit: true
  allowed_write_paths:
    - /etc/systemd/system

# Continuous with unit file editing enabled
- name: "Nginx"
  id: "nginx-service"
  type: "systemd_service"
  service_name: "nginx.service"
  interval: 60
  ssh_config:
    host: "web-01.example.com"
  allow_unit_file_edit: true
  allowed_write_paths:
    - /etc/systemd/system

`smart_disk`

Monitors SMART health of all physical disks over SSH. Discovers disks automatically via lsblk and runs smartctl -H on each one. USB-attached disks are probed with -d sat.

The SSH user must have passwordless sudo access to smartctl (e.g. vigil ALL=(ALL) NOPASSWD: /usr/bin/smartctl).

Option	Description
`interval`	Polling frequency in seconds (default: `60`, recommend `3600`)
`ssh_config`	SSH connection details — see SSH Config below

Metrics: disks_total, disks_ok, disks_failed

- name: "Ragnarok SMART"
  id: "ragnarok-smart"
  type: "smart_disk"
  interval: 3600
  ssh_config:
    host: "ragnarok.technet"

`zfs_health`

Monitors ZFS pool health states over SSH via zpool list -H -o name,health. Reports failed if any pool is in a DEGRADED, FAULTED, OFFLINE, UNAVAIL, or REMOVED state. Complements zfs_pool (capacity) with structural integrity monitoring.

Option	Description
`interval`	Polling frequency in seconds (default: `60`, recommend `3600`)
`ssh_config`	SSH connection details — see SSH Config below

Metrics: pools_total, pools_ok, pools_degraded

- name: "Ragnarok ZFS Health"
  id: "ragnarok-zfs-health"
  type: "zfs_health"
  interval: 3600
  ssh_config:
    host: "ragnarok.technet"

`disk_space`

Monitors disk space usage for a path or mountpoint over SSH via df. Works on any mounted Linux filesystem — no ZFS or other tools required. Marks the path failed when usage exceeds the configured threshold.

Option	Description
`path`	Filesystem path or mountpoint to monitor (e.g. `/`, `/Storage`)
`threshold`	Usage percentage that triggers a `failed` status (default: `90`)
`interval`	Polling frequency (default: `60`, recommend `10m`)
`ssh_config`	SSH connection details — see SSH Config below

Metrics: used_pct, size_gb, used_gb, avail_gb

- name: "Root Disk"
  id: "myhost-disk-root"
  type: "disk_space"
  path: "/"
  threshold: 90
  interval: 10m
  ssh_config:
    host: "myhost.example.com"

`cpu_usage`

Monitors CPU utilization over SSH. Takes two /proc/stat snapshots one second apart in a single SSH command and computes the usage delta — no agents or extra tools required.

Option	Description
`cpu_warning`	CPU % that triggers `warning` (default: `70`)
`cpu_threshold`	CPU % that triggers `failed` (default: `85`)
`interval`	Polling frequency (default: `60`)
`ssh_config`	SSH connection details — see SSH Config below

Metrics: cpu_pct

- name: "Heimdall CPU"
  id: "heimdall-cpu"
  type: "cpu_usage"
  interval: 1m
  cpu_warning: 70
  cpu_threshold: 85
  ssh_config:
    host: "heimdall.example.com"

`memory_usage`

Monitors memory usage over SSH via /proc/meminfo. Uses MemAvailable (not MemFree) so filesystem cache is not counted as used. Single SSH read — no sleep required.

Option	Description
`memory_warning`	Memory % that triggers `warning` (default: `75`)
`memory_threshold`	Memory % that triggers `failed` (default: `90`)
`interval`	Polling frequency (default: `60`)
`ssh_config`	SSH connection details — see SSH Config below

Metrics: memory_pct, memory_used_gb, memory_total_gb

- name: "Heimdall Memory"
  id: "heimdall-memory"
  type: "memory_usage"
  interval: 1m
  memory_warning: 75
  memory_threshold: 90
  ssh_config:
    host: "heimdall.example.com"

`temperature`

Monitors system temperature over SSH via /sys/class/thermal/thermal_zone*/temp. Reports the maximum temperature across all thermal zones. Gracefully stays online with no metric when no thermal zones are present (e.g. VMs).

Option	Description
`temp_warning`	°C that triggers `warning` (default: `70`)
`temp_threshold`	°C that triggers `failed` (default: `80`)
`interval`	Polling frequency (default: `60`)
`ssh_config`	SSH connection details — see SSH Config below

Metrics: temp_c

- name: "Heimdall Temperature"
  id: "heimdall-temperature"
  type: "temperature"
  interval: 1m
  temp_warning: 70
  temp_threshold: 80
  ssh_config:
    host: "heimdall.example.com"

`load_average`

Monitors system load averages over SSH via /proc/loadavg. Load values are normalized by CPU core count (via nproc) and stored as a percentage — 100% means the system is exactly at capacity. Falls back to treating core count as 1 if nproc is unavailable. Thresholds are optional — when unset, load is collected and displayed but does not affect status.

Option	Description
`load_warning`	1m load as % of cores that triggers `warning` (optional — omit to disable)
`load_threshold`	1m load as % of cores that triggers `failed` (optional — omit to disable)
`interval`	Polling frequency (default: `60`)
`ssh_config`	SSH connection details — see SSH Config below

Metrics: load_pct_1m, load_pct_5m, load_pct_15m

- name: "Heimdall Load"
  id: "heimdall-load"
  type: "load_average"
  interval: 1m
  load_warning: 70    # warn when 1m load exceeds 70% of available cores
  load_threshold: 100 # fail when 1m load exceeds 100% of available cores
  ssh_config:
    host: "heimdall.example.com"

`processes`

Monitors running processes over SSH via ps, sorted by CPU usage. Process data is ephemeral and stored in memory only — not persisted to the database. Per-row SIGTERM and SIGKILL buttons are available directly in the UI.

Option	Description
`max_processes`	Maximum number of processes to display (default: `20`)
`require_sudo`	Prefix kill commands with `sudo` (default: `false`)
`kill_signal`	Default signal for the kill action: `TERM` or `KILL` (default: `TERM`). Per-row buttons always offer both regardless of this setting.
`cpu_warning`	Top-process CPU % that triggers `warning` (optional — omit to disable)
`cpu_threshold`	Top-process CPU % that triggers `failed` (optional — omit to disable)
`interval`	Polling frequency (default: `60`)
`ssh_config`	SSH connection details — see SSH Config below

Metrics: process_count, top_cpu_pct

- name: "Heimdall Processes"
  id: "heimdall-processes"
  type: "processes"
  interval: 30s
  max_processes: 20
  cpu_warning: 80
  cpu_threshold: 95
  ssh_config:
    host: "heimdall.example.com"

`network_usage`

Monitors network interface throughput over SSH. Takes two snapshots of /proc/net/dev one second apart in a single SSH command — no extra tools required on the remote host.

The interface to monitor can be specified explicitly or auto-detected. In auto-detect mode, Vigil picks the non-virtual, non-loopback interface with the highest cumulative byte count, ignoring interfaces with prefixes like lo, veth, docker, virbr, br-, tun, and tap.

Option	Description
`interface`	(Optional) Interface name to monitor (e.g. `eth0`). Omit to auto-detect.
`interval`	Polling frequency (default: `60`). Shorter intervals give finer-grained trend history.
`ssh_config`	SSH connection details — see SSH Config below

Metrics: rx_kbps, tx_kbps

# Auto-detect the primary interface
- name: "Heimdall Network"
  id: "heimdall-network"
  type: "network_usage"
  interval: 30s
  ssh_config:
    host: "heimdall.example.com"

# Monitor a specific interface
- name: "Ragnarok Network"
  id: "ragnarok-network"
  type: "network_usage"
  interval: 30s
  interface: "eth0"
  ssh_config:
    host: "ragnarok.example.com"

`gpu`

Monitors NVIDIA GPU utilization, VRAM usage, and temperature over SSH via a single nvidia-smi --query-gpu call. Handles multiple GPUs per host — each gets its own per-GPU metrics, and the overall status is the worst level across utilization, memory, and temperature for any GPU.

If nvidia-smi isn't installed or no NVIDIA GPU is present, the monitor reports offline rather than failed, so it degrades gracefully on mixed fleets.

Option	Description
`util_warning` / `util_threshold`	GPU utilization % bounds (default: `85` / `95`)
`mem_warning` / `mem_threshold`	VRAM usage % bounds (default: `85` / `95`)
`temp_warning` / `temp_threshold`	Temperature °C bounds (default: `80` / `90`)
`ssh_config`	SSH connection details — target must have `nvidia-smi`

Metrics: gpu_util, gpu_mem_pct, gpu_temp (busiest GPU); gpu<idx>_util, gpu<idx>_mem_pct, gpu<idx>_temp (per GPU)

- name: "GPU"
  id: "server-gpu"
  type: "gpu"
  interval: 1m
  temp_threshold: 88
  ssh_config:
    host: "server.example.com"

`containers`

Monitors Docker or Podman containers over SSH via <runtime> ps -a, counting running vs. stopped containers. Paused/created containers are treated as benign. Named containers listed in expect_running are required — any that are missing or not running drive the status to failed and expose a per-container Restart action in the UI. Other unexpectedly-stopped containers drive warning (unless stopped_warning: false).

For safety, the restart action only ever targets containers explicitly listed in expect_running.

Option	Description
`runtime`	`docker` (default) or `podman`
`expect_running`	(Optional) List of container names that must be running (→ Restart actions)
`stopped_warning`	Treat any stopped container as a warning (default: `true`)
`ssh_config`	SSH connection details — see SSH Config below

Metrics: containers_total, containers_running, containers_stopped

- name: "Docker"
  id: "server-docker"
  type: "containers"
  interval: 1m
  runtime: "docker"
  expect_running:
    - "nginx"
    - "postgres"
  ssh_config:
    host: "server.example.com"

`raid`

Monitors Linux software RAID (mdadm) array health over SSH by parsing /proc/mdstat. Each array's [N/M] [UU__] status is checked: any array with a down disk (_) or fewer active disks than expected reports failed; an array undergoing resync/recovery/reshape reports warning; all-clean reports online. Complements the ZFS plugins for hosts using classic mdraid. Reports offline when no arrays are present.

Option	Description
`interval`	Polling frequency (default: `60`; `5m` is usually plenty)
`ssh_config`	SSH connection details — see SSH Config below

Metrics: arrays_total, arrays_ok, arrays_degraded

- name: "Software RAID"
  id: "server-raid"
  type: "raid"
  interval: 5m
  ssh_config:
    host: "server.example.com"

`command`

The generic escape hatch: runs an arbitrary command over SSH and derives status from it, for checks that don't warrant a dedicated plugin. Two modes:

Exit-code mode (no pattern): exit 0 → online, non-zero → failed (or warning with nonzero_is_warning: true).
Pattern mode (pattern set): a regex with one capture group extracts a number from stdout, stored as the value metric and charted, then compared against warning/threshold — same semantics as the numeric plugins. Set invert: true when lower is worse (e.g. free space, days-until-expiry).

Every run is wrapped in timeout so a hung target can't stall the polling loop.

Option	Description
`command`	Shell command to run on the target (required)
`timeout`	Per-run timeout in seconds (default: `30`)
`pattern`	(Optional) Regex with one capture group extracting a number
`warning` / `threshold`	Value bounds (pattern mode only)
`invert`	If true, values below the bounds are bad (default: `false`)
`nonzero_is_warning`	Treat non-zero exit as warning instead of failed (default: `false`)
`value_label` / `value_unit`	UI label / unit suffix for the extracted value
`ssh_config`	SSH connection details — see SSH Config below

Metrics: exit_code (always); value (pattern mode)

# Pattern mode: TLS cert expiry, fewer days left is worse
- name: "Cert Expiry"
  id: "server-cert"
  type: "command"
  interval: 6h
  command: 'echo "days=$(( ($(date -d "$(openssl x509 -enddate -noout -in /etc/ssl/cert.pem | cut -d= -f2)" +%s) - $(date +%s)) / 86400 ))"'
  pattern: 'days=(-?\d+)'
  warning: 21
  threshold: 7
  invert: true
  value_label: "DAYS LEFT"
  value_unit: " d"
  ssh_config:
    host: "server.example.com"

# Exit-code mode: pending reboot -> warning
- name: "Reboot Required"
  id: "server-reboot"
  type: "command"
  interval: 1h
  command: "test ! -f /var/run/reboot-required"
  nonzero_is_warning: true
  ssh_config:
    host: "server.example.com"

`filesystems`

Auto-discovers and monitors every mounted filesystem on the target over SSH via a single df call — no per-path configuration. This is the fleet-wide counterpart to disk_space (which watches one explicit path). Pseudo/virtual filesystems (tmpfs, proc, cgroup, overlay, …) are excluded so only real storage appears. Overall status is the worst usage across all filesystems.

Option	Description
`warning`	Usage % that triggers warning (default: `80`)
`threshold`	Usage % that triggers failed (default: `90`)
`ssh_config`	SSH connection details — see SSH Config below

Metrics: worst_used_pct; fs_<mount>_used_pct, fs_<mount>_size_gb per filesystem

- name: "Filesystems"
  id: "server-filesystems"
  type: "filesystems"
  interval: 5m
  warning: 80
  threshold: 90
  ssh_config:
    host: "server.example.com"

`folders`

Monitors the size of arbitrary directories over SSH via du — for watching things a filesystem check can't see: a growing log directory, a download spool, a media library nearing a soft cap. Each folder may set its own warning/threshold (in GB); a folder with neither is size-only. A folder that can't be read (missing/permission/timeout) reports failed.

Option	Description
`folders`	List of `{ path, warning?, threshold? }` — warning/threshold are sizes in GB
`timeout`	Per-`du` timeout in seconds (default: `60`)
`ssh_config`	SSH connection details — see SSH Config below

Metrics: worst_folder_gb; folder_<path>_gb per folder

- name: "Folders"
  id: "server-folders"
  type: "folders"
  interval: 1h
  folders:
    - path: "/var/log"
      warning: 5
      threshold: 10
    - path: "/srv/media"   # size-only
  ssh_config:
    host: "server.example.com"

`vms`

Monitors libvirt/KVM virtual machines over SSH via virsh list --all, counting running vs. off. Domains in an error state (paused, crashed) drive warning; "shut off" is treated as benign. Named domains in expect_running are required — any not running drives status to failed and exposes per-VM Start/Shutdown actions (restricted to listed domains for safety).

Option	Description
`uri`	libvirt connection URI (default: `qemu:///system`)
`expect_running`	(Optional) Domain names that must be running (→ Start/Shutdown)
`offline_warning`	Any error-state domain => warning (default: `true`)
`ssh_config`	SSH connection details — see SSH Config below

Metrics: vms_total, vms_running, vms_stopped

- name: "Virtual Machines"
  id: "server-vms"
  type: "vms"
  interval: 1m
  expect_running:
    - "web"
  ssh_config:
    host: "server.example.com"

`cloud`

Detects the cloud provider of the target and surfaces its instance metadata (id, type, region/zone) over SSH via the link-local metadata endpoint (169.254.169.254). Auto-detects across AWS (IMDSv2), GCP, and Azure, or query one provider explicitly. Informational — no thresholds; reports online when metadata is reachable, offline when the host isn't on a recognized cloud.

Option	Description
`provider`	`auto` (default), `aws`, `gcp`, or `azure`
`ssh_config`	SSH connection details — see SSH Config below

Metrics: on_cloud (1 = on a recognized cloud, 0 = not)

- name: "Instance Metadata"
  id: "server-cloud"
  type: "cloud"
  interval: 15m
  provider: "auto"
  ssh_config:
    host: "server.example.com"

`group`

A logical container for other monitors. Aggregates the worst-case status of all descendants and displays each child as a collapsible card. Expansion state is preserved across page refreshes within the same server session.

Groups support a CSS grid layout, configurable at both the group level and per-child.

Option	Description
`children`	A list of nested plugin definitions
`grid_columns`	Number of equal-width columns in the grid (default: `1` — full-width stacked layout)

Each child entry can also set:

Child Option	Description
`grid_col_span`	How many grid columns this child occupies (default: `1`)
`grid_height`	Explicit CSS height for the child cell, e.g. `"400px"` (default: auto). Adds a scrollbar if content overflows.

Groups can be nested to arbitrary depth. Inner groups inherit their own grid_columns independently.

- name: "System Stats"
  type: "group"
  grid_columns: 3       # 3 equal columns — one subgroup per host
  children:
    - name: "Ragnarok System"
      type: "group"
      grid_columns: 4   # 4 columns — one card per stat (CPU / Mem / Temp / Load)
      children:
        - name: "Ragnarok CPU"
          type: "cpu_usage"
          ...
        - name: "Ragnarok Memory"
          type: "memory_usage"
          ...

# Child spanning multiple columns
- name: "Overview"
  type: "group"
  grid_columns: 3
  children:
    - name: "Processes"
      type: "processes"
      grid_col_span: 2   # spans 2 of 3 columns
      grid_height: "600px"
      ...
    - name: "Uptime"
      type: "uptime"
      ...

Plugin Layout

Every leaf plugin supports a layout: key that controls how its widgets are arranged on the detail page. Without a layout: block the plugin uses its built-in default grid (defined in the plugin's _DEFAULT_LAYOUT).

`layout` option	Description
`grid_columns`	Number of equal-width columns in this plugin's detail grid. Defaults vary by plugin type.

Each named widget within a plugin can be overridden:

Per-widget option	Description
`col`	Start column (1-based). Omit to use CSS auto-placement.
`row`	Start row (1-based). Omit to use CSS auto-placement.
`col_span`	How many columns this widget occupies (default: `1`).
`row_span`	How many rows this widget occupies (default: `1`).
`height`	Explicit CSS height for this cell, e.g. `"400px"`. Adds a scrollbar on overflow (default: auto).
`visible`	`false` to hide the widget entirely (default: `true`).

Widget names by plugin type:

Plugin	Widget names
`uptime`	`host_card`, `status_card`, `latency_card`, `chart`, `logs`
`systemd_service`	`host_card`, `service_card`, `status_card`, `time_card`, `logs` (continuous) / `host_card`, `service_card`, `maxage_card`, `state_card`, `history`, `logs` (oneshot)
`cpu_usage`	`host_card`, `cpu_card`, `chart`, `logs`
`memory_usage`	`host_card`, `mem_pct_card`, `mem_used_card`, `chart`, `logs`
`temperature`	`host_card`, `temp_card`, `chart`, `logs`
`load_average`	`host_card`, `load_1m_card`, `load_5m_card`, `load_15m_card`, `chart`, `logs`
`processes`	`host_card`, `count_card`, `top_cpu_card`, `table`, `logs`
`network_usage`	`host_card`, `iface_card`, `rx_card`, `tx_card`, `rx_chart`, `tx_chart`, `logs`
`smart_disk`	`host_card`, `total_card`, `ok_card`, `failed_card`, `logs`
`disk_space`	`host_card`, `path_card`, `threshold_card`, `usage_card`, `avail_card`, `total_card`, `chart`, `logs`
`zfs_health`	`host_card`, `total_card`, `ok_card`, `degraded_card`, `logs`
`zfs_pool`	`host_card`, `pool_card`, `usage_card`, `threshold_card`, `chart`, `logs`

Examples:

# Make the chart taller and hide the logs panel
- name: "Ragnarok CPU"
  type: "cpu_usage"
  layout:
    chart:
      height: "500px"
    logs:
      visible: false

# Custom 3-column grid: stat cards left, chart occupies right two columns
- name: "Heimdall Memory"
  type: "memory_usage"
  layout:
    grid_columns: 3
    host_card:
      col: 1
      row: 1
    mem_pct_card:
      col: 1
      row: 2
    mem_used_card:
      col: 1
      row: 3
    chart:
      col: 2
      row: 1
      col_span: 2
      row_span: 3
    logs:
      col: 1
      row: 4
      col_span: 3

SSH Config

All SSH-based plugins (systemd_service, smart_disk, zfs_health, disk_space, network_usage) accept an ssh_config block:

Field	Description
`host`	Remote hostname or IP address
`user`	SSH username (defaults to the current OS user if omitted)
`port`	SSH port (default: `22`)
`key_file`	Path to a private key file (uses the SSH agent / default key if omitted)

ssh_config:
  host: "myhost.example.com"
  user: "vigil"
  port: 22
  key_file: "/home/vigil/.ssh/id_ed25519"

Vigil speaks SSH natively (AsyncSSH) rather than shelling out to the system ssh client, and opens one persistent connection per host — every SSH-based monitor on that host runs its commands as a channel on that one connection rather than a separate connection each. Host key verification is trust-on-first-use: the first successful connection to a host stores its key (under $VIGIL_SSH_CONTROL_DIR/known_hosts, defaulting to a vigil-ssh directory under the system temp dir), and every later connection is checked against it — a changed key is refused rather than silently accepted.

The number of monitors you can point at one host is bounded by how many concurrent SSH sessions that host's sshd allows (MaxSessions in sshd_config, default 10). Vigil caps its own concurrency per host below that default (8 regular monitors + 2 for long-running jobs like borg, at most 10 total in flight at once), so a host running its sshd at the OpenSSH default is safe by construction — extra monitors queue rather than fail. Hosts with many monitors, or where jobs may overlap with a burst of polling, benefit from raising MaxSessions in that host's own sshd_config (e.g. MaxSessions 50) to reduce queuing.

Getting Started

Prerequisites

Python 3.9+
SSH access to target machines (SSH key auth recommended)

Installation

pip install .

Quick Start

Create a config.yaml (see Configuration below).
Start the system: vigil --config config.yaml
Open your browser to http://localhost:8080.

Configuration

Vigil uses a YAML file to define the hierarchy of your infrastructure. The YAML config is the source of truth for infrastructure definitions; SQLite is used for runtime state and overrides.

database:
  path: "vigil.db"

plugins:
  - name: "Internal Network"
    type: "group"
    children:
      - name: "Core Gateway"
        id: "gateway-ping"
        type: "uptime"
        target_host: "192.168.1.1"
        interval: 30

  - name: "Web Servers"
    type: "group"
    children:
      - name: "Nginx Service"
        type: "systemd_service"
        service_name: "nginx.service"
        ssh_config:
          host: "web-01.example.com"
          user: "vigil"

Authentication

By default the dashboard and REST API are unauthenticated — anyone who can reach the port has full read access and can trigger control actions. Set auth.username and auth.password (or auth.password_file, to keep the secret out of the YAML config) to require HTTP Basic Auth on every route:

auth:
  username: "admin"
  password_file: "/run/secrets/vigil_dashboard_password"

password_file is read once at startup. If only one of username/password is set, auth stays disabled and a warning is logged.

Usage

The primary entry point starts both the background engine and the web dashboard:

vigil --config config.yaml

To run just the dashboard against an existing database:

vigil-gui --db vigil.db --port 8080

Nix Integration

Vigil supports Flakes for reproducible environments:

# Enter dev shell
nix develop

# Run via Nix
nix run . -- --config config.yaml

Integrations

Beyond the dashboard, Vigil exposes its state to external tools. All of the following are served on the same port as the web UI.

Events feed

An Events view in the sidebar shows a unified, filterable feed of every event Vigil has recorded across all monitors — status changes, threshold crossings, and collection errors — filterable by level, target host, and message text.

REST API

Read-only JSON endpoints for consuming Vigil's state programmatically:

Endpoint	Returns
`GET /api/health`	`{"status": "ok"}`
`GET /api/monitors`	All monitors with id, name, type, target, and current status
`GET /api/monitors/{id}`	A single monitor plus its latest metrics
`GET /api/metrics`	Latest value of every collected metric
`GET /api/events`	Recent events — supports `?level=`, `?target=`, `?search=`, `?limit=`

curl http://localhost:8080/api/monitors
curl "http://localhost:8080/api/events?level=ERROR&limit=50"

Prometheus

A Prometheus exposition endpoint is always available at GET /metrics (pull) — no configuration required. It exports vigil_up (per-monitor status: 1 online, 0.5 warning, 0 failed, -1 offline) and vigil_metric (every collected metric, labeled by monitor/target/metric). Point a Prometheus scrape config at it:

scrape_configs:
  - job_name: vigil
    static_configs:
      - targets: ['vigil-host:8080']

InfluxDB

An optional push exporter ships metrics to InfluxDB (1.x or 2.x) on an interval. Enable it under exporters: in config.yaml:

exporters:
  influxdb:
    url: "http://localhost:8086"
    interval: 30
    # InfluxDB 2.x:
    org: "my-org"
    bucket: "vigil"
    token: "my-api-token"
    # InfluxDB 1.x: use `database:` instead of org/bucket/token

Design Principles

Simplicity First: Configuration should be intuitive.
No Remote Agent: All logic stays on the Vigil server; remote hosts only need SSH.
Pure Plugins: Each plugin owns its domain logic (what to collect, how to interpret it, what actions it offers) as pure functions; the engine owns all IO and persistence.
Hierarchical Organization: Supports nested groups for organizing monitors by location, service, or environment.
Fail-Safe Control: Control actions must be logged and confirmable.
Standard-Aware: Aims for OpenTelemetry compatibility in data naming and export capability.

Name		Name	Last commit message	Last commit date
Latest commit History 246 Commits
.claude		.claude
nix		nix
tests		tests
vigil		vigil
.gitignore		.gitignore
DEVELOP.md		DEVELOP.md
LICENSE		LICENSE
README.md		README.md
config.yaml		config.yaml
flake.lock		flake.lock
flake.nix		flake.nix
pyproject.toml		pyproject.toml

Folders and files

Latest commit

History

Repository files navigation

Vigil

Features

Architecture

Data Flow

Project Structure

Technical Stack

Theme

Plugin Types

Summary

uptime

push

dns_record

ddns_updater

systemd_service

service_list

smart_disk

zfs_health

disk_space

cpu_usage

memory_usage

temperature

load_average

processes

network_usage

gpu

containers

raid

command

filesystems

folders

vms

cloud

group