Monitoring

How monitoring works

Polaris polls devices on a cadence you set, over a transport you choose, per stream. Nothing about that is a single global setting — it resolves from a hierarchy, per stream, per asset.

Companion pages: Polling methods · Monitor states and down detection · Dependency suppression · Maintenance windows.

---

The eight streams

Stream	Collects
responseTime	is it answering, and how fast — the liveness probe
cpuMemory	CPU and memory utilisation. Two transports also carry per-core CPU and a memory breakdown, and the System tab splits CPU & Memory into two charts on those (an Unraid / TrueNAS host splits too: per-core CPU, and memory as used against installed RAM; so does a Proxmox VE node: one CPU line, since Proxmox publishes no per-core figures, and memory as Used / ZFS ARC / free): the Polaris Agent (per logical core; processes / buffers / cache / free, plus swap or page file, as the guest OS accounts for it) and vCenter (per vCPU or physical core; the hypervisor's own bands — ballooned and host-swapped among them). FortiOS, SNMP, WinRM and SSH report one CPU figure and one memory figure and keep the combined chart
temperature	hardware sensors and their alarm bits
interfaces	per-port state, counters, PoE, IP, LLDP-adjacent data
lldp	LLDP neighbours
storage	filesystems and mounts
processes	the process / service inventory, plus CPU/RAM for pinned ones
eventLog	OS event log / journal entries, curated into the audit log

Each has its own cadence, its own queue and its own worker pool. A slow stream cannot starve a fast one.

Two other things on an asset are deliberately not streams, so they have no polling method of their own: path checks, which the Polaris Agent (or the Polaris server) runs on each check's own interval and which never move the host's state, and the installed software list, which the agent re-reads every six hours and Intune / Azure Arc supply on their discovery runs.

SD-WAN on a FortiGate is not a stream of its own. It is read only where the interfaces stream resolves to FortiOS REST, but it has its own cadence, queue and worker pool. The interval is set per integration, 60 seconds by default (see Integration-Fortinet), not by the interface scrape.

---

What "monitored" means

Asset.monitored is the operator's statement that Polaris should poll this device. It gates:

• whether the device is polled at all;
• whether it appears in the dashboard's fleet counts;
• whether any automation may fire about it (rule 37).

Four statuses cannot carry monitoring — decommissioned, disabled, storage, quarantined (rule 10). Staging one of them clears the flag in every write path.

quarantined is on that list because a quarantined device is isolated at the FortiGate, so every probe fails by design — monitoring it would turn a security action into an outage alert storm about the isolation working. Because quarantine is reversible, the flag is parked and restored on release, so releasing a quarantine hands the device back with monitoring still on.

maintenance is deliberately not on the list: a window pauses polling while monitored keeps your intent, so it survives the window.

---

Cadence

Each stream's interval resolves from the same hierarchy the polling method does — per asset → class override → integration → manual tier → hardcoded floor. The figure the monitor-settings card reports is the figure you get.

No clamp is applied on top — not even dependency suppression: a device whose parent is dark keeps its response-time probe at the normal interval (Dependency suppression).

What a probe can decline to be

A probe can return skipped, which writes no sample and moves no counter — it bumps the cadence anchor alone, so spacing is preserved. The sources:

Source	Why
vCenter unreachable	the thing that answers for the device is not the device — one vCenter outage must not down a virtual fleet
FortiManager unreachable	same reasoning, for the fortimanager method
Unraid / TrueNAS host API unreachable	same reasoning, for the VMs and containers on the unraid / truenas method (the host itself fails — its own API is the finding)
Proxmox VE API unreachable	the same, when no configured node address answers: guests on the proxmox method are skipped, nodes on it fail. A node its peers report offline is not a skip — the cluster answered, and the node probes down
A workload mid-transition	an Unraid / TrueNAS state such as TrueNAS's DEPLOYING says neither up nor down
responseTimePolling = "disabled"	the operator saying do not poll this

A skip is not a miss, so N consecutive misses still means N times the transport actually answered nothing about a device it could reach.

A re-queued job is not a second reading

The probe queue lets one job queue behind an active job with the same key, and a batched ICMP chunk can outlive the 5-second publisher tick, so the same poll can be queued twice, seconds apart. Jobs carry their resolved interval and the worker re-runs the due check at pickup; a duplicate records nothing at all, so a cycle never yields two samples, or two misses toward the threshold.

---

Transports, and which ones batch

Transport	Batching
ICMP	batched — due assets are published as chunks and pinged through one fping per distinct timeout, one echo each
SNMP	one job per asset — OIDs batch within a session, never across hosts
SSH / WinRM	one connection per host per tick; interfaces and storage ride the same one
REST (FortiOS)	one per asset, except two cross-device caches
vCenter	two warm caches, one SOAP round trip per integration per tick
Unraid / TrueNAS	one cached read of the host's API per integration per 30 s answers every asset on the host
Proxmox VE	one cached read of the cluster per integration per 30 s answers every node and guest; per-guest details come from discovery, and storage is re-read at most every 5 minutes
Agent	the agent pushes on its own schedule; a deployed agent that goes quiet counts as missed polls (Monitor states)

The cross-device batchers matter at scale: one FortiManager /dvmdb read serves every gate it manages, the vCenter warm caches answer four streams for every VM and host in one fetch, and one NAS read answers every container on it.

Reading timestamps correctly

An ICMP chunk runs until its slowest target times out, which can be a whole publisher tick. Each reading is recorded at the instant the echoes were sent, not when the chunk finished.

---

Vendor SNMP telemetry needs two things you supply

Standard SNMP covers a lot — interfaces, LLDP, PoE, bridge and VLAN tables, hrStorage, ENTITY sensors — and Polaris bakes the IETF and IEEE MIBs those live in into the product, so they work the moment SNMP does. Those are fixed: they cannot be edited or deleted, and they change when you update Polaris.

CPU, memory, hardware sensors and flash on network gear are different. Almost every vendor publishes them only in its own MIB. Polaris ships Cisco's — but into the MIB Database rather than baked in, so they sit alongside anything you upload and you can delete them, and only on a fresh install; an upgrade leaves your MIB Database exactly as you curated it. For every other vendor, two pieces have to be in place:

1. The vendor's MIB, uploaded at Server Settings → Credentials → MIB Database. This supplies the OID number behind a symbol name. Keeping it as an upload rather than baking it in means a vendor adding or moving an object is a MIB upload, not a wait for a Polaris release. A MIB file can be up to 6 MiB, room for a vendor that ships one large module (APC / Schneider Electric's PowerNet-MIB is about 3 MB).
2. A manufacturer profile, at Server Settings → Credentials → Manufacturer Profiles. This says which symbol is the CPU, which pair is the memory, and so on.

A profile is an override of what the generic MIBs already do, so Polaris ships one only where there is something to override, and only for a manufacturer whose MIB it also ships. MikroTik is the illustrative counter-case: RouterOS reports CPU, memory and storage through HOST-RESOURCES-MIB, which Polaris already reads, so there is nothing for a profile to override and none is shipped. A vendor the generic MIBs already answer needs neither a profile nor its MIB.

For anything else you create the profile once. The symbol names for the common vendors, and where to get each MIB, are in the install guide's "A vendor's SNMP CPU / memory / storage is empty" section — and the shipped Cisco profile is a worked example to copy, which is half of why it ships.

The Transform column only appears where Polaris applies it. Today that is a scalar Temperature row, which offers Tenths → Units for a sensor that reports tenths of a degree (MikroTik's temperature objects report 315 for 31.5 °C). Other metric rows show "—" instead of a dropdown. Double-scalar rows still offer their combiner, because it says which two readings the row's symbols are. Custom widgets offer the full list. Polaris always stores and alerts in Celsius; the install's temperature-unit setting converts only what you see.

Both shipped pieces are yours to delete. Removing a shipped MIB leaves its profile's rows reading unresolved and names the module to re-upload; removing a shipped profile falls back to the generic MIBs. Either way the device keeps being monitored — you lose the vendor-specific figures, not the monitoring.

Three things worth knowing when you build one:

• FortiGate MIBs come from the FortiGate. System → SNMP has download links for the FortiGate and Fortinet Core MIB files, so no support account is needed. FortiSwitch and FortiAP MIBs are support-portal downloads.
• Scope a row by device type, not just by model. A row keyed on a model regex only matches an asset whose model actually states the family, and FortiSwitches often arrive with an empty model. Adding the same row again scoped to a device type covers those. Where both exist, a stated model wins — so an asset that discovery typed wrongly still routes by what it says it is.
• A pattern that repeats a repeat is refused on save. The profile's "also applies when" pattern, a row's model pattern and a model parse pattern run against every device's SNMP text on every poll, and a pattern such as (.+)+ can take effectively forever on some inputs — stalling monitoring rather than slowing it. Polaris refuses any pattern that puts +, * or {n,} on a group already containing one, and names the part to rewrite. The check is deliberately cautious, so it also refuses a few patterns that would have been fine ((v\d+)+); write those without the outer repeat.

A profile row that cannot resolve says so: its MIB cell reads unresolved and names the module still missing, the profile header shows N UNRESOLVED, and one warning Event per profile appears in the Events log at startup. Nothing needs restarting once a MIB lands — collection resumes on the next cadence.

---

Packet loss is a separate measurement

A uniform burst of 5 echoes at every eligible asset each cycle — down and dependency-suppressed devices included — batched through fping (one process per 500 targets), falling back to per-host ping bursts where fping is not available.

Two things about it (rule 30):

• It never records a probe result. ICMP does not authenticate the device it reaches, so the sweep informs the loss ratio and can never move monitorStatus.
• It is uniform — every eligible asset, every cycle, not only while an asset looks unhealthy, which would itself be a sampling bias.

"fping not available" covers not installed and installed-but-unexecutable: on Debian, fping and ping carry cap_net_raw=ep, so a process without CAP_NET_RAW fails at exec.

---

Credentials

A polling method needing authentication walks four tiers, most specific first:

per-stream asset credential
  → the asset's generic monitor credential
    → the class override's per-stream credential
      → the integration's fallback

The integration fallback is: FortiManager / FortiGate firewalls read the integration's SNMP credential or its stored API token; AD-discovered hosts read the integration's bind DN and password.

The asset's credential picker is type-agnostic — you pair it with the polling method you chose. A mismatch is allowed and falls through to a clear error at probe time ("No WinRM credential selected"), rather than being refused at save.

Per-asset FortiOS REST credentials

An integration's API token is fleet-wide by construction — the field's own hint says it must be the same across all managed FortiGates. For a fleet where each gate carries its own api-user, a restapi-typed credential selected on an asset's stream is preferred. It contributes authentication, port and TLS verification only — the target stays the asset's own address, so one credential mistakenly selected on twenty gates cannot poll one device and file its CPU under all twenty.

A stored credential stays in force when you set the stream back to "Inherit". It is applied whenever the inherited method takes that credential type. The modal therefore shows the Credential picker on any stream that stores one, with a note saying so, even on "Inherit". To stop using it, choose Source default. Hiding the picker on "Inherit" would make the stream read as "nothing set" while Polaris kept sending the stored token, and a stale one can lock the server out of that gate's API (rule 96).

How Polaris paces FortiOS REST calls

FortiOS 7.6 locks a source IP out of API-key access after a few failed authorizations (admin-lockout-threshold, default 3), and doubles the lockout each time it repeats. Fortinet counts every non-OK answer as a failure, including transient ones during an HA transition or an upgrade. Everything Polaris does to a gate comes from one source IP, so a lockout stops discovery, monitoring, pushes and the Query API tool together. To stay clear of it (rule 96):

• At most 2 requests are in flight to one gate at a time (POLARIS_FORTIOS_PER_GATE_CONCURRENCY, per Polaris process). One bad moment then costs at most two failures, not a burst that trips the threshold.
• After a 401, Polaris sends that gate nothing for a while: 60 seconds, doubling on each consecutive 401 up to 30 minutes, and ending at the first answer that is not a 401. Requests during the pause fail at once with "Polaris has paused requests to this FortiGate" and never reach the gate.
• A probe Polaris did not send is not a missed poll. A paused gate's REST response-time probe, and the controller probe for the switches and APs behind it, are recorded as skipped. A token problem does not mark the gate and its whole site down.

---

Compatibility vs capability

Two different questions, and Polaris checks both:

• Compatibility — is this method meaningful for this source kind?
• Capability — does the collector actually exist?

A method with no collector would leave a stream in permanent quiet — resolving fine and collecting nothing while the tick records success. So the validators warn (not refuse — a 400 would punish re-saving an existing value), the dropdowns stop offering the combination, and an audit names what is already stored. One case is a hard refusal: ICMP on a non-response-time stream via the per-asset route.

What has no collector today:

Method	Not collected
icmp	anything but response time
ssh / winrm	temperature, LLDP
snmp	processes (declared, not implemented), event log
agent	LLDP
rest_api on a FortiGate	storage, processes, event log
rest_api on a managed FortiSwitch / FortiAP	anything but response time — plus CPU / memory and temperature on an AP, which ride the parent gate's controller table
rest_api outside Fortinet	anything but response time

---

Sub-asset pinning

Interfaces, IPsec tunnels and storage mounts each have a pin array on the asset. The pin is a gate on alerting, never a side effect of retention (rule 57) — every stream writes samples for unpinned members too, permanently inside the engine's lookback.

An unpinned member never alerts, and there is no opt-out. Un-pinning is how alerting stops.

Storage is where this matters most: a device reports every filesystem it has, so without the pin gate a fleet-wide "disk over 90 %" automation would alert on removable volumes, ISO mounts, recovery partitions, mapped network drives and archive shares that are full by design.

Pin from the asset's own tabs, from an integration's auto-monitor pass (additive only), or in bulk from Assets → Settings → Mass Pinning (which can also unpin).

---

Samples, rollups and retention

Samples land in TimescaleDB hypertables and are rolled up hourly and daily. Retention is tiered and configured at Server Settings → Retention.

Prune is by drop_chunks, never by row delete — sample tables carry no foreign key to Asset and are never row-updated where a row could sit in a compressed chunk. That is why deleting an asset does not cascade into its history.

Server Settings → Maintenance reports a capacity snapshot, a steady-state size projection and a disk forecast, all measured on your install.

---

Observability

Polaris exports Prometheus metrics at GET /metrics, gated by METRICS_TOKEN when set and additionally by an nginx allow block in production. Grafana dashboards ship in docs/grafana/.

GET /health answers 200 whenever the process is up and checks nothing at all — the setup wizard polls it before a database exists, so a database check here would deadlock provisioning.

GET /health/ready is the one a load balancer should monitor: 200 only when the local PostgreSQL is a writable primary, otherwise 503 with a reason (in-recovery, db-error, timeout). It runs on its own one-connection pool over the direct database URL, never the application pool — a saturated pool must not be able to flap a healthy site out of a load balancer.