feat(observ): Add observability engine, plugin tool bridge, notifications#165
Conversation
…ions FlatRun now ships a native observability capability built as a plugin that runs in-process off the same agent binary. It collects per-container metrics under OpenTelemetry container semantic-convention names, keeps a bounded recent-history window for the UI, and watches container health. Self-healing only ever acts on FlatRun-managed deployments (directories under the data path). External containers that happen to share the host are read for display but never auto-restarted, and a restarted container is retried at most a few times with a cooldown between attempts, so a genuinely broken external service can no longer trigger a restart loop. Plugins can now contribute three things beyond routes: UI sections injected into the deployments and settings views, a configuration schema, and tools the AI assistant can call. Tool names are namespaced per plugin so they never collide with built-ins. A tool that mutates state is gated on write access to the target deployment, and a container-scoped action refuses to touch a container outside the deployment the session is bound to. Notifications are part of the core: admins configure delivery targets (email or webhook), send a test, and plugins emit alerts into it (for example when a container is auto-restarted). Delivery goes through shoutrrr.
Code Review SummaryThis PR introduces a robust observability and self-healing system integrated directly into the FlatRun agent. It features a plugin-based architecture using self-re-execution to simplify deployment across architectures. Key additions include a time-series metric engine, health monitoring with auto-recovery, a notification system via 🚀 Key Improvements
💡 Minor Suggestions
|
![sourceant[bot]](https://avatars.githubusercontent.com/in/1161781?s=60&v=4){kind=small}
|
|
||
| func randomToken() string { | ||
| buf := make([]byte, 24) | ||
| if _, err := cryptoRand.Read(buf); err != nil { |
There was a problem hiding this comment.
A failure in crypto/rand is extremely rare but usually indicates a catastrophic system failure. Panicking is acceptable here since the system cannot operate securely without a CSPRNG. However, ensure that this function is only called during startup and not dynamically in a way that could crash the running agent unexpectedly.
| if _, err := cryptoRand.Read(buf); err != nil { | |
| if _, err := cryptoRand.Read(buf); err != nil { | |
| log.Fatalf("critical: crypto/rand unavailable: %v", err) | |
| } |
| case strings.HasSuffix(s, "B"): | ||
| s = strings.TrimSuffix(s, "B") | ||
| } | ||
| v, _ := strconv.ParseFloat(strings.TrimSpace(s), 64) |
There was a problem hiding this comment.
The use of strconv.ParseFloat here and casting back to uint64 could lead to precision loss for extremely large values, though unlikely for container memory. For consistency and precision with integer-based byte units, consider a more robust parsing approach.
| v, _ := strconv.ParseFloat(strings.TrimSpace(s), 64) | |
| v, _ := strconv.ParseUint(strings.TrimSpace(s), 10, 64) | |
| return v * mult |
|
|
||
| func writeJSON(w http.ResponseWriter, v any) { | ||
| w.Header().Set("Content-Type", "application/json") | ||
| _ = json.NewEncoder(w).Encode(v) |
There was a problem hiding this comment.
The json.NewEncoder can fail (though rare for these types). It is better to handle the error or at least log it, especially in a monitoring/observability tool.
| _ = json.NewEncoder(w).Encode(v) | |
| if err := json.NewEncoder(w).Encode(v); err != nil { | |
| // Log error or handle appropriately | |
| } |
1 participant
Delivers the observability and self-recovery item (#148) as a native FlatRun capability. It runs as a plugin but off the same agent binary (the agent re-execs itself into a hidden plugin subcommand), so there is one binary to ship and no per-arch plugin builds.
Per-container metrics are collected under OpenTelemetry container semantic-convention names and kept in a bounded history window for the UI, so the same data can later go over OTLP to any OpenTelemetry backend without a rename.
Self-healing is the part to review. It only acts on FlatRun-managed deployments (directories under the data path): an earlier build treated every host container as managed and restart-looped an unrelated external one. External containers are now read for display and never restarted, and a managed container is retried at most three times with a cooldown before giving up.
Two new extension points:
Follow-ups tracked in #158, none blocking: