Skip to content

[Feature]: add healthcheck-based alerting for docker containers #1647

Description

@lulusmx

Component

Hub

Describe the feature you would like to see

Implemented a trigger that monitors the Docker health_status. If a container transitions from healthy to unhealthy, an automated alert is dispatched to the configured notification channels.

Motivation / Use Case

Problem: A container can be "Running" while the application inside is crashed or unresponsive (e.g., database connection lost or API deadlock). Standard monitoring only tracks the process state, leading to "false positives" where the dashboard shows green but the service is actually down.

Solution: Implementing alerts based on Docker Health Status (healthy / unhealthy).

Key Benefits:

Detect Application Failures: Captures internal errors that don't stop the container process.

Proactive Alerting: Triggers notifications immediately when a healthcheck fails.

Accuracy: Ensures "Healthy" on the dashboard truly means the service is functional for the user.

Describe how you would like to see this feature implemented

To implement this feature effectively, the implementation should focus on the Docker Engine API integration and the Alerting Engine logic.

Here is a concise technical proposal for the implementation:

🛠️ Proposed Implementation

  1. API Monitoring Logic

State Polling: The agent should specifically poll the State.Health.Status field from the Docker inspect object, moving beyond just the State.Status (running/exited).

Event Listener: Ideally, utilize Docker events to listen for health_status: unhealthy transitions in real-time to minimize CPU overhead from polling.

  1. Alerting Engine Integration

Threshold Settings: Allow users to define a "Grace Period" (e.g., alert only after 2 consecutive unhealthy checks) to avoid noise during container startup.

Dynamic Variables: Include the container name, the specific healthcheck output (error log), and the last successful check time in the notification payload.

  1. User Interface (Dashboard)

Visual Priority: On the dashboard, a container with an unhealthy status should trigger the same visual urgency (Red color/Critical icon) as a container that is completely stopped.

Health Logs: Add a small "info" icon next to the health status to display the last 5 logs from the healthcheck command directly in the UI.

Screenshots

No response

Category

Metrics

Affected Metrics

CPU

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions