[Feature] Introduce AI-Powered Intelligent Inspection Workflow

[TJxiaobao]

Feature Request

Description

Background & Motivation

Currently, HertzBeat provides robust monitoring and alerting capabilities. However, as the system scales, O&M teams often face "alert fatigue" and the heavy burden of manual daily inspections. To evolve HertzBeat towards AIOps, we need a more proactive and intelligent way to summarize system health and diagnose potential risks.

We propose introducing an Intelligent Inspection Workflow within the hertzbeat-ai module. This feature will leverage LLMs to automate daily "system health checks," transforming raw metrics and alerts into actionable insights.

Proposed Feature: Intelligent Inspection

The Intelligent Inspection workflow is a composite Skill that orchestrates multiple atomic Tools.

Key Workflow Steps:

1. Data Harvesting: Automatically scan all monitors and active alerts (e.g., last 24h).
2. Deep Evidence Collection: For abnormal monitors, the AI automatically retrieves trend data (CPU, Memory, Latency) using existing tools.
3. AI Reasoning: The LLM performs correlation analysis (e.g., identifying if multiple alerts share a root cause) and risk assessment.
4. Report Generation: Generate a concise Markdown report summarizing health status, critical risks, and optimization suggestions.

Technical Implementation Ideas

• Module: Implement within hertzbeat-ai.
• Orchestration: Use a "Deterministic SOP + Agentic Diagnosis" hybrid approach.
• Token Optimization:
  • Funnel filtering: Only process abnormal/critical monitors via AI.
  • Data summarization: Send statistical features (max, avg, trend) instead of raw time-series data.
• Human-in-the-loop: Ensure all "Action" recommendations (like restarting a service) require manual confirmation.

Benefits

• Reduce Manual Toil: Automate the repetitive daily inspection task.
• Proactive Risk Detection: Identify "silent" risks (e.g., slow memory leaks) before they trigger critical alerts.
• Enhanced User Experience: Provide users with a high-level, intelligent overview of their entire infrastructure.

Request for Feedback

We would love to hear from the community:

1. What do you think about the "Intelligent Inspection" concept?
2. Are there specific inspection metrics or report formats you'd like to see?
3. Any suggestions on the technical architecture or token optimization strategies?

Is your feature request related to a problem? Please describe

No response

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Additional context

No response

[bigcyy]

Background

In hertzbeat-ai, the current "SOP Skill" concept is closer to an executable workflow/runbook than to an Agent Skill
in the prompt-engineering sense.

Agent Skills are usually prompt/context capabilities: they guide the model on how to use existing tools and can be
progressively disclosed only when relevant. The current HertzBeat SOP Skills, however, are YAML-defined backend
workflows with parameters, output configuration, and executable steps (tool / llm). They can be selected by the
model, but should be executed by the backend workflow engine.

This creates some conceptual ambiguity:

• "Skill" is overloaded between prompt-level Agent Skills and backend executable SOP workflows.
• SOP workflows are currently surfaced in the system prompt as a skill list.
• Execution depends on the generic executeSkill(skillName, paramsJson) tool.
• There is already a partial direction toward exposing SOPs as tools via SkillRegistry implements ToolCallbackProvider and SopToolCallback, but SopToolCallback.call() does not execute the workflow yet and the
  registry is not wired into the main chat tool provider.

Current Behavior

Current flow:

1. YAML SOP definitions live under hertzbeat-ai/src/main/resources/skills/*.yml.
2. SkillRegistry loads them into memory.
3. ChatClientProviderServiceImpl injects a summarized SOP skill list into the system prompt via
   {dynamically_injected_skills_list}.
4. The model calls executeSkill(skillName, paramsJson).
5. SkillToolsImpl finds the SOP definition and calls SopEngine.executeSync(...).
6. SopEngine executes the YAML steps.

This works, but it makes SOP workflows look like prompt-level skills, while they are actually executable backend
workflows.

Proposal

Rename and restructure SOP Skills as Workflow Tools.

Conceptually separate:

• Agent Skills: prompt-level, progressively disclosed instructions or capability packs.
• HB Tools: atomic HertzBeat tools such as query_monitors, query_realtime_metrics, create_alert_rule.
• Workflow Tools: YAML-defined executable workflows/runbooks such as daily inspection or MySQL slow query
  diagnosis.

Suggested tool groups:

agentTools
├── hbTools
│   ├── query_monitors
│   ├── add_monitor
│   ├── query_realtime_metrics
│   ├── create_alert_rule
│   └── ...
│
└── workflowTools
    ├── workflow_daily_inspection
    ├── workflow_mysql_slow_query_diagnosis
    └── ...

Each YAML workflow should become a dynamic structured tool with:

- tool name, preferably prefixed, e.g. workflow_daily_inspection
- description from the YAML workflow description
- input schema generated from YAML parameters
- execution delegated to SopEngine.executeSync(...)