Heartbeat 未处理 rate_limited 响应，导致节点频繁掉线

[chliuqi]

问题描述

使用 --loop 模式运行 evolver 时，即使 systemd 服务正常运行，节点在 EvoMap Hub 上频繁显示离线。

根本原因分析

Hub 限流策略:

{
    "error": "rate_limited",
    "retry_after_ms": 220373,
    "policy": {
        "limit": 1,
        "window_ms": 300000
    }
}

Hub 心跳端点 5 分钟只允许 1 次请求。

evolver 启动流程:

1. sendHelloToHub() - 向 Hub 注册
2. setTimeout(5s) → sendHeartbeat() - 第一次心跳
3. setInterval(2min) → sendHeartbeat() - 定时心跳

问题所在:

• 启动时 5 秒内发送了 2 次请求，触发限流
• evolver 代码没有处理 rate_limited 响应
• 后续心跳被静默丢弃 → 节点掉线

代码位置

src/gep/a2aProtocol.js 第 473-540 行，sendHeartbeat() 函数中没有检查 data.error === 'rate_limited'。

建议修复

1. 处理 rate_limited 响应，等待 retry_after_ms 后重试
2. 跳过启动时的第一次心跳（hello 已注册）
3. 调整默认心跳间隔为 6+ 分钟

环境

• evolver 版本: v1.27.2
• Node ID: node_c822aff1ef72c803
• Hub URL: https://evomap.ai

[autogame-17]

Thanks for the detailed analysis @chliuqi. You've correctly identified the issue -- the heartbeat implementation doesn't handle rate_limited responses from the Hub.

Here's a summary of what's happening and what we plan to do:

1. Root cause confirmed: The sendHelloToHub() call at startup counts as a request against the heartbeat rate limit. When sendHeartbeat() fires 5 seconds later, it triggers the 5-minute rate limit window. Since the rate_limited response is not handled, subsequent heartbeats are silently rejected.

2. Planned fixes:

   • Handle rate_limited responses in sendHeartbeat() by reading retry_after_ms and scheduling the next attempt accordingly
   • Skip the initial 5-second heartbeat since sendHelloToHub() already confirms the node is online
   • Adjust the default heartbeat interval to respect the rate limit window (>= 5 minutes)

3. Workaround for now: If you're running v1.27.2, you can manually increase the heartbeat interval by setting the environment variable EVOLVER_HEARTBEAT_INTERVAL_MS=360000 (6 minutes) to avoid triggering the rate limit.

We'll track the fix internally and include it in an upcoming release. Thanks for reporting this with such a thorough analysis.

[autogame-17]

Hi @chliuqi, we've implemented a fix based on your analysis. The changes will be included in the next public release:

What was fixed:

1. sendHeartbeat() now detects rate_limited responses from the hub and dynamically reschedules the next heartbeat using retry_after_ms from the response. When rate limited, a warning is logged with the actual backoff time.

2. Default heartbeat interval changed from 2 minutes to 6 minutes to stay within the hub's 5-minute rate limit window (1 request per 300s).

3. The startup sequence no longer fires a heartbeat 5 seconds after sendHelloToHub(). Instead, it waits for the hello to complete, then schedules the first heartbeat at the configured interval -- avoiding the immediate rate limit trigger you identified.

4. Internally, setInterval was replaced with a setTimeout chain so the interval can be adjusted dynamically when rate limiting is detected.

Workaround for current versions (v1.27.2):

Set HEARTBEAT_INTERVAL_MS=360000 (6 minutes) in your environment to avoid triggering the rate limit with the existing code.

Thanks again for the thorough report -- it made the fix straightforward.