Skip to content

feat: log on delayed heartbeat#2994

Merged
grutt merged 3 commits intomainfrom
feat--go-sdk-hearbeat-delay-logging
Feb 11, 2026
Merged

feat: log on delayed heartbeat#2994
grutt merged 3 commits intomainfrom
feat--go-sdk-hearbeat-delay-logging

Conversation

@grutt
Copy link
Copy Markdown
Contributor

@grutt grutt commented Feb 11, 2026

Description

Adds a log warning if heartbeats are delayed due to cpu contention.

Type of change

  • New feature (non-breaking change which adds functionality)

Copilot AI review requested due to automatic review settings February 11, 2026 12:29
@vercel
Copy link
Copy Markdown

vercel Bot commented Feb 11, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
hatchet-docs Ready Ready Preview, Comment Feb 11, 2026 2:20pm

Request Review

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a warning log when worker heartbeats are sent later than expected, to help diagnose CPU contention / scheduling delays impacting worker liveness signaling.

Changes:

  • Introduces a heartbeatInterval constant in the heartbeat goroutine.
  • Logs a warning when the measured heartbeat interval exceeds a threshold (20% over expected).
  • Updates heartbeat timing to reuse the captured now timestamp for lastHeartbeat.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread pkg/client/dispatcher.go Outdated
Comment on lines +343 to +347
// detect heartbeat delays caused by CPU contention or other scheduling issues
actualInterval := now.Sub(lastHeartbeat)
if actualInterval > heartbeatInterval*6/5 {
a.l.Warn().Msgf(
"worker %s heartbeat interval delay (%s >> %s), possible CPU resource contention",
Copy link

Copilot AI Feb 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This delay warning will always trigger on the first heartbeat: lastHeartbeat is initialized 5s in the past to force an immediate send, so actualInterval will be ~5s (> 4s*6/5). Consider initializing lastHeartbeat to time.Now().UTC().Add(-heartbeatInterval) (or -heartbeatInterval-ε) or gating the warning until after the first successful heartbeat to avoid a startup false-positive.

Copilot uses AI. Check for mistakes.
@grutt grutt merged commit 6aff6b6 into main Feb 11, 2026
41 checks passed
@grutt grutt deleted the feat--go-sdk-hearbeat-delay-logging branch February 11, 2026 14:33
@promptless-for-oss
Copy link
Copy Markdown

📝 Documentation updates detected!

New suggestion: Document Go SDK heartbeat delay warnings in troubleshooting guide


Tip: Assign suggestions to team members in the Promptless dashboard to claim work 👥

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants