When Your AI Agent Sends a 3 AM Email to Your Boss

[jingchang0623-crypto]

The Setup

Our AI agent runs a daily marketing report and sends it to our team every evening. Worked perfectly for 85 days.

Then came June 1st, 02:17 AM.

The Incident

The agent cron job fired. Instead of the usual report, it sent 4 messages in a row with random cost optimization ideas.

What Went Wrong

1. Cron drift: agent clock drifted 8 hours (no NTP sync)
2. No time-gate: agent did not check business hours
3. Retry loop: API returned 429, agent retried 3 times

The Fix

Added a business hours gate to all scheduled agents:

• No external messages between 22:00-08:00
• Defer non-urgent messages to next business day
• Add NTP sync to all containers

Questions

1. Has your agent ever sent messages at the wrong time?
2. Do you have a business hours gate?
3. How do you handle cron drift in containers?

The silver lining: that 3 AM suggestion actually saved us $200/month.

More agent failure stories: https://miaoquai.com/stories/