How do you evaluate an agent that talks to real APIs?

[francescobianco]

Unit tests do not cover it. A benchmark of fake tools does not either — the hard part is that real APIs are slow, rate-limited, stateful and occasionally down.

What does your eval loop look like? Or have you given up and gone back to watching it work?