-
Notifications
You must be signed in to change notification settings - Fork 0
FAQ
Common questions before you file an issue.
End-to-end (comment posted → bot reply visible): 5–15 seconds for
free verbs on small threads. Paid verbs like merge add a few hundred
ms for the GitHub state-change call. The 👀 reaction lands within ~1
second so you know your mention was received.
The breakdown:
- Webhook → 202: <100ms
- Anthropic call: 2–8 seconds depending on thread size and verb
- Comment post back to GitHub: 200–500ms
Most likely causes, in order:
-
The bot didn't see a valid mention. It needs
@agenticmailwith a non-word-character before the@(start of line or whitespace).(@agenticmailtriggers,,@agenticmaildoesn't. -
Mention was in a bot-authored comment. The self-loop guard
ignores anything where
comment.user.type === "Bot". - You're rate-limited. The bot posts a 🚦 cooldown comment if you exceed 60 mentions/installation/hour. Wait for the ETA in that comment.
- You invoked a paid verb on a free plan. The bot replies with a 🔒 upgrade-required comment — check that you see one.
-
The webhook function errored. Check
/api/github/audit(operator only) for afailedentry on that delivery UUID.
No. Comment text leaves the function once, over TLS, to Anthropic's API for inference. We don't write it to any blob store.
What we do write to durable storage:
- Webhook delivery metadata: delivery UUID, event type, latency, status
- Installation metadata: account login, repos selected, install timestamp
- Per-call token counts + cost estimate (no content)
- Plan records (account login, plan name)
- Audit logs (no comment bodies)
Anthropic's API processes the content per their enterprise terms (no training on it).
Not in v1. The auto-fire on issues.opened / pull_request.opened is
the bot's main value-prop — most installations want it. v2 may add a
per-repo or per-account toggle.
In the meantime, if it's noise on a specific repo, uninstall the App
from that repo only (Settings → Apps → AgenticMail → Configure → Only select repositories).
No. merge and review are explicit — they only run when a human
types @agenticmail merge or @agenticmail review. The review verb
always posts event: COMMENT, never APPROVE or REQUEST_CHANGES,
even if you ask it to. Branch protection's "1 human approval required"
rules stay satisfied.
There's no "auto-merge when CI passes" mode either. If you want that flow, GitHub has a native auto-merge feature for that purpose.
The bot's API call may still complete (the function instance is still
running) and post a final comment, then GitHub's permission revocation
kicks in on the next call and subsequent posts return 403. We catch
those and audit them as failed. Worst case: one orphan comment with
the bot's last reply, then silence.
The App needs Contents: write permission to merge. If the App was
installed with only Issues + Pull requests + Metadata, the merge call
returns 403. Solution: the App owner needs to add Contents: write to
the App's permissions (via the App settings page), then the installation
needs to accept the new permission on the installation page.
At current claude-haiku-4-5 OAuth pricing ($1/MTok input, $5/MTok output) and typical call sizes:
-
triageon an opening-only issue: ~200–400 input + 100–200 output ≈ $0.001 -
summarizeon a 10-comment thread: ~600–1500 input + 200 output ≈ $0.002 -
summarizeon a PR with 20 changed files: ~5000–15000 input + 300 output ≈ $0.01–0.02 -
reviewon a moderate PR: ~5000 input + 800 output ≈ $0.01
Aggregate by account via GET /api/github/usage?account=<login> if
you're the operator.
Not in v1. Every call is a stateless prompt against the thread; the bot doesn't see your style guide, contributing docs, or past review patterns. v2 will support per-org system prompts that inject "Conventions:" context. For now, treat the bot as a draft-quality junior reviewer — useful for the first pass, not a substitute for a human reviewer.
LLM outputs are probabilistic. The bot has the thread context but not your codebase, your design docs, or your team's history. A few common failure modes:
-
Confident hallucination of issue numbers — the bot can mention
numbers it saw in the thread, but on long threads it sometimes
invents related issues. The
link relatedprompt explicitly tells it not to invent, but it still happens occasionally. -
Over-confident reviews on small PRs —
reviewon a 3-line PR produces a review that's longer than the change. Working on tightening the prompt. - Triage labels that don't exist in your repo — the bot suggests labels by guessing what's reasonable, not by reading your label set. Treat as advisory.
If a reply is misleading, leave a 👎 reaction on the comment. Maintainers audit those.
Yes. The function source is at
agenticmail/frontend/netlify/functions/github-webhook.mts.
Set the env vars listed in Setup, register your own GitHub App at
Settings → Developer settings → GitHub Apps, point its webhook at
your deployment, and you're running the same code as the hosted version.
| Who pays | |
|---|---|
| Anthropic inference costs | The App operator (us, for the hosted version) |
| GitHub API calls | Free for the App |
| Marketplace plan subscriptions | The customer, via GitHub |
| Hosting (Netlify) | The App operator |
| SendGrid (welcome + ops emails) | The App operator |
If you're self-hosting, you are the App operator.
github.com/agenticmail/github-app/issues.
The bot will auto-triage. Include the delivery UUID from
/api/github/audit if you have access — makes it trivially fast to
cross-reference with our function logs.
For security-related issues, email security@agenticmail.io instead.