Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 20 additions & 2 deletions config.example.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -128,10 +128,28 @@ sync:
github:
enabled: true
schedule: "*/15 * * * *"
# Guardrails — limit which repos reach the inbox (case-insensitive globs).
# allow_repos = allowlist (empty = all); deny_repos takes precedence.
# Guardrails (case-insensitive globs; deny wins; non-empty allow is
# fail-closed). allow_* = allowlist (empty = all pass).
# repo_name — the notification's repo full_name ("ClickHouse/nerve").
# actors — every login involved (author, assignees, commenters).
# reason — GitHub's notification reason: mention, author,
# review_requested, assign, team_mention, comment,
# subscribed, ci_activity, state_change, ...
# ci_branch — branch of a CI run (empty for everything else), so this
# one is deny-only: an allow list would fail closed and
# drop every non-CI notification.
allow_repos: [] # Example: ["ClickHouse/*", "myorg/myrepo"]
deny_repos: []
allow_actors: []
deny_actors: []
allow_reasons: []
deny_reasons: [] # Example: ["comment", "subscribed"] — mute
# follow-up churn on threads you only
# commented on / watch; keep mentions,
# review requests, and your own PRs.
deny_ci_branches: [] # Example: ["main", "master"] — keep CI
# failures on your own PR branches, drop
# default-branch syncs and scheduled runs.
github_repos: # Monitor a set of repos for NEW issues & PRs
enabled: false
schedule: "*/15 * * * *"
Expand Down
70 changes: 69 additions & 1 deletion docs/sources.md
Original file line number Diff line number Diff line change
Expand Up @@ -141,8 +141,9 @@ visible rather than failing the fetch.
- **Cursor:** ISO 8601 timestamp of the newest notification's `updated_at`
- **First run:** Fetches from the last 24 hours
- **Subsequent runs:** Uses `since=<cursor + 1s>` with `Z` suffix (not `+00:00` — the `+` in a URL query string is decoded as a space, breaking the filter)
- **Filter:** `participating=true` (assigned, review requested, mentioned)
- **Filter:** `participating=true` (assigned, review requested, mentioned). Note it never returns `ci_activity` — workflow-run notifications are not "participating", so CI never reaches the inbox through this source at all (verified against a live account: 25 `ci_activity` with `all=true`, 0 with `participating=true`)
- **Enrichment:** Each notification is enriched with actual content from the subject (PR/issue body, state, assignees, labels) and the latest comment, fetched in parallel (up to 5 concurrent `gh api` calls)
- **Metadata:** `reason`, `repo_name`, `actors` (every login involved) and `ci_branch` (branch of a CheckSuite run, `""` otherwise) — all four are guardrail-filterable, see [Guardrails](#guardrails-inbox-filtering)
- **Default schedule:** `*/15 * * * *` (every 15 min)

### GitHub Events
Expand Down Expand Up @@ -221,6 +222,9 @@ sync:
deny_repos: [] # Guardrail denylist — always dropped (takes precedence)
allow_actors: [] # Guardrail allowlist of GitHub logins — empty = all. Example: ["alice", "bob"]
deny_actors: [] # Guardrail denylist of GitHub logins — always dropped (takes precedence)
allow_reasons: [] # Guardrail allowlist of GitHub reasons — empty = all. Example: ["mention"]
deny_reasons: [] # Guardrail denylist of reasons. Example: ["comment", "subscribed"]
deny_ci_branches: [] # Drop CI runs on these branches. Example: ["main", "master"]

github_events:
enabled: true
Expand Down Expand Up @@ -315,6 +319,70 @@ default), all actors pass — behavior is unchanged. The repo and actor rules AN
| `github.allow_actors` | list | `[]` | Allowlist of GitHub login globs. Empty = all actors pass |
| `github.deny_actors` | list | `[]` | Denylist of GitHub login globs. Takes precedence over `allow_actors` |

### GitHub reason guardrail

`reason` is GitHub's own answer to "why am I being told this": `author` (you opened the
thread), `mention`, `team_mention`, `assign`, `review_requested`, `comment` (you commented
on it once), `subscribed` (you only watch it), `manual` (you subscribed by hand),
`ci_activity`, `state_change`, `security_alert`.

```yaml
sync:
github:
deny_reasons: ["comment", "subscribed", "manual", "state_change"]
```

That denylist is the compact way to say *"only threads that are mine or that call me by
name"* — it keeps `author`, `mention`, `team_mention`, `assign`, `review_requested` and
`ci_activity`. Prefer `deny_reasons` over `allow_reasons`: the allowlist is fail-closed, so
a reason GitHub adds later would be silently dropped.

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `github.allow_reasons` | list | `[]` | Allowlist of reason globs. Empty = all reasons pass |
| `github.deny_reasons` | list | `[]` | Denylist of reason globs. Takes precedence over `allow_reasons` |

### GitHub CI-branch guardrail

Keeping `ci_activity` lets in every workflow-run notification, including default-branch
runs that have nothing to do with your work — upstream syncs, scheduled cleanup jobs,
deploys. GitHub has no server-side setting for this: one workflow file serves both the
`push`-to-`main` runs and the `pull_request` runs, and the Actions notification preference
is account-wide.

CheckSuite notifications carry no `subject.url` and can't be enriched, but the branch is
right there in the title (`"CI workflow run failed for main branch"`). The source parses it
into the `ci_branch` metadata key, which is `""` for every other notification:

```yaml
sync:
github:
deny_ci_branches: ["main", "master"]
```

You keep CI failures on your own PR branches and drop the default-branch noise. This rule
is **deny-only** by design — `ci_branch` is empty for non-CI records, and a non-empty allow
list is fail-closed, so it would drop the entire feed.

**Currently inert:** the fetch uses `participating=true`, which never returns `ci_activity`
(see the GitHub adapter notes above), so no CheckSuite record reaches the guardrail today.
The rule exists for the day that filter is relaxed — without it, dropping `participating`
would flood the inbox with default-branch runs.

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `github.deny_ci_branches` | list | `[]` | Denylist of branch globs for CI runs. Empty = all CI notifications pass |

### Debugging drops

Dropped records are never persisted, so the run summary count is backed by per-record log
lines naming the rule that fired:

```
Source github: guardrail dropped 3/5 records (e.g. '[owner/repo] CI workflow run failed …')
Source github: dropped 19542… on ci_branch='main' — [owner/repo] CI workflow run failed …
```

### Extending to other sources

Adding a guardrail to another source is a config field plus one registry line. In
Expand Down
21 changes: 21 additions & 0 deletions nerve/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -1208,6 +1208,24 @@ class GitHubSyncConfig:
# matching actor is dropped before it reaches the inbox). Empty = all pass.
allow_actors: list[str] = field(default_factory=list)
deny_actors: list[str] = field(default_factory=list)
# Reason guardrails — limit which GitHub notification "reason" values reach
# the inbox, matched on the "reason" metadata key (GitHub's own reason:
# mention, author, review_requested, assign, team_mention, comment,
# subscribed, ci_activity, state_change, ...). Same semantics — case-
# insensitive globs, deny wins, non-empty allow is fail-closed. Denying
# "comment"/"subscribed" drops follow-up churn on threads you merely
# commented on or watch, while keeping mentions, review requests and
# activity on your own PRs/issues. Empty = all reasons pass.
allow_reasons: list[str] = field(default_factory=list)
deny_reasons: list[str] = field(default_factory=list)
# CI guardrail — drop workflow-run notifications for branches that aren't
# yours, matched on the "ci_branch" metadata key (the branch parsed out of
# a CheckSuite title; "" for every other notification). Denying
# "main"/"master" keeps CI failures on your own PR branches while dropping
# upstream-sync and scheduled runs on the default branch. deny only — an
# allow list is fail-closed and would drop every non-CI notification, since
# their ci_branch is empty.
deny_ci_branches: list[str] = field(default_factory=list)

@classmethod
@_coerced
Expand All @@ -1224,6 +1242,9 @@ def from_dict(cls, d: dict) -> GitHubSyncConfig:
deny_repos=d.get("deny_repos", []),
allow_actors=d.get("allow_actors", []),
deny_actors=d.get("deny_actors", []),
allow_reasons=d.get("allow_reasons", []),
deny_reasons=d.get("deny_reasons", []),
deny_ci_branches=d.get("deny_ci_branches", []),
)


Expand Down
11 changes: 11 additions & 0 deletions nerve/sources/filters.py
Original file line number Diff line number Diff line change
Expand Up @@ -126,6 +126,17 @@ def passes(self, record: SourceRecord) -> bool:
"""Return True if *record* passes all rules (should be kept)."""
return all(r.passes(record) for r in self.rules)

def rejects(self, record: SourceRecord) -> FieldRule | None:
"""The first rule that drops *record*, or None if it passes.

Dropped records are never persisted, so this is the only way to tell
*why* something vanished (see the runner's drop logging).
"""
for rule in self.rules:
if not rule.passes(record):
return rule
return None

def partition(
self, records: list[SourceRecord],
) -> tuple[list[SourceRecord], list[SourceRecord]]:
Expand Down
23 changes: 23 additions & 0 deletions nerve/sources/github.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
import asyncio
import json
import logging
import re
from datetime import datetime, timedelta, timezone
from typing import Any

Expand All @@ -29,6 +30,27 @@
# Concurrent API calls for enrichment.
_MAX_CONCURRENT_FETCHES = 5

# CheckSuite notifications carry no ``subject.url``, so they can never be
# enriched — the branch in the title is the only clue about whose run it was:
# "CI workflow run failed for main branch"
# "CI workflow run, Attempt #2 failed for chore/drop-anyio-patch branch"
# A run on the default branch is an upstream sync or a schedule, never a PR.
_CI_BRANCH_RE = re.compile(r"\bfor (?P<branch>\S+) branch$")


def _ci_branch(subject_type: str, title: str) -> str:
"""Branch of a CI run, or ``""`` for anything else.

Surfaced as the ``ci_branch`` metadata key so the inbox guardrail can deny
runs on branches that aren't yours (see :mod:`nerve.sources.filters`).
Empty for non-CheckSuite subjects and for titles we can't parse — a *deny*
rule never matches the empty string, so those records pass untouched.
"""
if subject_type != "CheckSuite":
return ""
match = _CI_BRANCH_RE.search(title or "")
return match.group("branch") if match else ""


def _collect_actors(
subject_user: str,
Expand Down Expand Up @@ -240,6 +262,7 @@ async def fetch(self, cursor: str | None, limit: int = 100) -> FetchResult:
"repo_name": repo_name,
"repo_url": repo.get("html_url", ""),
"actors": actors,
"ci_branch": _ci_branch(subject_type, subject_title),
},
))

Expand Down
13 changes: 10 additions & 3 deletions nerve/sources/registry.py
Original file line number Diff line number Diff line change
Expand Up @@ -160,12 +160,16 @@ def build_source_runners(
source = GitHubSource()
# Guardrails: restrict which repos (matched on the "repo_name" metadata
# key) and which GitHub actors (matched on the "actors" metadata key —
# every login involved in a notification) reach the inbox. The two rules
# every login involved in a notification) reach the inbox. The rules
# AND together; within each, deny wins and a non-empty allow is
# fail-closed.
# fail-closed. ci_branch is deny-only on purpose: it is empty for
# everything except CheckSuite notifications, so an allow list there
# would fail closed and drop the whole feed.
gh_filter = InboxFilter(rules=[
FieldRule(field="repo_name", allow=gh.allow_repos, deny=gh.deny_repos),
FieldRule(field="actors", allow=gh.allow_actors, deny=gh.deny_actors),
FieldRule(field="reason", allow=gh.allow_reasons, deny=gh.deny_reasons),
FieldRule(field="ci_branch", deny=gh.deny_ci_branches),
])
runners.append(SourceRunner(
source=source,
Expand All @@ -180,10 +184,13 @@ def build_source_runners(
if gh_filter.active:
logger.info(
"Registered source: github (batch=%d, guardrail: "
"repos allow=%s deny=%s; actors allow=%s deny=%s)",
"repos allow=%s deny=%s; actors allow=%s deny=%s; "
"reasons allow=%s deny=%s; ci_branches deny=%s)",
gh.batch_size,
gh.allow_repos or "*", gh.deny_repos or [],
gh.allow_actors or "*", gh.deny_actors or [],
gh.allow_reasons or "*", gh.deny_reasons or [],
gh.deny_ci_branches or [],
)
else:
logger.info("Registered source: github (batch=%d)", gh.batch_size)
Expand Down
15 changes: 15 additions & 0 deletions nerve/sources/runner.py
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,10 @@ def is_backed_off(self) -> bool:
# to a fast LLM for extraction/condensation.
_CONDENSE_THRESHOLD = 800 # chars

# Per-record guardrail drop lines emitted per run (a noisy source could
# otherwise drop hundreds at once).
_MAX_DROP_LOG_LINES = 10

_CONDENSE_PROMPT = (
"Extract the essential information from this source record content.\n"
"Rules:\n"
Expand Down Expand Up @@ -269,6 +273,17 @@ async def _run_locked(self) -> IngestResult:
self.source.source_name, dropped_count, len(records),
dropped[0].summary,
)
# Name the rule that fired — the summary alone can't tell a
# repo drop from an actor or CI-branch drop, and the record
# itself is gone after this point. Capped to keep logs sane.
for record in dropped[:_MAX_DROP_LOG_LINES]:
rule = self.inbox_filter.rejects(record)
field = rule.field if rule else "?"
logger.info(
"Source %s: dropped %s on %s=%r — %s",
self.source.source_name, record.id, field,
(record.metadata or {}).get(field), record.summary,
)
records = kept

# 2. Persist to inbox (post-preprocess, pre-condense — human-readable)
Expand Down
Loading