Skip to content

feat(signals): Agentic report generation (repo discovery -> signals research -> actionability -> priority -> summary)#51408

Open
sortafreel wants to merge 111 commits intomasterfrom
signals/report-generation-agent
Open

feat(signals): Agentic report generation (repo discovery -> signals research -> actionability -> priority -> summary)#51408
sortafreel wants to merge 111 commits intomasterfrom
signals/report-generation-agent

Conversation

@sortafreel
Copy link
Copy Markdown
Contributor

@sortafreel sortafreel commented Mar 17, 2026

Problem

  • Signal reports were summarized by a single LLM call (title + summary), then judged for actionability, with no ability to investigate the underlying codebase, production data, or verify signal claims before deciding what to do.

How to make it work

  • Ensure you can run local sandboxes (products/tasks/backend/temporal/process_task/SETUP_GUIDE.md)
  • Ensure you have signals-agentic-report-generation FF enabled for the team (or make signals_agentic_report_gate_activity return True when testing)

Changes

  • Add agentic report research flow behind a feature flag: after the safety judge passes, a sandbox agent clones the relevant repo, investigates each signal against the actual code/data (via MCP tools + gh CLI), and produces per-signal findings, and per-report actionability, priority, and a title/summary
  • Add repository selection step — picks the most relevant repo from the team's GitHub integrations (single repo: direct, N repos: sandbox agent explores candidates)
  • Support re-promotion of ready reports: when enough new signals accumulate, the workflow re-runs and reuses the previous repo selection + lightly validates previous findings instead of re-researching from scratch
  • Add local testing harnesses (analyze_report, select_repo, parse_sandbox_log) for exercising the flow against synthetic signals — intended to be reworked into evals

How did you test this code?

Repo selection

CleanShot 2026-03-18 at 18 22 14

Report generation

CleanShot 2026-03-16 at 16 12 53

👉 Stay up-to-date with PostHog coding conventions for a smoother review.

Publish to changelog?

Docs update

@sortafreel sortafreel requested review from a team, joshsny and tatoalo March 21, 2026 06:26
try:
task_run = TaskRun.objects.select_related("task__created_by").get(id=input.run_id)
except TaskRun.DoesNotExist:
error_msg = "Task run not found"
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tatoalo @joshsny Pinged you for a review as I did a change to follow-ups (raise when failing), plus added a generic custom_prompt_multi_turn_runner.py. Don't expect you to check the whole thing :)

Copy link
Copy Markdown
Member

Twixes commented Mar 24, 2026

This stack of pull requests is managed by Graphite. Learn more about stacking.

Copy link
Copy Markdown
Member

@Twixes Twixes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't upset me

Comment on lines +75 to +80
@field_validator("explanation")
@classmethod
def explanation_must_not_be_empty(cls, v: str) -> str:
if not v.strip():
raise ValueError("Explanation must not be empty")
return v
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like a complex way of doing min_length=1

Comment on lines +34 to +39
membership = (
OrganizationMembership.objects.select_related("user")
.filter(organization=team.organization)
.order_by("id")
.first()
)
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, this will be completely false, we should have a "system" user instead

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please do test commands in a stacked PR in the future, as this really inflates the mental overhead of the PR vs. just its core functionality

Comment on lines +18 to +19
# Small public repo to copy into the sandbox to init. Could be removed later when it's possible to create sandboxes without repos.
REPO_SELECTION_DUMMY_REPOSITORY = "PostHog/.github"
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is really dumb and a waste of time, but I get that that the sandbox doesn't yet allow not cloning a repo

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants