Skip to content

fix(moderation): narrow the off-scope criterion to curb false positives#29

Merged
albanm merged 1 commit into
mainfrom
fix-strict-moderation
Jun 17, 2026
Merged

fix(moderation): narrow the off-scope criterion to curb false positives#29
albanm merged 1 commit into
mainfrom
fix-strict-moderation

Conversation

@albanm

@albanm albanm commented Jun 17, 2026

Copy link
Copy Markdown
Member

Rework the moderation gate's system prompt so its off-scope criterion
targets genuine misuse rather than any "heavy task."

What changed:

  • The old rule blocked "a heavy task clearly unrelated to a data platform
    (e.g. generate a large program)", which over-fired on legitimate
    detailed/technical work — notably sub-agent task delegations.
  • The off-scope criterion now targets using the assistant as a free
    general-purpose tool unrelated to the platform's data (general chatbot
    use, essays, substantial general-purpose software).
  • Data exploration/analysis/visualization, working with dataset/file
    content, small scripts that consume the platform's data or API, and
    delegated sub-agent tasks stay in scope even when technical. Writing
    general-purpose code stays out; the ambiguous middle resolves to allow.
  • Updates the unit test (asserts the in/out-of-scope wording) and the
    architecture doc.

Why: false-positive blocks were hitting sub-agents running targeted
technical/data tasks; the gate was too strict on legitimate detailed work.

Regression risks:

  • The prompt is shared by every moderated org and the summary endpoint, so
    this widens the allow-set platform-wide, not just for sub-agent traffic.
  • The classifier sees only message text and can't verify sub-agent
    provenance (modelRole is attacker-controlled, so it intentionally doesn't
    relax the policy). The small-script-vs-general-software line is a
    text-only judgment — expect some leakage in both directions.
  • Mock-based API/e2e tests ignore the prompt, so the behavioral change is
    not covered by CI; only the prompt wording is asserted.

The off-scope rule ("a heavy task clearly unrelated to a data platform,
e.g. generate a large program") over-fired on legitimate detailed/technical
work, notably sub-agent task delegations, blocking them as off-platform.

Reframe the moderation prompt around the actual abuse target: using the
assistant as a free general-purpose tool unrelated to the platform's data.
Data exploration/analysis/visualization, working with dataset/file content,
small scripts that consume the platform's data or API, and delegated
sub-agent tasks now stay in scope even when technical; writing substantial
general-purpose software stays out; the ambiguous middle resolves to allow.

Update the unit test and architecture doc to match.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@github-actions github-actions Bot added the fix label Jun 17, 2026
@albanm albanm merged commit 19f3721 into main Jun 17, 2026
4 checks passed
@albanm albanm deleted the fix-strict-moderation branch June 17, 2026 20:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant