Skip to content

Safety Guide

Jump to bottom
Nick edited this page Nov 24, 2025 · 3 revisions

PATAS Safety & Enforcement Guide

Purpose: This document explains how PATAS should be integrated into spam detection infrastructure, with clear safety boundaries and enforcement recommendations.

Enforcement & Safety Model

PATAS is designed as a high-precision spam pattern signal engine, not as a standalone enforcement system.

Core Architecture: PATAS Core is a deterministic, rule-based engine. ML/LLM integration (if used) is optional and external, running inside your infrastructure for pattern discovery.

Core Principle

PATAS provides signals and patterns. You control enforcement decisions.

We strongly recommend:

✅ Conservative Profile → Low-Risk Automatic Actions

Use the Conservative profile only for low-impact automatic actions:

  • ✅ Spam labeling (marking messages as spam for filtering)
  • ✅ Hiding / deprioritizing messages (reducing visibility without blocking)
  • ✅ Rate-limiting (temporary throttling of suspicious senders)
  • ✅ Soft actions that can be easily reversed

Metrics:

  • Ham hit rate: ≤ 1.5%
  • Spam recall: 20-40%
  • Precision: ≥ 98%

Safety: Conservative profile is safe for automatic actions because it only includes SAFE_AUTO tier patterns with very low false positive rates.

⚠️ Balanced / Aggressive Profiles → Signals Only

Use Balanced and Aggressive profiles as signals only:

  • ✅ Input features into your existing risk-scoring systems
  • ✅ Prioritization for manual review queues
  • ✅ Research / experiments for pattern discovery
  • ❌ NOT for auto-ban or irreversible actions

Metrics:

  • Balanced: 9.75% ham hit rate, 66.87% recall
  • Aggressive: 19.12% ham hit rate, 72.84% recall

Why signals only: These profiles have higher recall but also higher false positive rates. They should be combined with other signals before making enforcement decisions.

❌ Permanent / High-Impact Enforcement

Keep permanent or high-impact enforcement (account bans, global blocks, long-term penalties) under your existing systems that combine:

  • ✅ User reports
  • ✅ Account history and trust scores
  • ✅ Device / network heuristics
  • ✅ Behavioral patterns
  • ✅ PATAS signals as just one of the inputs

We do NOT recommend wiring PATAS patterns directly into irreversible enforcement without combining them with your internal signals and policies.

Safety Profiles Overview

Profile Patterns Ham Hit Rate Spam Recall Use Case Auto-Ban?
Conservative SAFE_AUTO only ≤ 1.5% 20-40% Low-impact auto actions ✅ Yes (low-risk)
Balanced SAFE_AUTO + top REVIEW_ONLY ≤ 12% ≥ 60% Signals, prioritization ❌ No
Aggressive All except FEATURE_ONLY ≤ 20% ≥ 70% Research, experiments ❌ No

Pattern Quality Tiers

SAFE_AUTO (AUTO_SAFE)

  • Precision: ≥ 95% (from shadow evaluation)
  • False Positive Rate: ≤ 1% (precision ≥ 99%)
  • Minimum hits: ≥ 10 (sufficient sample size)
  • Usage: Can be auto-promoted to active rules for Conservative profile
  • Criteria: Must pass deterministic checks AND (high precision evaluation OR whitelist patterns with AND conditions)

REVIEW_ONLY

  • Precision: 90-98% OR ham hit rate 1-5%
  • Usage: Requires manual review before promotion
  • Use in: Balanced profile (top patterns only)

FEATURE_ONLY

  • Precision: < 90% OR ham hit rate > 5%
  • Usage: ML features only, never standalone rules
  • Use in: Aggressive profile (for research)

Running Safety Evaluation

Before deploying PATAS patterns to production:

patas safety-eval

This command:

  • Evaluates all profiles against safety thresholds
  • Generates SAFETY_EVAL_REPORT.json
  • Exits with code 0 if all thresholds pass, non-zero if any threshold is violated

DO NOT DEPLOY if patas safety-eval fails.

Integration Recommendations

1. Start with Conservative Profile

  • Deploy only Conservative profile patterns initially
  • Monitor ham hit rate in production logs
  • Track user complaints about false positives

2. Gradually Add Balanced Profile

  • Use Balanced profile patterns as signals only
  • Combine with existing risk-scoring systems
  • Do NOT use Balanced patterns for auto-ban

3. Use Aggressive Profile for Research

  • Aggressive profile is for offline analysis only
  • Use for pattern discovery and experiments
  • Never use for production enforcement

Safety Guardrails in Code

PATAS includes safety guardrails enforced in code:

  • SafetyMode enum: CONSERVATIVE, BALANCED, OFF
  • Conservative mode: Only AUTO_SAFE patterns can trigger auto-ban
  • Balanced mode: Patterns produce scores/flags only, never auto-ban
  • SQL safety: Only SELECT queries, whitelisted columns, no match-everything rules
  • Rule Safety Classifier: Two-category system (AUTO_SAFE / REQUIRES_REVIEW)
    • AUTO_SAFE: Precision ≥ 95%, FPR ≤ 1%, whitelist patterns with AND conditions
    • REQUIRES_REVIEW: All other rules require manual review
  • Auto-fix SQL errors: Simple SQL errors automatically fixed in shadow evaluation
  • Stop words filtering: Common words blocked to prevent false positives

See app/v2_safety_mode.py, app/v2_sql_safety.py, and app/v2_rule_safety_classifier.py for implementation details.

Monitoring & Rollback

Continuous Monitoring

After deployment:

  1. Monitor ham hit rate in production logs
  2. Track user complaints about false positives
  3. Re-run patas safety-eval if:
    • Pattern mining discovers new patterns
    • Pattern quality thresholds change
    • Significant changes to pattern SQL generation

Rollback Procedure

If safety thresholds are violated:

  1. Immediately disable affected patterns
  2. Review SAFETY_EVAL_REPORT.json for violations
  3. Fix issues before re-enabling patterns
  4. Re-run patas safety-eval to confirm fixes

Summary

TL;DR for Engineers:

  • ✅ Use Conservative profile for low-impact auto actions (labeling, hiding, throttling)
  • ⚠️ Use Balanced profile as signals only (NOT for auto-ban)
  • ❌ Use Aggressive profile for research only (NOT for production)
  • 🔒 Keep permanent enforcement (bans, blocks) in your existing systems
  • ✅ Always run patas safety-eval before deployment
  • 📊 Monitor ham hit rates and user complaints continuously

PATAS provides the "brain" for pattern detection. You control the "trigger" for enforcement.

Last Updated: 2025-11-21

Clone this wiki locally