FAQ

PATAS Frequently Asked Questions (FAQ)

This FAQ addresses common questions from different user groups: product managers, engineering teams, and operations teams working with PATAS in production.

---

Product & Business Questions

How does PATAS work in production? Does it scale to billions of messages?

PATAS uses a two-stage pipeline designed for large-scale processing:

1. Stage 1 (Fast Scanning): Processes all messages with deterministic patterns (URLs, phone numbers, keywords). This is fast and catches obvious spam cases without using expensive AI models.

2. Stage 2 (Deep Analysis): Only suspicious patterns from Stage 1 get expensive semantic analysis and AI treatment. This reduces API costs by 70-90% while maintaining high-quality pattern discovery.

Performance benchmarks:

• Ingestion: ~13,000 messages/second
• Pattern mining: 100K messages in ~7 seconds (two-stage pipeline)
• Scales to millions of messages efficiently

The system discovers patterns by meaning (not just keywords), groups similar messages using DBSCAN clustering, and generates transparent SQL rules with precision/recall metrics.

How much does PATAS cost?

PATAS is designed to minimize costs through the two-stage approach:

• Stage 1: No AI costs (deterministic patterns only)
• Stage 2: Only 3-10% of messages require AI processing
• Cost savings: 70-90% reduction compared to processing all messages with AI

Cost factors:

• Embedding API calls (for semantic mining)
• LLM API calls (optional, for rule generation)
• Infrastructure (database, compute)

On-premise option: You can use local models (vLLM/TGI/Ollama) to eliminate API costs entirely. See Configuration Examples for details.

How quickly will we see results?

Initial setup to first results:

• Installation: 10-15 minutes
• Data ingestion: Depends on volume (~13K msg/s)
• First pattern mining: 7-30 seconds for 100K messages
• Rule generation: Immediate after pattern discovery

Typical timeline:

• Day 1: Setup and initial data ingestion
• Day 2-3: First patterns discovered, rules generated
• Week 1: Rules evaluated in shadow mode
• Week 2: First rules promoted to active (if metrics meet thresholds)

What if PATAS generates incorrect rules and blocks legitimate users?

PATAS has multiple safety mechanisms:

1. Shadow Evaluation: All rules are tested on historical data before activation
2. Conservative Profile: Default profile requires precision ≥ 0.95 (≤5% false positives)
3. Automatic Degradation: Rules are automatically deprecated if precision drops >10%
4. Manual Review: Rules go through candidate → shadow → active lifecycle, allowing review at each stage
5. SQL Safety: All generated SQL is validated for safety (whitelisted tables/columns, SELECT-only queries)

You can configure aggressiveness profiles (conservative, balanced, aggressive) based on your tolerance for false positives.

How does PATAS integrate with existing systems?

PATAS is API-first and designed for minimal integration effort:

• REST API: Standard HTTP endpoints for all operations
• Rule Export: Export rules in SQL, ROL (Rule Orchestrator Language), or custom formats
• Database: Works with PostgreSQL or SQLite
• No code changes required: PATAS runs as a separate service

Integration options:

1. API Integration: Call PATAS API from your application
2. Database Integration: PATAS reads from your message database
3. File Import: Import logs via CSV/JSONL files
4. Rule Export: Export generated rules to your rule engine

See API Reference for details.

Can we use PATAS on-premise? Our data cannot leave our datacenter.

Yes, PATAS supports full on-premise deployment:

• Local Embedding Models: Use BGE-M3 or similar models via vLLM/TGI/Ollama
• Local LLM Models: Use Mistral-7B or similar models for rule generation
• Privacy Mode STRICT: Disables all external API calls by default
• No data leaves your infrastructure: All processing happens locally

Recommended on-premise models:

• Embeddings: BAAI/bge-m3 (multilingual, strong performance)
• LLM: mistralai/Mistral-7B-Instruct-v0.2 (compact, Apache 2.0 license)

See Configuration Examples for on-premise setup.

How often should we run pattern mining? Is it automatic or manual?

Recommended schedule:

• Initial setup: Run once to discover patterns from historical data
• Continuous operation: Run daily or weekly depending on spam volume
• Real-time: Can be triggered via API when new spam patterns are detected

Automation options:

• CLI: Use cron jobs or scheduled tasks
• API: Trigger via HTTP requests from your monitoring system
• Event-driven: Integrate with your message processing pipeline

Pattern mining is designed to be idempotent and can be run safely multiple times.

How can we verify that rules are working? What metrics are available?

PATAS provides comprehensive metrics for each rule:

Core Metrics:

• Precision: Fraction of matches that are actually spam (spam_hits / total_hits)
• Recall: Fraction of spam messages caught (spam_hits / total_spam_count)
• F1-Score: Harmonic mean of precision and recall
• Coverage: Fraction of traffic matched by the rule
• Drift Detection: Tracks precision degradation over time

Access metrics via:

• API endpoints: /api/v1/rules/{id} with evaluation data
• Database: Query rule_evaluations table directly
• Prometheus: Metrics exported for monitoring (see Monitoring)

Rules are evaluated continuously in shadow mode before promotion to active.

What if spammers change their patterns? Does PATAS adapt automatically?

Yes, PATAS adapts automatically:

1. Continuous Monitoring: Rules are evaluated regularly on new data
2. Drift Detection: Automatic detection when rule precision drops >10%
3. Automatic Deprecation: Degraded rules are automatically deprecated
4. New Pattern Discovery: Regular pattern mining discovers new spam patterns
5. Rule Evolution: System can update rules based on new data

Recommended workflow:

• Run pattern mining weekly to discover new patterns
• Monitor rule evaluations daily
• System automatically handles rule lifecycle (promotion/deprecation)

Can we review generated rules before applying them?

Yes, PATAS provides full transparency:

Rule Lifecycle:

1. Candidate: Newly generated rules (can be reviewed)
2. Shadow: Rules tested on historical data (evaluation metrics available)
3. Active: Rules in production (monitored continuously)
4. Deprecated: Rules that degraded (automatically or manually)

Review options:

• View rules via API: /api/v1/rules?status=shadow
• Check evaluation metrics before promotion
• Manual promotion: Only promote rules that meet your criteria
• Export rules for external review

You can configure the system to require manual approval before promoting rules to active.

---

Technical & Architecture Questions

Why the two-stage pipeline? What are the specific benefits?

Two-stage pipeline benefits:

1. Cost Efficiency: 70-90% reduction in AI/LLM API costs

   • Stage 1: No AI costs (deterministic patterns)
   • Stage 2: Only 3-10% of messages require AI processing

2. Performance: Faster processing of large volumes

   • Stage 1: Fast scanning of all messages
   • Stage 2: Deep analysis only for suspicious patterns

3. Scalability: Handles millions of messages efficiently

   • Stage 1: O(n) complexity, processes in large batches
   • Stage 2: O(n log n) with DBSCAN, processes smaller batches

4. Quality: Maintains high-quality pattern discovery

   • Stage 1: Catches obvious spam patterns
   • Stage 2: Finds subtle semantic patterns

Benchmark comparison:

• Single-stage: ~2 hours for 10M messages (with AI)
• Two-stage: ~20 minutes for 10M messages (70-90% cost savings)

How does DBSCAN clustering work? Why DBSCAN instead of K-means?

DBSCAN advantages for spam pattern discovery:

1. Automatic cluster count: No need to specify number of clusters (unlike K-means)
2. Noise handling: Automatically identifies outliers (noise points)
3. Arbitrary cluster shapes: Finds clusters of any shape (not just spherical)
4. Performance: O(n log n) complexity with optimizations
5. Density-based: Groups messages by semantic similarity density

Why not K-means:

• Requires knowing number of clusters in advance
• Assumes spherical clusters
• Doesn't handle noise well
• Less suitable for semantic similarity clustering

DBSCAN parameters:

• eps: Cosine distance threshold (derived from similarity threshold, default 0.75)
• min_samples: Minimum messages per cluster (default 3)

How does PATAS handle multilingual content?

Multilingual support:

1. Embedding Models: Uses multilingual embedding models (e.g., BGE-M3) that support 100+ languages
2. Language Detection: Detects message language automatically
3. Language-Aware Clustering: Groups messages by semantic meaning across languages
4. Pattern Discovery: Finds patterns regardless of language

Supported languages:

• Primary: English, Russian, Ukrainian (well-tested)
• Extended: All languages supported by multilingual embedding models
• Quality: Best on languages with sufficient training data in embedding model

Configuration:

• Embedding model: BAAI/bge-m3 (recommended for multilingual)
• Language detection: Automatic via message metadata or text analysis

What is the database architecture? How does it scale?

Database architecture:

• ORM: SQLAlchemy 2.0 with async/await
• Supported databases: PostgreSQL (production), SQLite (development)
• Migrations: Alembic-based migrations (see Migration Guide)

Key tables:

• messages: Normalized message storage
• patterns: Discovered spam patterns
• rules: SQL blocking rules with lifecycle status
• rule_evaluations: Evaluation metrics (precision, recall, F1, coverage)

Scaling recommendations:

• PostgreSQL: Use connection pooling (default: 20 connections)
• Indexing: Automatic indexes on frequently queried columns
• Partitioning: Consider partitioning messages table by timestamp for large volumes
• Archiving: Archive old messages (>90 days) to reduce table size

Performance:

• Ingestion: ~13K messages/second
• Pattern mining: Optimized for large datasets with chunking
• Evaluation: Efficient queries with proper indexing

What models do you recommend for on-premise? Are there benchmarks?

Recommended on-premise models:

Embeddings:

• BAAI/bge-m3: Multilingual, strong performance for RU/UK/EN spam logs
• Hardware: 8GB+ RAM, GPU optional but recommended
• Performance: ~1000 embeddings/second on CPU, ~5000/second on GPU

LLM:

• mistralai/Mistral-7B-Instruct-v0.2: Compact 7B model, Apache 2.0 license
• Hardware: 16GB+ RAM, GPU recommended (24GB+ VRAM)
• Performance: ~10-20 tokens/second on GPU

Deployment options:

• vLLM: Fast inference, good for production
• TGI: HuggingFace Text Generation Inference
• Ollama: Easy setup, good for development

Benchmarks:

• See Performance Guide for detailed benchmarks
• Quality: Comparable to OpenAI models for spam detection tasks
• Cost: $0 API costs, only infrastructure costs

How does embedding caching work? Will there be memory issues?

Embedding cache strategy:

• Cache storage: In-memory cache with configurable size limits
• Cache key: Message text hash (deduplicates identical messages)
• Cache eviction: LRU (Least Recently Used) when cache is full
• Memory management: Configurable cache size to prevent OOM

Memory usage:

• Per embedding: ~1KB (1536 dimensions × 4 bytes)
• 100K messages: ~100MB cached embeddings
• 1M messages: ~1GB cached embeddings

Recommendations:

• Set cache size based on available memory
• Monitor cache hit rate (should be >80% for repeated patterns)
• Use Redis for distributed caching in multi-instance deployments

Configuration:

embedding_cache:
  max_size: 100000  # Maximum cached embeddings
  ttl_seconds: 86400  # Cache TTL (24 hours)

Is LLM optional? How much does quality drop without LLM?

LLM is optional and can be disabled:

With LLM:

• Pattern explanation and refinement
• Better rule generation for complex patterns
• Higher quality for semantic patterns

Without LLM:

• Deterministic pattern discovery still works
• Semantic clustering still works (via embeddings)
• Rule generation uses simpler heuristics
• Quality drop: ~10-15% for complex semantic patterns

Recommendation:

• Start without LLM: Use deterministic + semantic mining only
• Add LLM later: If you need better rule quality for complex patterns
• On-premise: Use local LLM if API costs are a concern

Configuration:

pattern_mining:
  use_llm: false  # Disable LLM, use embeddings only

How does SQL safety validation work? Is sqlparse sufficient?

Multi-layer SQL safety:

1. SQL Parsing: Uses sqlparse library for accurate SQL parsing
2. Whitelist Validation: Only allows queries on whitelisted tables/columns
3. Operation Restrictions: Only SELECT queries allowed (no UPDATE/DELETE/INSERT)
4. Injection Prevention: No semicolons, no subqueries, no UNION SELECT
5. Coverage Limits: Rejects rules matching >80% of messages (too broad)
6. LLM Validation: Optional LLM-based validation for additional safety

Safety checks:

• ✅ Whitelisted tables: messages, reports only
• ✅ Whitelisted columns: Only safe columns (text, is_spam, timestamp, etc.)
• ✅ No dangerous operations: DROP, DELETE, UPDATE, etc. are blocked
• ✅ No command chaining: Semicolons are rejected
• ✅ No complex queries: Subqueries and UNION are blocked

Testing:

• Comprehensive test suite for SQL safety
• Penetration testing recommended for production deployments
• See Security for details

What if LLM generates dangerous SQL? Is there additional validation?

Yes, multiple validation layers:

1. LLM Prompt Engineering: Prompts explicitly request safe SELECT queries only
2. SQL Safety Validator: All generated SQL is validated before storage
3. Whitelist Enforcement: Only whitelisted tables/columns are allowed
4. Coverage Check: Rules matching too many messages are rejected
5. Fallback: If validation fails, rule is rejected and logged

Error handling:

• Invalid SQL: Rule is rejected, error logged
• Retry: System can retry rule generation with adjusted prompts
• Fallback: Falls back to deterministic rules if LLM fails

Monitoring:

• All validation failures are logged
• Metrics track validation success rate
• Alerts can be configured for high failure rates

How are PII (Personally Identifiable Information) data handled?

PII redaction support:

Redacted data types:

• Social Security Numbers (SSN)
• Passport numbers
• Driver license numbers
• Bank account numbers
• Credit card numbers
• Phone numbers (optional)
• Email addresses (optional)

Redaction modes:

• STANDARD: Redacts PII from logs and API responses
• STRICT: Additional redaction, disables external API calls by default

OCR text redaction:

• PII in OCR-extracted text is also redacted
• Supports multiple languages and formats

Configuration:

privacy_mode: STRICT  # Enable strict PII redaction
pii_redaction:
  enabled: true
  redact_phones: true
  redact_emails: true

Compliance:

• GDPR-compliant data handling
• No PII stored in embeddings or LLM prompts (in STRICT mode)
• See Privacy and Data Protection for details

What is the throughput? How many messages per second can PATAS process?

Performance benchmarks:

Ingestion:

• Throughput: ~13,000 messages/second
• Latency: <10ms per message (batch processing)

Pattern Mining:

• 100K messages: ~7 seconds (two-stage pipeline)
• 1M messages: ~2-3 minutes
• 10M messages: ~20-30 minutes

Rule Evaluation:

• Per rule: <100ms for 100K messages
• Batch evaluation: Parallel evaluation of multiple rules

API Endpoints:

• Analyze endpoint: ~500ms for 1000 messages
• Health check: <10ms

Scaling:

• Horizontal scaling: Run multiple instances behind load balancer
• Database: Use read replicas for evaluation queries
• Caching: Embedding cache reduces redundant API calls

Hardware recommendations:

• Small (<1M messages/day): 4 CPU, 8GB RAM
• Medium (1-10M messages/day): 8 CPU, 16GB RAM
• Large (>10M messages/day): 16+ CPU, 32GB+ RAM, GPU recommended

How does PATAS behave under load? Is there rate limiting and graceful degradation?

Load handling mechanisms:

1. Rate Limiting: Configurable rate limits per API key
2. Graceful Degradation: Falls back to simpler processing when overloaded
3. Queue Management: Request queuing when system is busy
4. Circuit Breakers: Stops calling external services if they fail repeatedly
5. Timeout Handling: Configurable timeouts prevent hanging requests

Graceful degradation:

• Embedding service down: Falls back to deterministic patterns only
• LLM service down: Uses rule generation heuristics instead
• Database slow: Returns cached results when available
• High load: Prioritizes critical operations, queues non-critical

Configuration:

rate_limiting:
  enabled: true
  requests_per_minute: 1000
graceful_degradation:
  enabled: true
  fallback_to_deterministic: true

How much memory/CPU is required to process 1M messages?

Resource requirements for 1M messages:

Memory:

• Base system: ~500MB
• Message storage: ~50MB (compressed)
• Embedding cache: ~1GB (if caching enabled)
• Pattern mining: ~2GB peak (temporary during processing)
• Total: ~3-4GB recommended

CPU:

• Ingestion: 2-4 cores
• Pattern mining: 4-8 cores (benefits from parallelization)
• Evaluation: 2-4 cores
• Total: 8 cores recommended for 1M messages

Storage:

• Database: ~100MB for 1M messages
• Logs: ~50MB
• Total: ~150MB

Scaling:

• 10M messages: 16GB RAM, 16 cores
• 100M messages: 64GB+ RAM, 32+ cores, GPU recommended

Optimization tips:

• Use database connection pooling
• Enable embedding caching
• Archive old messages (>90 days)
• Use two-stage pipeline to reduce AI costs

---

Production & Operations Questions

Why do some rules have precision 0.95 but still block legitimate users?

Understanding precision:

• Precision 0.95 means 95% of matches are spam, 5% are false positives
• For high-traffic systems, 5% false positives can still be significant
• Example: 1M messages/day, 10% coverage = 100K matches, 5K false positives

Solutions:

1. Use Conservative Profile: Requires precision ≥ 0.98 (2% false positives)
2. Adjust Coverage: Rules with lower coverage have fewer false positives
3. Manual Review: Review high-coverage rules before promotion
4. Custom Thresholds: Configure custom precision thresholds per rule type

Interpreting metrics:

• Precision: Fraction of matches that are spam (higher is better)
• Recall: Fraction of spam caught (higher is better)
• F1-Score: Balance between precision and recall
• Coverage: Fraction of traffic matched (lower = fewer false positives)

Recommendation: Use precision ≥ 0.98 for high-traffic systems, precision ≥ 0.95 for low-traffic systems.

Rules are too specific - they only catch exact matches. How to find variations?

Enable semantic mining:

Semantic mining finds patterns by meaning, not exact words:

1. Enable Semantic Mining: Set enable_semantic_mining: true
2. Adjust Similarity Threshold: Lower threshold (0.70-0.75) finds more variations
3. Use Embeddings: Semantic similarity via embeddings catches synonyms and variations
4. DBSCAN Clustering: Groups semantically similar messages

Configuration:

pattern_mining:
  use_semantic: true
  semantic_similarity_threshold: 0.75  # Lower = more variations
  semantic_min_cluster_size: 3

Tuning:

• Higher threshold (0.80+): More precise, fewer variations
• Lower threshold (0.70-0.75): More variations, may include false positives
• Test and adjust: Start with 0.75, adjust based on results

Some patterns are obvious to humans but PATAS doesn't find them. Why?

Common reasons:

1. Threshold too high: Pattern doesn't meet minimum spam count threshold
2. Insufficient data: Need at least 10 spam messages for pattern (configurable)
3. Semantic mining disabled: Deterministic patterns only catch exact matches
4. Pattern too diverse: Messages vary too much for clustering

Solutions:

1. Lower thresholds: Reduce min_spam_count and min_spam_ratio
2. Enable semantic mining: Finds patterns by meaning, not exact words
3. Increase data: More historical data improves pattern discovery
4. Manual patterns: Add manual patterns for known spam types

Diagnostics:

• Check pattern mining logs for rejected patterns
• Review min_spam_count and min_spam_ratio settings
• Verify semantic mining is enabled
• Check if messages are too diverse (low similarity scores)

Rules work well on historical data but poorly on new data. How often should we retrain?

Recommended schedule:

• Initial training: Once on historical data (last 30-90 days)
• Continuous updates: Run pattern mining weekly or bi-weekly
• Rule evaluation: Daily or weekly to detect drift
• Automatic retraining: System automatically discovers new patterns

Drift detection:

• Automatic monitoring: Tracks precision over time
• Degradation threshold: Rules deprecated if precision drops >10%
• New pattern discovery: Weekly pattern mining finds new spam patterns

Configuration:

rule_lifecycle:
  shadow_evaluation:
    evaluation_window_days: 7  # Evaluate weekly
  deprecation:
    precision_drop_threshold: 0.10  # 10% drop triggers review

Best practices:

• Monitor rule evaluations weekly
• Run pattern mining weekly to discover new patterns
• Review deprecated rules to understand why they degraded
• Adjust thresholds based on spam pattern evolution

Pattern mining takes 2 hours for 10M messages. Is this normal? Can we speed it up?

Optimization strategies:

1. Use Two-Stage Pipeline: Reduces processing time by 70-90%
2. Increase Chunk Size: Larger chunks reduce overhead (default: 10K)
3. Disable LLM: LLM adds significant time, disable if not needed
4. Optimize Embeddings: Use faster embedding models or disable semantic mining
5. Parallel Processing: Run pattern mining on time windows in parallel

Expected performance:

• Without optimization: ~2 hours for 10M messages
• With two-stage: ~20-30 minutes for 10M messages
• With optimizations: ~10-15 minutes for 10M messages

Configuration:

pattern_mining:
  enable_two_stage_processing: true
  stage1_chunk_size: 10000  # Increase for faster processing
  stage2_chunk_size: 1000
  use_llm: false  # Disable if not needed
  use_semantic: true  # Keep enabled for quality

Hardware:

• CPU: More cores = faster processing (parallelization)
• GPU: Accelerates embedding generation (5-10x faster)
• Memory: More RAM allows larger chunks

Embedding generation takes too long. Can we use faster models?

Faster embedding options:

1. Smaller Models: Use smaller embedding models (faster, slightly lower quality)

   • text-embedding-3-small (OpenAI): Fast, good quality
   • all-MiniLM-L6-v2 (local): Very fast, lower quality

2. Batch Optimization: Increase batch size (default: 2048)

   • Larger batches = fewer API calls = faster processing
   • Local models: Use batch size 512-1024

3. GPU Acceleration: Use GPU for local models (5-10x faster)

   • vLLM/TGI with GPU: ~5000 embeddings/second
   • CPU only: ~1000 embeddings/second

4. Caching: Enable embedding cache to avoid regenerating embeddings

5. Disable Semantic Mining: Use deterministic patterns only (fastest)

Performance comparison:

• OpenAI text-embedding-3-small: ~2000 embeddings/second
• BGE-M3 (GPU): ~5000 embeddings/second
• BGE-M3 (CPU): ~1000 embeddings/second
• all-MiniLM-L6-v2 (CPU): ~2000 embeddings/second

Recommendation: Use GPU-accelerated BGE-M3 for best performance/quality balance.

Database is growing too fast. How much data should we keep? Can we archive old messages?

Data retention strategy:

Recommended retention:

• Messages: 90 days (sufficient for pattern discovery)
• Patterns: Keep indefinitely (small size)
• Rules: Keep indefinitely (small size)
• Evaluations: 180 days (for trend analysis)

Archiving:

• Archive messages older than 90 days to separate table or storage
• Keep message IDs and metadata for reference
• Patterns and rules reference message IDs (not full text)

Configuration:

data_retention:
  messages_days: 90
  evaluations_days: 180
  archive_enabled: true
  archive_table: "messages_archive"

Storage optimization:

• Compression: Enable database compression
• Partitioning: Partition messages table by timestamp
• Indexing: Only index frequently queried columns
• Cleanup: Regular cleanup of old evaluation data

Scripts:

• Use provided archiving scripts to move old data
• See Deployment Guide for details

System works well on 1M messages but fails on 100M. How to scale?

Horizontal scaling strategies:

1. Multiple Instances: Run multiple PATAS instances behind load balancer
2. Database Scaling: Use read replicas for evaluation queries
3. Distributed Processing: Partition data by time windows, process in parallel
4. Caching Layer: Use Redis for distributed embedding cache
5. Message Queue: Use message queue for async processing

Architecture:

Load Balancer
    ↓
[PATAS Instance 1] [PATAS Instance 2] [PATAS Instance 3]
    ↓                    ↓                    ↓
[PostgreSQL Primary] ← [PostgreSQL Replica] ← [PostgreSQL Replica]
    ↓
[Redis Cache]

Configuration:

• Connection pooling: Increase pool size for multiple instances
• Shared cache: Use Redis for distributed caching
• Partitioning: Partition pattern mining by time windows

Limitations:

• Pattern mining is currently single-instance (can be parallelized by time windows)
• Database becomes bottleneck at very large scales
• Consider data partitioning and sharding for 100M+ messages

Can we run pattern mining in parallel on different time windows?

Yes, pattern mining can be parallelized:

1. Time Window Partitioning: Split data into time windows (e.g., daily/weekly)
2. Parallel Execution: Run pattern mining on each window in parallel
3. Result Merging: Merge discovered patterns from all windows
4. Deduplication: Remove duplicate patterns across windows

Implementation:

• Use separate database connections for each parallel job
• Process different time ranges concurrently
• Merge results after all jobs complete

Example:

# Process last 30 days in 6 parallel jobs (5 days each)
windows = [
    (0, 5), (5, 10), (10, 15), (15, 20), (20, 25), (25, 30)
]
# Run pattern mining on each window in parallel

Considerations:

• Database connection limits
• Memory usage per job
• Result merging complexity
• Pattern deduplication

How to distribute load across multiple instances? Is clustering supported?

Multi-instance deployment:

Current support:

• Stateless API: API endpoints are stateless, can run multiple instances
• Shared Database: All instances share same database
• Distributed Cache: Use Redis for shared embedding cache

Not yet supported:

• Automatic clustering/discovery
• Shared state management
• Distributed pattern mining coordination

Recommended setup:

1. Load Balancer: Route requests to multiple PATAS instances
2. Shared Database: All instances connect to same PostgreSQL
3. Redis Cache: Shared embedding cache across instances
4. Session Affinity: Not required (stateless API)

Configuration:

deployment:
  instances: 3
  load_balancer: nginx  # or AWS ALB, etc.
  database: shared_postgresql
  cache: redis_cluster

Limitations:

• Pattern mining should run on single instance (or coordinate via external scheduler)
• Rule promotion should be coordinated (use database locks)

What if embedding service goes down? Does PATAS continue working?

Graceful degradation:

If embedding service is unavailable:

• Falls back to deterministic patterns: System continues with URL/keyword/phone patterns
• No semantic mining: Semantic pattern discovery is disabled
• Rule generation continues: Uses deterministic rule generation
• No data loss: All functionality except semantic mining continues

Configuration:

graceful_degradation:
  enabled: true
  fallback_to_deterministic: true
  embedding_service_timeout: 30  # seconds

Monitoring:

• Health checks detect embedding service failures
• Metrics track fallback usage
• Alerts can be configured for service failures

Recovery:

• System automatically resumes semantic mining when service recovers
• No manual intervention required

How are LLM rule generation errors handled? What if LLM returns invalid SQL?

Error handling for LLM failures:

1. SQL Validation: All generated SQL is validated before storage
2. Retry Logic: Failed generations are retried (up to 3 times by default)
3. Fallback: Falls back to deterministic rule generation if LLM fails
4. Error Logging: All failures are logged for debugging

Invalid SQL handling:

• Validation failure: Rule is rejected, error logged
• Retry: System retries with adjusted prompt
• Fallback: Uses deterministic rule generation heuristics
• No data corruption: Invalid rules are never stored

Configuration:

llm:
  max_retries: 3
  timeout_seconds: 30
  fallback_to_deterministic: true

Monitoring:

• Track LLM success/failure rates
• Alert on high failure rates
• Review error logs for pattern issues

Database crashed during pattern mining. Do we need to restart the entire process?

Idempotency and recovery:

Pattern mining is designed to be idempotent:

• Can be safely rerun multiple times
• Duplicate patterns are automatically deduplicated
• No data corruption from interrupted runs

Recovery:

1. Restart pattern mining: Simply rerun the command
2. Deduplication: System automatically handles duplicate patterns
3. No manual cleanup: No need to clean up partial results

Best practices:

• Run pattern mining in transactions (automatic rollback on failure)
• Use database backups before large pattern mining runs
• Monitor pattern mining progress via logs

Checkpointing (future enhancement):

• Planned: Save progress checkpoints during long runs
• Resume from last checkpoint if interrupted

---

Monitoring & Configuration Questions

How do we know if PATAS is working correctly? What metrics are critical?

Critical metrics to monitor:

1. Pattern Discovery Rate: Number of new patterns discovered per day/week
2. Rule Quality: Average precision/recall of active rules
3. False Positive Rate: Number of legitimate messages blocked
4. System Performance: API latency, throughput, error rates
5. Cost Metrics: LLM/embedding API costs, infrastructure costs

Prometheus metrics:

• patas_patterns_discovered_total: Total patterns discovered
• patas_rules_active: Number of active rules
• patas_rules_precision: Average precision of active rules
• patas_api_latency_seconds: API response latency
• patas_llm_calls_total: Total LLM API calls
• patas_embedding_calls_total: Total embedding API calls

Grafana dashboards:

• See Monitoring Guide for ready-made dashboards
• Custom dashboards can be created from Prometheus metrics

SLA recommendations:

• API latency: P95 < 500ms
• Pattern discovery: >10 patterns/week (depends on spam volume)
• Rule precision: >0.95 average
• System uptime: >99.9%

How to track if a new rule degraded quality? Are there automatic alerts?

Drift detection and alerting:

Automatic drift detection:

• Tracks precision over time for each rule
• Detects precision drops >10% (configurable)
• Automatically deprecates degraded rules

Alerting configuration:

monitoring:
  alerts:
    enabled: true
    precision_drop_threshold: 0.10  # Alert if precision drops 10%
    false_positive_threshold: 20  # Alert if 20+ false positives
    rule_degradation_threshold: 0.15  # Alert if precision drops 15%

Integration options:

• Prometheus Alertmanager: Configure alerts via Prometheus
• Webhooks: Send alerts to Slack, PagerDuty, etc.
• Email: Email notifications for critical alerts

Metrics:

• patas_rule_precision_drop: Precision drop percentage
• patas_rule_false_positives: Number of false positives
• patas_rules_deprecated_total: Total deprecated rules

Best practices:

• Monitor rule evaluations daily
• Set up alerts for precision drops >10%
• Review deprecated rules weekly to understand degradation causes

How much does the system cost? Is there cost tracking for LLM calls?

Cost tracking:

CostGuard module:

• Tracks all LLM and embedding API calls
• Calculates costs based on provider pricing
• Provides cost reports and alerts

Cost factors:

1. Embedding API: Cost per 1K tokens (varies by provider)
2. LLM API: Cost per 1K tokens (varies by model)
3. Infrastructure: Database, compute, storage costs
4. On-premise: Only infrastructure costs (no API costs)

Cost optimization:

• Two-stage pipeline: 70-90% cost reduction
• Embedding cache: Reduces redundant API calls
• Disable LLM: Eliminates LLM costs (slight quality trade-off)
• On-premise: Zero API costs

Monitoring:

• Daily/weekly cost reports
• Budget alerts when costs exceed thresholds
• Cost breakdown by operation (mining, evaluation, etc.)

Configuration:

cost_guard:
  enabled: true
  budget_limit: 1000  # USD per month
  alert_threshold: 0.80  # Alert at 80% of budget

How to configure thresholds for our specific use case? We have a unique spam type.

Threshold calibration guide:

Key thresholds:

1. Pattern Mining Thresholds:

   • min_spam_count: Minimum spam messages for pattern (default: 10)
   • min_spam_ratio: Minimum spam ratio (default: 0.05 = 5%)
   • semantic_similarity_threshold: Clustering threshold (default: 0.75)

2. Rule Promotion Thresholds:

   • min_precision: Minimum precision for promotion (default: 0.95)
   • max_ham_hits: Maximum false positives allowed (default: 5)
   • min_sample_size: Minimum evaluation sample (default: 100)

Calibration process:

1. Start with defaults: Use conservative defaults
2. Run pattern mining: Discover initial patterns
3. Evaluate rules: Check precision/recall metrics
4. Adjust thresholds: Lower thresholds if missing patterns, raise if too many false positives
5. Iterate: Repeat until optimal balance

Recommendations by spam type:

• Concentrated spam (same pattern repeated): Lower min_spam_count, higher min_spam_ratio
• Distributed spam (many variations): Lower semantic_similarity_threshold, enable semantic mining
• High false positive tolerance: Lower min_precision, higher max_ham_hits
• Low false positive tolerance: Higher min_precision, lower max_ham_hits

Tools:

• Use calibration scripts to find optimal thresholds
• See Configuration Guide for details

Conservative profile is too strict, but balanced is too aggressive. Can we create a custom profile?

Custom aggressiveness profiles:

Current profiles:

• Conservative: Precision ≥ 0.95, max 5 false positives
• Balanced: Precision ≥ 0.90, max 10 false positives
• Aggressive: Precision ≥ 0.85, max 20 false positives

Creating custom profile:

from app.v2_promotion import AggressivenessProfile

custom_profile = AggressivenessProfile(
    min_precision=0.92,  # Between conservative and balanced
    max_ham_hits=8,      # Between conservative and balanced
    min_coverage=0.01,
    min_sample_size=100,
)

Configuration:

rule_lifecycle:
  promotion:
    min_precision: 0.92
    max_ham_hits: 8
    min_coverage: 0.01
    min_sample_size: 100

API:

• Use promotion API with custom thresholds
• See API Reference for details

How to configure semantic similarity threshold? This affects clustering quality.

Semantic similarity threshold tuning:

Threshold ranges:

• 0.80-0.85: Very strict, only very similar messages (fewer false positives, may miss variations)
• 0.75-0.80: Balanced (recommended default)
• 0.70-0.75: More lenient, catches more variations (may include some false positives)
• <0.70: Very lenient, many variations (higher false positive risk)

Tuning process:

1. Start with 0.75: Good default balance
2. Evaluate clusters: Check if similar messages are grouped together
3. Adjust based on results:
   • Too many small clusters → Lower threshold
   • Too many false positives → Raise threshold
   • Missing variations → Lower threshold
4. Test on sample data: Validate before applying to full dataset

Tools:

• Use clustering visualization to inspect cluster quality
• Run calibration script to find optimal threshold
• See Configuration Guide for details

Recommendation: Start with 0.75, adjust ±0.05 based on results.

How to integrate with our alerting system? We need Slack/PagerDuty notifications.

Alerting integration options:

Current support:

• Prometheus Alertmanager: Native integration
• Webhooks: HTTP webhooks for custom integrations
• Metrics export: Prometheus metrics for external alerting

Webhook configuration:

monitoring:
  webhooks:
    - url: "https://hooks.slack.com/services/YOUR/WEBHOOK/URL"
      events: ["rule_degraded", "high_false_positives"]
    - url: "https://api.pagerduty.com/incidents"
      events: ["critical_errors"]

Prometheus Alertmanager:

• Configure alert rules in Prometheus
• Route alerts to Slack, PagerDuty, email, etc.
• See Monitoring Guide for examples

Future enhancements:

• Direct Slack integration (planned)
• Direct PagerDuty integration (planned)
• More webhook event types

Can we use PATAS only for analysis, without automatically applying rules?

Yes, PATAS supports analysis-only mode:

Propose-only mode:

• Generate patterns and rules
• Evaluate rules in shadow mode
• Do not automatically promote to active
• Manual review and promotion required

Configuration:

modes:
  propose_only: true  # Generate but don't auto-activate
  shadow_mode: true   # Test rules before activation
  safe_auto_ban: false  # Disable automatic promotion

Workflow:

1. Run pattern mining → Discover patterns
2. Generate rules → Rules created as "candidate"
3. Evaluate rules → Rules tested in shadow mode
4. Manual review → Review metrics and decide
5. Manual promotion → Promote rules that meet criteria

API:

• Use promotion API with auto_promote: false
• Review rules via API before promotion
• See API Reference for details

How to export rules to our custom format? We need a custom export format.

Custom rule export:

Current export formats:

• SQL: Standard SQL SELECT queries
• ROL: Rule Orchestrator Language (JSON format)

Creating custom backend:

from app.v2_rule_backend import RuleBackend

class CustomRuleBackend(RuleBackend):
    def render_rule(self, rule: Rule) -> str:
        # Convert rule to your custom format
        return custom_format_string
    
    def export_rules(self, rules: List[Rule]) -> Any:
        # Export multiple rules
        return custom_export_format

Integration:

• Register custom backend in code
• Use export API with custom backend type
• See Rule Export Guide for details

Future enhancement:

• Plugin system for custom backends (planned)

---

Additional Resources

• Quick Start Guide - Get started with PATAS
• API Reference - Complete API documentation
• Configuration Guide - Configuration options
• Deployment Guide - Production deployment
• Performance Guide - Performance optimization
• Monitoring Guide - Monitoring and alerting

For more questions or support, please open an issue on GitHub.