feat: pull-based event processing with Worker + EventHandler pattern

[rorybyrne]

Summary

Complete redesign of event processing from push-based EventListener to pull-based Worker + EventHandler pattern. Workers claim events using FOR UPDATE SKIP LOCKED, enabling concurrent processing without coordination.

Key Changes

• EventHandler base class with configuration via classvars (__routing_key__, __batch_size__, __poll_interval__, __max_retries__, __claim_timeout__)
• Simplified Worker that accepts handler_type directly and reads config from classvars
• WorkerPool.register() for handler-based registration
• Per-backend indexing via routing keys - FanOutToIndexBackends creates one IndexRecord per backend
• VectorIndexHandler (batch_size=100) and KeywordIndexHandler (batch_size=1) with routing key filtering
• Fair queuing - round-robin event fetching prevents high-volume event types from starving others
• Crash recovery - events remain claimed until explicitly marked delivered/failed
• Stale claim cleanup - periodic task resets claims older than claim_timeout

Architecture

WorkerPool
├── Worker(TriggerInitialSourceRun)     # ServerStarted → SourceRequested
├── Worker(PullFromSource)              # SourceRequested → pulls from source
├── Worker(ValidateDeposition)          # DepositionSubmitted → ValidationCompleted
├── Worker(AutoApproveCuration)         # ValidationCompleted → DepositionApproved
├── Worker(ConvertDepositionToRecord)   # DepositionApproved → RecordPublished
├── Worker(FanOutToIndexBackends)       # RecordPublished → IndexRecord per backend
├── Worker(VectorIndexHandler)          # IndexRecord[routing_key=vector]
└── Worker(KeywordIndexHandler)         # IndexRecord[routing_key=keyword]

Database Migration

Adds columns to events table:

• routing_key - for per-backend event targeting
• retry_count - for tracking retry attempts
• claimed_at - for stale claim detection
• updated_at - for state tracking

Plus partial indexes for efficient claiming.

Test Plan

• Unit tests (108 tests)
• Integration tests (15 tests for claiming and concurrency)
• Contract tests (worker lifecycle)
• Lint and type checks pass
• Manual test: start server, trigger source ingestion, verify records indexed

[rorybyrne]

Closing - will re-PR from 041 branch directly against main since PR #40 already merged the original 035 work.