feat: implement monitoring system with automated tests, production ru…

[victor62-art]

…nbook, and dashboard UI

• Add 10 new comprehensive automated tests for RPC failure recovery, duplicate handling, and checkpoint persistence
• Create production runbook with deployment procedures, scaling guidelines, alert thresholds, and recovery procedures
• Implement real-time monitoring dashboard with Recharts charts, health status indicators, and action buttons
• Add migration for idempotent event processing with event signature hash uniqueness constraint
• Add storage models for DisputeResolved events with audit trail fields
• Add health check endpoint /health/sync for indexer synchronization status
• Comprehensive documentation with quick start guides and visual overviews

Tests:

• test_rpc_failure_recovery_connection_drop: Verify RPC retry logic
• test_rpc_failure_recovery_timeout: Test timeout handling
• test_checkpoint_persistence_after_failure: Verify checkpoint persistence
• test_checkpoint_resume_position: Test resume position
• test_duplicate_event_handling_same_ledger_twice: Verify duplicate prevention
• test_duplicate_event_idempotency: Test idempotency
• test_duplicate_event_signature_hash_uniqueness: Test unique constraint
• test_checkpoint_persistence_atomic_write: Verify atomic writes
• test_checkpoint_persistence_recovery_from_crash: Test crash recovery
• test_checkpoint_persistence_multiple_restarts: Test multiple restarts

Runbook:

• Docker and Kubernetes deployment procedures
• Resource limits: 2 CPU, 4GB RAM
• Horizontal scaling with ledger sharding
• Alert thresholds (critical and warning)
• Recovery procedures for all failure scenarios
• Troubleshooting guide and performance tuning

Dashboard:

• Real-time health status card with green/red indicators
• Event processing rate chart (Line, Blue)
• Error count chart (Bar, Red)
• Ledger lag over time chart (Area, Green/Red)
• Processing duration chart (Line, Orange)
• Restart Indexer button with confirmation
• Re-scan Ledger Range button with confirmation
• Auto-refresh toggle and manual refresh
• Responsive design with monospace fonts for ledger numbers

Files:

• backend/src/indexer_tests.rs: Enhanced with 10 new tests
• docs/INDEXER_RUNBOOK.md: Complete production runbook
• apps/web/components/dashboard/indexer-monitoring.tsx: Dashboard component
• apps/web/app/admin/indexer-monitoring/page.tsx: Dashboard page
• backend/migrations/20260428000003_add_event_idempotency.sql: Idempotency migration
• backend/src/storage/models.rs: Storage models
• backend/src/routes/health.rs: Added /health/sync endpoint
• backend/src/routes/mod.rs: Wired /health/sync route
• backend/src/storage/mod.rs: Exported storage models

Documentation:

• MONITORING_IMPLEMENTATION_SUMMARY.md: Implementation details
• MONITORING_DASHBOARD_QUICK_START.md: Quick reference guide
• IMPLEMENTATION_COMPLETE_MONITORING.md: Complete summary
• MONITORING_COMPONENTS_OVERVIEW.md: Visual diagrams
• DELIVERY_SUMMARY.md: Delivery status

closes #194

[drips-wave]

@victor62-art Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits