Skip to content

🔍 v0.9.7 - Operation Health Monitoring (Schema Introspection Alternative)

Choose a tag to compare

View all tags
@keithah keithah released this 15 Sep 22:48
· 31 commits to main since this release
v0.9.7
30b50c7

🔍 Operation Health Monitoring

Problem Solved: Schema Introspection Disabled

During implementation of schema monitoring, we discovered that MonarchMoney disables GraphQL introspection for non-admin users. This release provides an even better solution: operation-based health monitoring.

🚀 New Operation Health Monitoring

Why This Approach is Better

  • ✅ Tests Real Operations: Validates what users actually use
  • ✅ Detects Actual Failures: Finds operations that are broken for users
  • ✅ No Admin Access Needed: Works with regular user accounts
  • ✅ Comprehensive Analysis: Categorizes schema vs business logic errors

Core Features

🤖 Automated Daily Monitoring

  • GitHub Actions Workflow: Runs daily at 6 AM UTC
  • Automatic Issue Creation: Creates detailed GitHub issues for problems
  • Health Artifacts: Stores comprehensive health check results

🧪 Operation Testing

# Manual health check
python tools/operation_health_checker.py --email EMAIL --password PASS --mfa-secret SECRET

# Quick check
python tools/operation_health_checker.py --email EMAIL --password PASS --quick

📊 Comprehensive Analysis

The system tests key operations across all services:

  • AccountService: get_accounts
  • TransactionService: get_transaction_categories, get_transactions
  • InvestmentService: get_security_details
  • BudgetService: get_budgets

🔍 Smart Error Classification

Schema Errors (Breaking):

  • "Something went wrong while processing"
  • "Cannot query field"
  • "Unknown field"

Business Logic Errors (Expected):

  • Authentication failures
  • Invalid parameters
  • Empty results

📈 Detailed Reporting

GitHub Issues Include:

  • Overall health status (Healthy/Degraded/Unhealthy)
  • Success rate percentage
  • Schema error count and details
  • Field availability changes
  • Service-by-service breakdown
  • Actionable recommendations

Health Check Artifacts:

  • JSON results with full details
  • Markdown reports for humans
  • Historical tracking data

🎯 Real-World Discovery

The first health check run immediately discovered actual issues:

🚨 Schema Errors Found:

  • TransactionService.get_transaction_categories: "Something went wrong" error
  • BudgetService.get_budgets: "Something went wrong" error

⚠️ Field Changes Detected:

  • AccountService.get_accounts: Missing expected fields
  • InvestmentService.get_security_details: Missing expected fields

This proves the approach works and provides immediate value by detecting real problems affecting users.

🔄 Integration with Robust Operations

This monitoring system works perfectly with the robust GraphQL operations from v0.9.6:

  1. Health checks detect issues → Automatic GitHub issues created
  2. Developers update robust operations → Field specifications adjusted
  3. Operations adapt automatically → Graceful handling of schema changes
  4. Users experience reliability → No more "Something went wrong" errors

📋 Usage Examples

Daily Automated Monitoring

The workflow runs automatically and will:

  • Test all critical operations
  • Create GitHub issues for problems
  • Store detailed health reports
  • Track changes over time

Manual Health Checks

# Full health check with reports
python tools/operation_health_checker.py \\
  --email your-email@example.com \\
  --password your-password \\
  --mfa-secret YOUR_MFA_SECRET \\
  --output ./health_reports

# Exit codes indicate health status:
# 0 = Healthy, 1 = Unhealthy, 2 = Degraded

CI/CD Integration

- name: Check API Health
  run: |
    python tools/operation_health_checker.py \\
      --email ${{ secrets.MM_EMAIL }} \\
      --password ${{ secrets.MM_PASSWORD }} \\
      --mfa-secret ${{ secrets.MM_MFA_SECRET }}

🛡️ Monitoring Benefits

  • Proactive Detection: Find issues before users report them
  • Comprehensive Coverage: Test all critical user operations
  • Automatic Notifications: GitHub issues with detailed analysis
  • Historical Tracking: Monitor API stability over time
  • Zero Admin Requirements: Works with any MonarchMoney account

🔧 Technical Implementation

Operation Health Checker (tools/operation_health_checker.py):

  • Comprehensive CLI tool for manual health checks
  • Detailed reporting in JSON and Markdown formats
  • Configurable test coverage and output options

GitHub Actions Workflow (.github/workflows/operation-health-check.yml):

  • Daily automated health monitoring
  • MFA authentication support
  • Artifact storage and issue creation
  • Slack integration (optional)

📖 What's Next

This monitoring system provides the foundation for:

  • Trend Analysis: Track API stability over time
  • Performance Monitoring: Measure operation response times
  • Change Detection: Identify schema modifications immediately
  • Reliability Metrics: Generate uptime and success rate reports

Note: This approach turned out to be more valuable than traditional schema introspection because it tests the actual user experience rather than just the schema definition.

Full Changelog: v0.9.6...v0.9.7

Assets 2
Loading