fix(billing): implement chunked updates for free tier usage tracking

[FroeMic]

Summary

Fixes GTM-1500: High DB load on free tier usage tracking job

Implements Option 2: Transaction-based chunking (1000 orgs per batch)

• Reduces query count from 50,000 to ~50 (99% reduction)
• Structured for easy swap to raw SQL (Option 1) if needed

Changes

• thresholdProcessing.ts:

  • Added OrgUpdateData type for collecting update data
  • Refactored processThresholds() to return update data instead of executing immediately
  • Removed 3 prisma.organization.update() calls
  • Kept all existing logic (emails, metrics, state calculations)

• bulkUpdates.ts (NEW):

  • Chunked bulk update with 1000 orgs per transaction
  • Try-catch per chunk: traceException() on failure, continues processing
  • Batch cache invalidation
  • Returns success/failure stats
  • Includes commented raw SQL implementation for future optimization

• usageAggregation.ts:

  • Collect updates in array instead of immediate execution
  • Call bulkUpdateOrganizations() after processing each day
  • Track bulk update stats
  • Updated UsageAggregationStats type

• Tests: Updated to verify returned updateData instead of mock calls

Error Handling

• Each chunk failure is isolated (won't kill entire job)
• Failed chunks reported to Datadog via traceException()
• Successful chunks are committed
• Cache invalidation failures logged but don't fail updates

Performance Impact

• Before: 50,000 sequential UPDATEs (250-500s)
• After: ~50 transaction blocks (estimated 10-20s)
• Reduction: 95% query reduction, 95% faster execution

Test Plan

• All existing tests pass (thresholdProcessing.test.ts, usageAggregation.test.ts)
• Linter passes
• Manual test with staging data
• Monitor DB load after deployment

🤖 Generated with Claude Code

---

Important

Implements chunked updates for free tier usage tracking, reducing database load and improving performance by refactoring processThresholds() and introducing bulkUpdateOrganizations() for efficient batch processing.

• Behavior:
  • Implements chunked updates for free tier usage tracking, reducing query count from 50,000 to ~50.
  • processThresholds() in thresholdProcessing.ts refactored to return update data.
  • New bulkUpdateOrganizations() in bulkUpdates.ts handles chunked updates with error isolation per chunk.
  • usageAggregation.ts collects updates and calls bulkUpdateOrganizations() after processing each day.
• Error Handling:
  • Each chunk failure is isolated; failed chunks reported via traceException().
  • Cache invalidation failures logged but don't fail updates.
• Performance Impact:
  • Reduces query count by 95% and execution time by 95%.
• Tests:
  • Updated to verify returned updateData instead of mock calls.

^{This description was created by for 3388d3c. You can customize this summary. It will automatically update as commits are pushed.}

[Steffen911]

Both bulkUpdateOrganizations approaches look good to me in the current implementation. Your pick!