DataInfrastructure

Data Infrastructure Overhaul

Context

Our current data infrastructure is excessively complex and disproportionately spread across multiple AWS and GCP resources. It presents significant challenges in terms of management, scalability, and cost-effectiveness, despite handling relatively low operational throughput. As it stands, we have:

AWS Resources:

• RDS: 6 databases with 233 tables and 9.3 million records, introducing unnecessary fragmentation and operational overhead for what should be a more streamlined relational model.
• DynamoDB: 56 tables managing 149 million records, with 30 indexes, adding substantial complexity.
• Athena: 9 databases with 68 tables for querying, suggesting inefficiencies in data aggregation and exploration.
• S3: 143 buckets holding over 31 million objects.

GCP Resources:

• BigQuery: 10 datasets spanning 89 tables and over 47 million rows.

Problem Statement

Our current data infrastructure is overly complex and misaligned with its operational throughput. Despite its scale, the actual data usage and processing requirements are minimal, creating an imbalance between resource allocation and workload demands.

Key Issues

• Operational Overhead: High maintenance burden across cloud platforms.
• Cost Inefficiency: Underutilised resources lead to inflated cloud spend.
• Performance Bottlenecks: Scattered models hinder efficient data access.
• Scalability Challenges: Fragmentation limits growth potential.
• Technical Debt Risk: Complexity increases likelihood of mismanagement.

Goals and Objectives

• Reduce Complexity: Consolidate and streamline resources.
• Improve Cost Efficiency: Eliminate redundant infrastructure.
• Enhance Performance: Create unified, scalable architecture.
• Future-Proof: Ensure maintainability and adaptability.

Proposed Solution

Redesign around two fixed-size data structures: Events and States.

Events

• Immutable, explicitly recorded records that modify state.
• Executed in deterministic order.
• Retention: Archived after snapshot.
• Versioning: Single immutable version with ID.

States

• Derived, mutable representations of current system view.
• Always reconstructable by replaying events.
• Structures include:
  • Account
  • Payer
  • Payee
  • Organisation
  • Adapter

Infrastructure Overview

S3 Tables (Preview):

• Central component for storing immutable event logs and state snapshots.
• Uses Apache Iceberg for schema and compaction.

EventBridge:

• Orchestrates decoupled, event-driven workflows.

Action Plan

Phase 1: Assessment and Preparation

• Audit current infrastructure and identify redundant resources.
• Define Event and State models and map legacy data.
• Analyse cost inefficiencies.

Phase 2: Infrastructure Consolidation

• Merge RDS databases, simplify DynamoDB schemas.
• Rationalise S3 buckets and migrate low-use data.

Phase 3: Core Event Architecture

• Implement EventBridge routing (ref).
• Store events in S3 Tables.

Phase 4: Analytics Optimisation

• Use Athena for S3 Table queries.
• Optimize BigQuery schemas.
• Enable real-time dashboards via Looker.

Phase 5: Monitoring and Testing

• Setup CloudWatch monitoring.
• Conduct performance and consistency testing.
• Document architecture and train teams.

Phase 6: Transition and Review

• Gradual migration to new system.
• Conduct post-implementation assessment.
• Collect feedback and optimise further.

Review and Assessment Plan

Success Metrics

• Cost Reduction: Monthly spend comparison.
• Performance Improvement: Query latency.
• Complexity Reduction: Resource count deltas.
• Operational Overhead: Maintenance effort logged.

Monitoring

• EventBridge throughput
• Athena/S3 Table performance

Continuous Improvement

• Monthly and quarterly KPI reviews.
• Data consistency audits.
• Feedback loop from users.

Final Validation

• 6-month post-review.
• Validate cost, performance, and scalability improvements.

References and Notes

• LMAX: How to Do 100K TPS at <1ms Latency
• 00210: Use EventBridge as Eventbus
• Amazon S3 Tables Announcement