-
Notifications
You must be signed in to change notification settings - Fork 1
03 Chaos Engineering
Chaos Engineering is the discipline of experimenting on a system to build confidence in its capability to withstand turbulent conditions in production. It involves intentionally introducing failures and observing how the system responds, allowing you to identify weaknesses before they cause real problems.
"Chaos Engineering is the discipline of experimenting on a distributed system in order to build confidence in the system's capability to withstand turbulent conditions in production." — Principles of Chaos Engineering
Modern software systems are increasingly complex and distributed. Even with thorough testing, unexpected failures can occur:
- Network failures - Timeouts, packet loss, DNS issues
- Service degradation - Slow responses, high latency
- Resource exhaustion - Memory leaks, CPU spikes, disk space
- External dependencies - Third-party API failures, database unavailability
- Infrastructure issues - Server crashes, container restarts, cloud provider outages
Traditional testing approaches (unit tests, integration tests) verify that systems work under expected conditions. Chaos Engineering verifies that systems survive under unexpected conditions.
The Principles of Chaos Engineering define the approach:
Define what "normal" looks like for your system using metrics like response time, error rate, or throughput. Example:
- "Our API responds within 500ms for 99% of requests"
- "Database queries complete successfully with less than 0.1% error rate"
Introduce realistic failure scenarios:
- Network issues: Latency, timeouts, connection failures
- Server failures: Process crashes, container restarts
- Resource constraints: CPU throttling, memory pressure
- Dependency failures: Database unavailability, third-party API errors
While you can start in test/staging environments, the ultimate goal is to run experiments in production (with appropriate safeguards) because that's where real user traffic and real complexity exist.
Manual chaos testing is valuable, but automation ensures continuous validation as your system evolves.
Start small and gradually increase scope. Use circuit breakers, rate limiters, and other resilience patterns to contain failures.
- Define steady state - Establish baseline metrics
- Hypothesize - Predict what will happen during failure
- Design experiment - Choose failure to introduce
- Run experiment - Execute in controlled manner
- Measure impact - Compare against steady state
- Improve system - Fix weaknesses discovered
- Repeat - Continuously validate
Hypothesis: "Our e-commerce checkout will handle database connection failures gracefully using retry logic."
Experiment:
- Monitor successful order rate (baseline: 99.9%)
- Introduce 20% database connection failures
- Observe system behavior with retry policy active
- Measure: Are orders still processed? What's the success rate?
- Result: Orders succeed after retry, success rate remains 99.5%
Murphy implements key resilience patterns used in Chaos Engineering:
Problem: Transient failures (temporary network glitch, brief service unavailability)
Solution: Automatically retry the operation after a delay
Use When:
- Network requests that can fail temporarily
- Database operations that might timeout
- Any operation where immediate retry might succeed
Problem: Cascading failures when a service is down (waste resources, increase load, slow response times)
Solution: After detecting failures, stop calling the service temporarily to give it time to recover
Use When:
- Calling external services that might be down
- Operations that can fail repeatedly
- You want to fail fast instead of waiting for timeout
Problem: What to do when an operation fails completely?
Solution: Provide a degraded but functional alternative
Use When:
- You have cached data as alternative
- You can return default values
- You can provide simplified functionality
Problem: Too many requests can overwhelm a system
Solution: Limit the rate of operations to prevent overload
Use When:
- Calling rate-limited APIs
- Protecting resources from excessive use
- Preventing DoS-like behavior in your own system
| Aspect | Traditional Testing | Chaos Engineering |
|---|---|---|
| Focus | "Does it work?" | "What breaks it?" |
| Scope | Known scenarios | Unknown scenarios |
| Environment | Test/staging | Ideally production |
| Approach | Verify correctness | Discover weaknesses |
| Failures | Avoided | Intentionally introduced |
| Goal | Catch bugs | Build resilience |
Both approaches are complementary and necessary for robust systems.
- Increased confidence - Know your system can handle failures
- Faster incident resolution - Understand failure modes before they occur in production
- Better design decisions - Identify architectural weaknesses early
- Improved monitoring - Discover gaps in observability
- Reduced downtime - Fewer production incidents
- Cost savings - Prevent costly outages
- Better customer experience - More reliable services
- Competitive advantage - Higher reliability than competitors
- Latency injection: Slow down network requests
- Packet loss: Drop some network packets
- Connection failures: Simulate network unavailability
- Process crashes: Kill application processes
- Resource exhaustion: Consume CPU, memory, or disk
- Clock skew: Change system time
- Server failures: Shutdown instances
- Container restarts: Restart Docker containers
- Availability zone failures: Simulate datacenter outages
- Database failures: Simulate DB unavailability or slowness
- API failures: Inject errors in API responses
- Cache failures: Simulate cache misses or Redis unavailability
Murphy for Delphi provides the resilience patterns needed to implement Chaos Engineering in your Delphi applications:
- Implement resilience patterns - Use Murphy's policies in your code
- Test with failures - Intentionally trigger exceptions to test your policies
- Validate behavior - Ensure your system degrades gracefully
- Iterate and improve - Refine your policies based on observations
// 1. Implement resilience
RetryPolicy := TRetryBuilder.Create
.Handle(EIdHTTPProtocolException)
.Retry(3)
.Wait(TTimeSpan.FromSeconds(2))
.Build;
// 2. Test with intentional failure
procedure TestWithFailures;
var
FailCount: Integer = 0;
begin
RetryPolicy.Execute(
procedure
begin
Inc(FailCount);
if FailCount < 3 then
raise EIdHTTPProtocolException.Create('Simulated failure');
// Third attempt succeeds
WriteLn('Success after retries!');
end);
end;- Study resilience patterns (Retry, Circuit Breaker, etc.)
- Identify critical paths in your application
- Implement Murphy policies for those paths
- Run chaos experiments in development/staging
- Verify policies behave as expected
- Adjust configuration based on results
- Start with low-impact experiments
- Monitor system behavior closely
- Gradually increase experiment scope
- Automate successful experiments
- Pattern Guides - Detailed guides for each pattern
- API Reference - Complete API documentation
- Combining Patterns - Advanced resilience strategies
- Principles of Chaos Engineering
- Chaos Engineering: System Resiliency in Practice - O'Reilly Book
- Netflix Tech Blog - Pioneering work in Chaos Engineering
- Awesome Chaos Engineering - Curated list of resources
- Failures are inevitable - Plan for them, don't hope they won't happen
- Test failure scenarios - Don't wait for production to discover weaknesses
- Build resilience proactively - Use patterns like retry, circuit breaker, and fallback
- Start small - Begin with simple experiments in safe environments
- Iterate continuously - Chaos Engineering is an ongoing practice, not a one-time effort