# Chaos Engineering Basics

## What is Chaos Engineering?

**Chaos Engineering** is the discipline of experimenting on a system to build confidence in its capability to withstand turbulent conditions in production. It involves intentionally introducing failures and observing how the system responds, allowing you to identify weaknesses before they cause real problems.

> *"Chaos Engineering is the discipline of experimenting on a distributed system in order to build confidence in the system's capability to withstand turbulent conditions in production."*
> — [Principles of Chaos Engineering](https://principlesofchaos.org/)

## Why Chaos Engineering?

Modern software systems are increasingly complex and distributed. Even with thorough testing, unexpected failures can occur:

- **Network failures** - Timeouts, packet loss, DNS issues
- **Service degradation** - Slow responses, high latency
- **Resource exhaustion** - Memory leaks, CPU spikes, disk space
- **External dependencies** - Third-party API failures, database unavailability
- **Infrastructure issues** - Server crashes, container restarts, cloud provider outages

Traditional testing approaches (unit tests, integration tests) verify that systems work under expected conditions. Chaos Engineering verifies that systems **survive** under unexpected conditions.

## Core Principles

The [Principles of Chaos Engineering](https://principlesofchaos.org/) define the approach:

### 1. Build a Hypothesis Around Steady-State Behavior

Define what "normal" looks like for your system using metrics like response time, error rate, or throughput. Example:
- "Our API responds within 500ms for 99% of requests"
- "Database queries complete successfully with less than 0.1% error rate"

### 2. Vary Real-World Events

Introduce realistic failure scenarios:
- **Network issues**: Latency, timeouts, connection failures
- **Server failures**: Process crashes, container restarts
- **Resource constraints**: CPU throttling, memory pressure
- **Dependency failures**: Database unavailability, third-party API errors

### 3. Run Experiments in Production

While you can start in test/staging environments, the ultimate goal is to run experiments in production (with appropriate safeguards) because that's where real user traffic and real complexity exist.

### 4. Automate Experiments to Run Continuously

Manual chaos testing is valuable, but automation ensures continuous validation as your system evolves.

### 5. Minimize Blast Radius

Start small and gradually increase scope. Use circuit breakers, rate limiters, and other resilience patterns to contain failures.

## Chaos Engineering in Practice

### The Chaos Engineering Lifecycle

1. **Define steady state** - Establish baseline metrics
2. **Hypothesize** - Predict what will happen during failure
3. **Design experiment** - Choose failure to introduce
4. **Run experiment** - Execute in controlled manner
5. **Measure impact** - Compare against steady state
6. **Improve system** - Fix weaknesses discovered
7. **Repeat** - Continuously validate

### Example Experiment

**Hypothesis**: "Our e-commerce checkout will handle database connection failures gracefully using retry logic."

**Experiment**:
1. Monitor successful order rate (baseline: 99.9%)
2. Introduce 20% database connection failures
3. Observe system behavior with retry policy active
4. Measure: Are orders still processed? What's the success rate?
5. Result: Orders succeed after retry, success rate remains 99.5%

## Resilience Patterns

Murphy implements key resilience patterns used in Chaos Engineering:

### 1. Retry Pattern

**Problem**: Transient failures (temporary network glitch, brief service unavailability)

**Solution**: Automatically retry the operation after a delay

**Use When**:
- Network requests that can fail temporarily
- Database operations that might timeout
- Any operation where immediate retry might succeed

### 2. Circuit Breaker Pattern

**Problem**: Cascading failures when a service is down (waste resources, increase load, slow response times)

**Solution**: After detecting failures, stop calling the service temporarily to give it time to recover

**Use When**:
- Calling external services that might be down
- Operations that can fail repeatedly
- You want to fail fast instead of waiting for timeout

### 3. Fallback Pattern

**Problem**: What to do when an operation fails completely?

**Solution**: Provide a degraded but functional alternative

**Use When**:
- You have cached data as alternative
- You can return default values
- You can provide simplified functionality

### 4. Rate Limit Pattern

**Problem**: Too many requests can overwhelm a system

**Solution**: Limit the rate of operations to prevent overload

**Use When**:
- Calling rate-limited APIs
- Protecting resources from excessive use
- Preventing DoS-like behavior in your own system

## Chaos Engineering vs. Traditional Testing

| Aspect | Traditional Testing | Chaos Engineering |
|--------|-------------------|-------------------|
| **Focus** | "Does it work?" | "What breaks it?" |
| **Scope** | Known scenarios | Unknown scenarios |
| **Environment** | Test/staging | Ideally production |
| **Approach** | Verify correctness | Discover weaknesses |
| **Failures** | Avoided | Intentionally introduced |
| **Goal** | Catch bugs | Build resilience |

Both approaches are complementary and necessary for robust systems.

## Benefits of Chaos Engineering

### For Development Teams

- **Increased confidence** - Know your system can handle failures
- **Faster incident resolution** - Understand failure modes before they occur in production
- **Better design decisions** - Identify architectural weaknesses early
- **Improved monitoring** - Discover gaps in observability

### For Organizations

- **Reduced downtime** - Fewer production incidents
- **Cost savings** - Prevent costly outages
- **Better customer experience** - More reliable services
- **Competitive advantage** - Higher reliability than competitors

## Common Chaos Engineering Scenarios

### Network Chaos
- **Latency injection**: Slow down network requests
- **Packet loss**: Drop some network packets
- **Connection failures**: Simulate network unavailability

### Application Chaos
- **Process crashes**: Kill application processes
- **Resource exhaustion**: Consume CPU, memory, or disk
- **Clock skew**: Change system time

### Infrastructure Chaos
- **Server failures**: Shutdown instances
- **Container restarts**: Restart Docker containers
- **Availability zone failures**: Simulate datacenter outages

### Dependency Chaos
- **Database failures**: Simulate DB unavailability or slowness
- **API failures**: Inject errors in API responses
- **Cache failures**: Simulate cache misses or Redis unavailability

## How Murphy Helps

Murphy for Delphi provides the **resilience patterns** needed to implement Chaos Engineering in your Delphi applications:

1. **Implement resilience patterns** - Use Murphy's policies in your code
2. **Test with failures** - Intentionally trigger exceptions to test your policies
3. **Validate behavior** - Ensure your system degrades gracefully
4. **Iterate and improve** - Refine your policies based on observations

### Example: Testing a Retry Policy

```pascal
// 1. Implement resilience
RetryPolicy := TRetryBuilder.Create
  .Handle(EIdHTTPProtocolException)
  .Retry(3)
  .Wait(TTimeSpan.FromSeconds(2))
  .Build;

// 2. Test with intentional failure
procedure TestWithFailures;
var
  FailCount: Integer = 0;
begin
  RetryPolicy.Execute(
    procedure
    begin
      Inc(FailCount);
      if FailCount < 3 then
        raise EIdHTTPProtocolException.Create('Simulated failure');
      // Third attempt succeeds
      WriteLn('Success after retries!');
    end);
end;
```

## Getting Started with Chaos Engineering

### Phase 1: Learn and Prepare
1. Study resilience patterns (Retry, Circuit Breaker, etc.)
2. Identify critical paths in your application
3. Implement Murphy policies for those paths

### Phase 2: Test in Safe Environment
1. Run chaos experiments in development/staging
2. Verify policies behave as expected
3. Adjust configuration based on results

### Phase 3: Graduate to Production
1. Start with low-impact experiments
2. Monitor system behavior closely
3. Gradually increase experiment scope
4. Automate successful experiments

## Resources

### Murphy Documentation
- [Pattern Guides](Home.md#pattern-guides) - Detailed guides for each pattern
- [API Reference](API-Reference/README.md) - Complete API documentation
- [Combining Patterns](Patterns/Combining-Patterns.md) - Advanced resilience strategies

### External Resources
- [Principles of Chaos Engineering](https://principlesofchaos.org/)
- [Chaos Engineering: System Resiliency in Practice](https://www.oreilly.com/library/view/chaos-engineering/9781492043850/) - O'Reilly Book
- [Netflix Tech Blog](https://netflixtechblog.com/) - Pioneering work in Chaos Engineering
- [Awesome Chaos Engineering](https://github.com/dastergon/awesome-chaos-engineering) - Curated list of resources

## Key Takeaways

1. **Failures are inevitable** - Plan for them, don't hope they won't happen
2. **Test failure scenarios** - Don't wait for production to discover weaknesses
3. **Build resilience proactively** - Use patterns like retry, circuit breaker, and fallback
4. **Start small** - Begin with simple experiments in safe environments
5. **Iterate continuously** - Chaos Engineering is an ongoing practice, not a one-time effort

---

[← Getting Started](02-Getting-Started.md) | [Back to Index](Home.md) | [Architecture →](04-Architecture.md)