feat(kuma-cp): CircuitBreaker proposal

[lobkovilya]

Summary

An initial version of the proposal for the new policy "CircuitBreaker"

[jakubdyszkiewicz]

We use outlier detection already for passive healthchecks https://kuma.io/docs/0.5.0/policies/health-check/ (check ClusterWithHealthChecks method). I think it makes sense to add more settings to passive healthchecks or get rid of passive HC and create Circuit Breaker.

[jakubdyszkiewicz]

This configuration is also important and can be a blocker when adopting Kuma. Especially max_connections

[lobkovilya]

@subnetmarco looks like we have several ways to implement CircuitBreaker:

• Get rid of passive HealthCheck and have all kinds of circuit breaking in one Policy:

  type: CircuitBreaker
  mesh: default
  name: cb-1
  selector:
    - match:
        service: backend 
        region: was
        version: 3
  conf:
    maxConnections: 3
    maxPendingRequest: 10
    ...
    interval: 1s
    baseEjectionTime: 30s
    maxEjectionPercent: 20
    splitExternalAndLocalErrors: false
    detectors:
      errors:
        total: 20
        gateway: 10
        local: 7
      standardDeviation:
        requestVolume: 10
        minimumHosts: 5
        factor: 1.9
      failure:
        requestVolume: 10
        minimumHosts: 5
        threshold: 85

  This approach closer to design patter called "Circuit breaker".

• Leave HealthCheck policy as it is. Enhance passive HealthCheck with additional parameters for outlier detection. Create CircuitBreaker policy that easily maps to Envoy's cluster.CircuitBreakers:

  type: CircuitBreaker
  mesh: default
  name: cb-1
  selector:
    - match:
        service: backend 
        region: was
        version: 3
  conf:
    maxConnections: 3
    maxPendingRequest: 10
    maxRequests: 10
    maxRetries: 3

  Write documentation page "Circuit Breakers" where we will show both passive HealthCheck and CircuitBreaker.
  This approach probably closer to Envoy users.

Do we have any notion what users expect from Kuma when it comes to Circuit breakers?

[subnetmarco]

I suggest that we add the CircuitBreaker resource and remove passive healthchecking from the Healthcheck resource.