Skip to content

Commit

Permalink
Update README.md
Browse files Browse the repository at this point in the history
  • Loading branch information
kailuowang committed Aug 18, 2016
1 parent 8d766a5 commit a1e8063
Showing 1 changed file with 53 additions and 13 deletions.
66 changes: 53 additions & 13 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,19 +10,17 @@
#### A set of work dispatchers implemented using Akka actors
Note: kanaloa work dispatchers are not Akka [MessageDispatcher](http://doc.akka.io/docs/akka/snapshot/scala/dispatchers.html).

### Migration from 0.2.x -> 0.3.0
1. Moved metrics reporting configuration to dispatcher level and added the on/off flag. Default metrics report settings is now in the default-dispatcher section. See the [reference configuration](src/main/resources/reference.conf) for details.

### This library is still pre-beta
Right now we are at 0.4.0, the plan is the stablize the API from 0.5.0 going on. So there will still be major API changes coming.

### Motivation
Kanaloa work dispatchers sit in front of your service and dispatch received work to them. They make your service more resilient through the following means:
1. **Autothrottle** - it dynamically figures out the optimal number of concurrent requests your service can handle, and make sure that at any given time your service handles no more than that number of concurrent requests. This simple mechanism was also ported and contributed to Akka as [Optimal Size Exploring Resizer](http://doc.akka.io/docs/akka/2.4.1/scala/routing.html#Optimal_Size_Exploring_Resizer) with some limitations. See details of the algorithm below.
2. **Back pressure control** - this control is [Little's law](https://en.wikipedia.org/wiki/Little%27s_law) inspired. It rejects requests when estimated wait time for which exceeds a certain threshold.
3. **Circuit breaker** - when error rate from your service goes above a certain threshold, the kanaloa dispatcher stops all requests for a short period of time to give your service a chance to "cool down".
Kanaloa work dispatchers sit in front of your service and provide auto back pressure through the following means:
1. **Autothrottle** - it dynamically figures out the optimal number of concurreny your service can handle, and make sure that at any given time your service handles no more than that number of concurrent requests. This simple mechanism was also ported and contributed to Akka as [Optimal Size Exploring Resizer](http://doc.akka.io/docs/akka/2.4.1/scala/routing.html#Optimal_Size_Exploring_Resizer) with some limitations. See details of the algorithm below.
2. **PIE** - A traffic regulator based on the PIE algo (Proportional Integral controller Enhanced) suggested in this paper https://www.ietf.org/mail-archive/web/iccrg/current/pdfB57AZSheOH.pdf by Rong Pan and his collaborators.
3. **Circuit breaker** - when error rate from your service goes above a certain threshold, the kanaloa will slow down sending work to them for a short period of time to give your service a chance to "cool down".
4. **Real-time monitoring** - a built-in statsD reporter allows you to monitor a set of critical metrics (throughput, failure rate, queue length, expected wait time, service process time, number of concurrent requests, etc) in real time. It also provides real-time insights into how kanaloa dispatchers are working. An example on Grafana:
![Dashboard](https://github.com/iheartradio/docker-grafana-graphite/blob/master/dashboard.png)
![Dashboard](https://cloud.githubusercontent.com/assets/83257/17714327/449e09d2-63cd-11e6-8527-3ee9e9bbf755.png)

For the detailed algorithm please see the [implementation detail](#impl) below.

### Get started

Expand All @@ -43,9 +41,6 @@ kanaloa {
startingPoolSize = 8
}
workTimeout = 3s
circuitBreaker {
errorRateThreshold = 0.7
}
}
}
}
Expand Down Expand Up @@ -148,7 +143,7 @@ We provide a [grafana dashboard](grafana/dashboard.json) if you are using grafan
We also provide a [docker image](https://github.com/iheartradio/docker-grafana-graphite) with which you can quickly get up and running a statsD server and visualization web app. Please follow the instructions there.


### <a name="impl"></a>Implementation Detail
### <a name="impl"></a>Implementation Detail for Autothrottle

*Disclosure: some of the following descriptions were adapted from the documentation of Akka's [OptimalSizeExploringResizer](http://doc.akka.io/docs/akka/2.4.1/scala/routing.html#Optimal_Size_Exploring_Resizer), which was also written by the original author of this document.*

Expand All @@ -169,6 +164,51 @@ By constantly exploring and optimizing, the resizer will eventually walk to the
If autothrottle is all you need, you should consider using [OptimalSizeExploringResizer](http://doc.akka.io/docs/akka/2.4.1/scala/routing.html#Optimal_Size_Exploring_Resizer) in an Akka router. (The caveat is that you need to make sure that your routees block on handling messages, i.e. they can’t simply asynchronously pass work to the backend service).


### <a name="implPIE"></a>Implementation Detail for PIE
A traffic regulator based on the PIE algo (Proportional Integral controller Enhanced)
suggested in this paper https://www.ietf.org/mail-archive/web/iccrg/current/pdfB57AZSheOH.pdf by Rong Pan and his collaborators.
The algo drop request with a probability, here is the pseudocode
Every update interval Tupdate
1. Estimation current queueing delay

```
currentDelay = queueLength / averageDequeueRate
```

2. Based on current drop probability, p, determine suitable step scales:

```
if p < 1% : α = α΄ / 8, β = β΄ / 8
else if p < 10% : α = α΄ / 2, β = β΄ 2
else : α = α΄, β = β΄
```
3. Calculate drop probability as:

```
p = p
+ α * (currentDelay - referenceDelay) / referenceDelay
+ β * (currentDelay - oldDelay) / referenceDelay
```

4. Update previous delay sample rate as

```
OldDelay - currentDelay
```
The regulator allows for a burst, here is the calculation

1. ```if burstAllowed > 0
enqueue request bypassing random drop
2. upon Tupdate
if p == 0 and currentDelay < referenceDelay / 2 and oldDelay < referenceDelay / 2
burstAllowed = maxBurst
else
burstAllowed = burstAllowed - timePassed (roughly Tupdate)
### Contribute
Any contribution and feedback is more than welcome.
Expand Down

0 comments on commit a1e8063

Please sign in to comment.