Merge pull request #138 from mjayprateek/refactoring_retries_logic

Refactors exponential backoff logic and releases it as an alpha feature

CHANGELOG.md

@@ -3,6 +3,18 @@ All notable changes to this project will be documented in this file. This change
 
 ## Unreleased Changes
 
+## 3.2.0 - 2020-01-09
+- Changes Exponential backoff config contract.
+    - Adds a `:type` key to retry-config
+    - Adds a limit on the number of retries possible in exponential backoff
+    - Releasing exponential backoff as an alpha feature
+- Fixes [issue](https://github.com/gojek/ziggurat/issues/136) where dead-set replay doesn't send the message to the retry-flow
+- Fixes [issue](https://github.com/gojek/ziggurat/issues/129) by updating tools.nrepl dependency to nrepl/nrepl
+- Fixes [this bug](https://github.com/gojek/ziggurat/issues/133) where dead set replay broke on Ziggurat upgrade from 2.x to 3.x .
+- Fixes [this bug](https://github.com/gojek/ziggurat/issues/115) in RabbitMQ message processing flow
+- Adds support for exponential backoffs in channels and normal retry flow
+- exponential backoffs can be enabled from the config
+
 ## 3.2.0-alpha.5 - 2019-12-17
 - Fixes [issue](https://github.com/gojek/ziggurat/issues/136) where dead-set replay doesn't send the message to the retry-flow
 

README.md

@@ -338,6 +338,40 @@ All Ziggurat configs should be in your `clonfig` `config.edn` under the `:ziggur
 * jobs - The number of consumers that should be reading from the retry queues and the prefetch count of each consumer
 * http-server - Ziggurat starts an http server by default and gives apis for ping health-check and deadset management. This defines the port and the number of threads of the http server.
 
+## Alpha (Experimental) Features
+The contract and interface for experimental features in Ziggurat can be changed as we iterate towards better designs for that feature.
+For all purposes these features should be considered unstable and should only be used after understanding their risks and implementations. 
+
+### Exponential Backoff based Retries
+In addition to linear retries, Ziggurat users can now use exponential backoff strategy for retries. This means that the message
+timeouts after every retry increase by a factor of 2. So, if your configured timeout is 100ms the backoffs will have timeouts as
+`200, 300, 700, 1500 ..`. These timeouts are calculated using the formula `(queue-timeout-ms * ((2**exponent) - 1))` where `exponent` falls in this range `[1,(min 25, configured-retry-count)]`.
+
+The number of retries possible in this case are capped at 25. The number of queues created in the RabbitMQ are equal to the configured-retry-count or 25, whichever is smaller.
+
+Exponential retries can be configured as described below.
+
+```$xslt
+:ziggurat {:stream-router {:default {:application-id "application_name"...}}}
+           :retry         {:type   [:exponential :keyword]
+                           :count  [10 :int]
+                           :enable [true :bool]} 
+
+```
+
+Exponential retries can be configured for channels too. Additionally, a user can specify a custom `queue-timeout-ms` value per channel. 
+Timeouts for exponential backoffs are calculated using `queue-timeout-ms`. This implies that each channel can have separate count of retries
+and different timeout values.
+
+```$xslt
+:ziggurat {:stream-router {:default {:application-id "application_name"...
+                                     :channels {:channel-1 .....
+                                                           :retry {:type   [:exponential :keyword]
+                                                                                      :count  [10 :int]
+                                                                                      :queue-timeout-ms 2000
+                                                                                      :enable [true :bool]}}}}} 
+```
+
 ## Contribution
 - For dev setup and contributions please refer to CONTRIBUTING.md
 

docker-compose.yml

@@ -2,9 +2,10 @@ version: '3.3'
 
 services:
   rabbitmq:
-    image: 'rabbitmq'
+    image: 'rabbitmq:3.8.2-management-alpine'
     ports:
      - '5672:5672'
+     - '15672:15672'
     container_name: 'ziggurat_rabbitmq'
   zookeeper:
     image: 'bitnami/zookeeper:latest'

project.clj

@@ -1,4 +1,4 @@
-(defproject tech.gojek/ziggurat "3.2.0-alpha.5"
+(defproject tech.gojek/ziggurat "3.2.0"
   :description "A stream processing framework to build stateless applications on kafka"
   :url "https://github.com/gojektech/ziggurat"
   :license {:name "Apache License, Version 2.0"

resources/config.test.ci.edn

@@ -24,7 +24,8 @@
                                                  :exchange-name "application_name_instant_exchange_test"}
                                    :dead-letter {:queue-name    "application_name_dead_letter_queue_test"
                                                  :exchange-name "application_name_dead_letter_exchange_test"}}
-            :retry                {:count   [5 :int]
+            :retry                {:type    [:linear :keyword]
+                                   :count   [5 :int]
                                    :enabled [true :bool]}
             :http-server          {:port         [8010 :int]
                                    :thread-count [100 :int]}
@@ -34,7 +35,8 @@
                                                           :origin-topic         "topic"
                                                           :upgrade-from         "1.1"
                                                           :channels             {:channel-1 {:worker-count [10 :int]
-                                                                                             :retry        {:count   [5 :int]
+                                                                                             :retry        {:type    [:linear :keyword]
+                                                                                                            :count   [5 :int]
                                                                                                             :enabled [true :bool]}}}
                                                           :producer             {:bootstrap-servers                     "localhost:9092"
                                                                                  :acks                                  "all"

resources/config.test.edn

@@ -25,6 +25,7 @@
                                    :dead-letter {:queue-name    "application_name_dead_letter_queue_test"
                                                  :exchange-name "application_name_dead_letter_exchange_test"}}
             :retry                {:count   [5 :int]
+                                   :type    [:linear :keyword]
                                    :enabled [true :bool]}
             :http-server          {:port         [8010 :int]
                                    :thread-count [100 :int]}
@@ -34,7 +35,8 @@
                                                           :origin-topic         "topic"
                                                           :upgrade-from         "1.1"
                                                           :channels             {:channel-1 {:worker-count [10 :int]
-                                                                                             :retry        {:count   [5 :int]
+                                                                                             :retry        {:type    [:linear :keyword]
+                                                                                                            :count   [5 :int]
                                                                                                             :enabled [true :bool]}}}
                                                           :producer             {:bootstrap-servers                     "localhost:9092"
                                                                                  :acks                                  "all"