[request] A/B testing + Canary releasing

[lemonhall]

nginx has build in split_user modules...

so .....

[thibaultcha]

I am not sure about the value of that for an API gateway? I don't think this is in the scope of this project, except if I am missing something maybe?

[sonicaghi]

@lemonhall can you pls describe in more details your use case?

[montanaflynn]

Never heard of this so I did some research into split_user module and I do think it's something which would make a good Kong plugin. Basically it allows for sending certain percentages of requests to different target URLs.

Here's some potential use cases:

You have an existing service that you're rewriting. You think it's production ready but you want to verify that first by sending a small portion of traffic to it and monitor for any errors or performance issues. With a "split" plugin you could say to send x% of traffic to your new service and see how it does before switching all the traffic to it.

There's another use case for A|B testing websites better described in this article which basically is just serving different HTML pages for an audience and setting some tracking header or file path to compare completed goals against later.

[thibaultcha]

Nice use case for the production ready API. For websites, which is the first case I thought about, it's outside of the scope of this project imho.

We need to know if this is doable through the Lua API now.

On May 8, 2015, at 9:17 PM, Montana Flynn notifications@github.com wrote:

Never heard of this so I did some research into split_user module and I do think it's something which would make a good Kong plugin. Basically it allows for sending certain percentages of requests to different target URLs.

Here's some potential use cases:

You have an existing service that you're rewriting. You think it's production ready but you want to verify that first by sending a small portion of traffic to it and monitor for any errors or performance issues. With a "split" plugin you could say to send x% of traffic to your new service and see how it does before switching all the traffic to it.

There's another use case for A|B testing websites better described in this article which basically is just serving different HTML pages for an audience and setting some tracking header or file path to compare completed goals against later.

—
Reply to this email directly or view it on GitHub.

[montanaflynn]

On the other hand if I'm already doing this for my website and I want to use Kong for my API it would be nice to let Kong power my website as well. For instance apistatus.org has a single nginx file that serves the website, the API, and caches any images. I could see myself using a "split" plugin for A|B testing the website. I will agree that calling the plugin "A|B" doesn't make sense as it limits the potential use cases which is why I've been referring to it as "Split".

[sonicaghi]

example:

[lucamaraschi]

👍

[sonicaghi]

https://github.com/SinaMSRE/ABTestingGateway

[petemounce]

I would use this, so +1.

I would also like the ability to send traffic to more than one endpoint and compare the responses, speeds, and sizes. Again, ready for production is the use case. Give traffic to a new version at the same time as current, and compare at a functional level and performance level whether new one is ready.

[tecnobrat]

I really wish I had this at my last job.... this is an awesome feature.

[thibaultcha]

On mobile currently but just pointing out that the issue here is that this would need to be reimplemented manually with bare Lua unless I am unaware of an already existing module handling this for openresty.

[sonicaghi]

maybe worth pinging @agentzh

[sonicaghi]

http://martinfowler.com/bliki/CanaryRelease.html

[skippy]

+1

[mtakaki]

+1

I'd love to see this.

[tgirardi]

+1 I would use it

[sonicaghi]

another useful resource http://cloudnative.io/blog/2015/02/the-dos-and-donts-of-bluegreen-deployment/

[schellkenig]

+1

[HenryChelsea]

+1

[billylaing]

This would be awesome.

[Tieske]

A good write-up of A/B testing, Canary releases and BlueGreen deployments; http://blog.christianposta.com/deploy/blue-green-deployments-a-b-testing-and-canary-releases/

[coopr]

Is this issue satisfied by new documentation Kong/docs.konghq.com#351 - or is there still an open issue here?

[Tieske]

On the level where Kong operates A/B and Canary are pretty much the same thing.

The ring-balancer can now be used to do canary, but only on generic load. Especially for A/B it would be good to have a selection criterion determining wether to go for A or B per request. eg. regex on a header, ip addresses, etc.

[p0pr0ck5]

These docs are merged and published! https://getkong.org/docs/0.10.x/loadbalancing/#blue-green-deployments

Defining how the balanced can split requests is a separate scope of discussion from a generic a/b and canary implantation, which was made available in 0.10.

[sandy-adi]

@p0pr0ck5 is this available on the community edition or only as part of the enterprise edition? Just making sure

[coopr]

@sandy-adi you'll see a Canary plugin for Kong Enterprise Edition on https://konghq.com/plugins/

Note that also we document how to perform Blue/Green deployments and Canary releases with Kong Community Edition here https://getkong.org/docs/latest/loadbalancing/#blue-green-deployments