-
Notifications
You must be signed in to change notification settings - Fork 123
Open
Labels
Description
Currently, the probability of a pod being killed depends on the number of pods being in the target group. This is bad if you want to run chaoskube
as a cluster addon and opt-in to being killed via annotations because you cannot estimate how often that would happen.
Proposal
Allow specifying or at least somehow keep track of what's going on so Pod terminations happen in a somewhat predictable way. For example, instead of terminating a single pod every 10 minutes, each pod may have a probability of X% of being killed per hour. This, hopefully, would make pod terminations independent of the number of pods running in total.
Activity
klautcomputing commentedon Jun 30, 2017
Would you like this to be pod specific or a cluster wide probability?
linki commentedon Jul 6, 2017
I was thinking about making it pod specific but I also see value in a global version of it like you propose in #34.
For the pod specific version I thought one would annotate a PodSpec with something like:
chaos.alpha.kubernetes.io/frequency=2/day
for "kill this twice per day"chaos.alpha.kubernetes.io/frequency=10/hour
for "kill this ten times per hour"chaos.alpha.kubernetes.io/frequency=1/week
for "kill this once a week"To implement this: one invokes
chaoskube
at a certain interval like before and then calculate a probability per pod based on the desired frequency and how oftenchaoskube
is invoked.For instance, let's assume
chaoskube
runs at an interval of 1 minute and a pod has the annotation set to twice a day (2/day). Then on each iterationchaoskube
would calculate a probability that this pod should be killed like this:24*60/2
=> every 720 minutesor for ten times an hour (10/hour).
60/10
=> every 6 minutesThis would also work with different intervals I think.
I'm not sure if this is correct but if it is it would allow
chaoskube
to remain stateless and pods would be killed at roughly the same pace over time regardless of cluster size.klautcomputing commentedon Jul 6, 2017
Sounds good.
Just a couple of thoughts/questions I had when I read this: