Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow to specify daily time periods without chaos #62

Merged
merged 5 commits into from Feb 17, 2018
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
2 changes: 1 addition & 1 deletion Gopkg.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

30 changes: 17 additions & 13 deletions README.md
Expand Up @@ -31,8 +31,7 @@ INFO[4804] Killing pod chaoskube/nginx-701339712-51nt8
...
```

`chaoskube` allows to filter target pods by namespaces, labels and annotations.
[See below](#filtering-targets) for details.
`chaoskube` allows to filter target pods [by namespaces, labels and annotations](#filtering-targets) as well as [exclude certain weekdays or times of day](#limit-the-chaos) from chaos.

## How

Expand Down Expand Up @@ -139,21 +138,26 @@ spec:
...
```

## Limiting the Chaos
## Limit the Chaos

You can limit the time when chaos is introduced. To turn on this feature, add a comma-separated list of abbreviated weekdays via the `--excluded-weekdays` option and specify a `--timezone` in which to interpret those weekdays. Use `UTC`, `Local` or pick a timezone name from the [(IANA) tz database](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones). If you're testing `chaoskube` from your local machine then `Local` makes the most sense. Once you deploy `chaoskube` to your cluster you should deploy it with a specific timezone, e.g. where most of your team members are living, so that both your team and `chaoskube` have a common understanding when a particular weekday begins and ends. If your team is spread across multiple time zones it's probably best to pick `UTC` which is also the default. Picking the wrong timezone shifts the meaning of, e.g., Saturday by a couple of hours between you and the server.
You can limit the time when chaos is introduced by weekdays, time periods of a day or both.

Add a comma-separated list of abbreviated weekdays via the `--excluded-weekdays` options and/or a comma-separated list of time periods via the `--excluded-times-of-day` option and specify a `--timezone` by which to interpret them.

Use `UTC`, `Local` or pick a timezone name from the [(IANA) tz database](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones). If you're testing `chaoskube` from your local machine then `Local` makes the most sense. Once you deploy `chaoskube` to your cluster you should deploy it with a specific timezone, e.g. where most of your team members are living, so that both your team and `chaoskube` have a common understanding when a particular weekday begins and ends, for instance. If your team is spread across multiple time zones it's probably best to pick `UTC` which is also the default. Picking the wrong timezone shifts the meaning of a particular weekday by a couple of hours between you and the server.

## Flags

| Option | Description | Default |
|-----------------------|----------------------------------------------------------------------|------------------------|
| `--interval` | interval between pod terminations | 10m |
| `--labels` | label selector to filter pods by | (matches everything) |
| `--annotations` | annotation selector to filter pods by | (matches everything) |
| `--namespaces` | namespace selector to filter pods by | (all namespaces) |
| `--excluded-weekdays` | weekdays when chaos is to be suspended, e.g. "Sat,Sun" | (no weekday excluded) |
| `--timezone` | timezone from tz database, e.g. "America/New_York", "UTC" or "Local" | (UTC) |
| `--dry-run` | don't kill pods, only log what would have been done | true |
| Option | Description | Default |
|---------------------------|----------------------------------------------------------------------|----------------------------|
| `--interval` | interval between pod terminations | 10m |
| `--labels` | label selector to filter pods by | (matches everything) |
| `--annotations` | annotation selector to filter pods by | (matches everything) |
| `--namespaces` | namespace selector to filter pods by | (all namespaces) |
| `--excluded-weekdays` | weekdays when chaos is to be suspended, e.g. "Sat,Sun" | (no weekday excluded) |
| `--excluded-times-of-day` | times of day when chaos is to be suspended, e.g. "22:00-08:00" | (no times of day excluded) |
| `--timezone` | timezone from tz database, e.g. "America/New_York", "UTC" or "Local" | (UTC) |
| `--dry-run` | don't kill pods, only log what would have been done | true |

## Contributing

Expand Down
37 changes: 26 additions & 11 deletions chaoskube/chaoskube.go
Expand Up @@ -13,6 +13,8 @@ import (
"k8s.io/apimachinery/pkg/selection"
"k8s.io/client-go/kubernetes"
"k8s.io/client-go/pkg/api/v1"

"github.com/linki/chaoskube/util"
)

// Chaoskube represents an instance of chaoskube
Expand All @@ -27,6 +29,8 @@ type Chaoskube struct {
Namespaces labels.Selector
// a list of weekdays when termination is suspended
ExcludedWeekdays []time.Weekday
// a list of time periods of a day when termination is suspended
ExcludedTimesOfDay []util.TimePeriod
// the timezone to apply when detecting the current weekday
Timezone *time.Location
// an instance of logrus.StdLogger to write log messages to
Expand All @@ -48,23 +52,27 @@ var msgVictimNotFound = "No victim could be found. If that's surprising double-c
// msgWeekdayExcluded is the log message when termination is suspended due to the weekday filter
var msgWeekdayExcluded = "This day of the week is excluded from chaos."

// msgTimeOfDayExcluded is the log message when termination is suspended due to the time of day filter
var msgTimeOfDayExcluded = "This time of day is excluded from chaos."

// New returns a new instance of Chaoskube. It expects a kubernetes client, a
// label, annotation and/or namespace selector to reduce the amount of affected
// pods as well as whether to enable dryRun mode and a seed to seed the randomizer
// with. You can also provide a list of weekdays and corresponding time zone when
// chaoskube should be inactive.
func New(client kubernetes.Interface, labels, annotations, namespaces labels.Selector, excludedWeekdays []time.Weekday, timezone *time.Location, logger log.StdLogger, dryRun bool, seed int64) *Chaoskube {
func New(client kubernetes.Interface, labels, annotations, namespaces labels.Selector, excludedWeekdays []time.Weekday, excludedTimesOfDay []util.TimePeriod, timezone *time.Location, logger log.StdLogger, dryRun bool, seed int64) *Chaoskube {
c := &Chaoskube{
Client: client,
Labels: labels,
Annotations: annotations,
Namespaces: namespaces,
ExcludedWeekdays: excludedWeekdays,
Timezone: timezone,
Logger: logger,
DryRun: dryRun,
Seed: seed,
Now: time.Now,
Client: client,
Labels: labels,
Annotations: annotations,
Namespaces: namespaces,
ExcludedWeekdays: excludedWeekdays,
ExcludedTimesOfDay: excludedTimesOfDay,
Timezone: timezone,
Logger: logger,
DryRun: dryRun,
Seed: seed,
Now: time.Now,
}

rand.Seed(c.Seed)
Expand Down Expand Up @@ -131,6 +139,13 @@ func (c *Chaoskube) TerminateVictim() error {
}
}

for _, tp := range c.ExcludedTimesOfDay {
if tp.Includes(c.Now().In(c.Timezone)) {
c.Logger.Printf(msgTimeOfDayExcluded)
return nil
}
}

victim, err := c.Victim()
if err == ErrPodNotFound {
c.Logger.Printf(msgVictimNotFound)
Expand Down