Co-Process for backup/recovery, Auto Deployments and Centralized Configuration management for ElasticSearch
Java Shell
Clone or download
brharrington and gndcshv More flexible naming for index management (#138)
This change updates the index metadata to allow for arbitrary [DateTimeFormat] patterns to be used for matching the index names. In addition the retention period can be specified using the [ISO Period format]. This change maintains backwards compatibility with the previous JSON format of the index metadata. This allows the index management to support a much larger variety of date schemes including:

- Dashes: `prefix-2018-03-06`
- Dots: `prefix.2018.03.06`
- Suffixes: `prefix.2018-03-06.suffix`
- Other units such as weeks: `prefix-2018-48`

The index metadata is an object that supports the following fields:

- `indexNamePattern`: [DateTimeFormat] pattern used to match   the name of the index. If this field is present, then both `indexName` and `retentionType` will be ignored for the purposes of matching the index names.
- `retentionPeriod`: [ISO Period format] string specifying the how long to keep indexes. After subtracting the period from the current date the time will be rounded based on the name pattern. For example, if it is March 6th and I have a period of 1 month with indices like `prefix-YYYY-MM`, then the index for February would be retained as February 6th will get rounded to the start of February based on the name pattern and thus is not before the cutoff time. If the value is an integer, then it will converted to a period using the `retentionType` as the unit.
- `preCreate`: boolean indicating whether or not to pre-create the index. In order to pre-create one of the following must be significant for the name pattern: minute, hour, day, week, month or year.

Legacy fields supported to maintain backwards compatibility:

- `indexName`: prefix applied to the index. If the `indexNamePattern` is not specified, then this will be used along with the `retentionType` to create a name pattern that matches the old behavior.
- `retentionType`: one of hourly, daily, monthly, or yearly. The default is daily.

[DateTimeFormat patterns]: https://www.joda.org/joda- time/apidocs/org/joda/time/format/DateTimeFormat.html
[ISO Period format]: https://www.joda.org/joda-time/apidocs/org/joda/time/format/ISOPeriodFormat.html#standard--
Latest commit 3e272ed Apr 9, 2018

README.md


Build Status

Raigad is a process/tool that runs alongside Elasticsearch to automate the following:

  • Snapshot backup and restore.
  • Configured deployments for a dedicated master/data/search approach.
  • Tribe node deployments.
  • Publishing Elasticsearch monitoring metrics.
  • Support for AWS environment.

Elasticsearch is open source, distributed, real-time search and analytics engine: https://www.elastic.co/products/elasticsearch/

The name "Raigad" refers to the name of an Indian fort in state of Maharashtra.

Raigad is actively developed and used at Netflix.

Features:

  • Support multi-region Elasticsearch deployment in AWS via public IP.
  • Automated security group update in multi-region environment.
  • REST APIs for backup/restore and other operations.