Qbox Disaster Recovery for ES clusters
These docker-compose files have been created for use if Qbox customer clusters become unreachable.
Each docker-compose file uses the official Elastic released images of the last major versions that are Apache2 licensed. Versions 1, 2, 5 do not contain any x-pack features and Version 6 and 7 are the explicit Apache2 releases.
Each Elasticsearch
version comes with its own instance of a NGINX
reverse proxy for basic authentication, a Kibana
instance for visualizations, and a monitor
for metrics. All monitor instances are the last version of ElasticHQ
other than ES version 1, which uses Cerebro
.
https://www.elastichq.org/
https://github.com/lmenezes/cerebro
Once everything is up and running, there will be a 3 node Elasticsearch cluster available on port 80
, Kibana on /kibana
, and a monitor at /monitor
. Kibana for version 2 and 5 can only reached at localhost:5601
.
Be sure to run this repo from an instance large enough to handle your application load, data set, and Elasticsearch containers.
These files were created on MacOS using Docker Desktop.
Prerequisites
Docker: https://docs.docker.com/engine/install/
Docker-Compose: https://docs.docker.com/compose/install/
How to run
- Clone the repo and
cd
in the appropriate ES version folder. - Set environment variables in the
.env
file.USER_NAME
="Username to authenticate - can be anything you like."PASSWORD
="Password used to authenticate - can be anything you like."SETUP_LOCAL
="Used to create NGINX basic auth credentials - it is recommended to point to a location within the ES version directory."DATA_LOCATION1/2/3
="Location of where your persistent ES data will live. - it is recommended to always have at least 40% of disk free."ES_LOG_LOCAL1/2/3
="Location of where ES logs will be written to."MEM_LIMIT
="How much RAM you want each ES node to have. - Defaults to 2GB."ALLOCATED_HEAP=
"Set to half ofMEM_LIMIT
- Defaults to 1GB."- It is recommended to leave any extra parameters found with in
ALLOCATED_HEAP
as they are needed for that particular ES version.
- It is recommended to leave any extra parameters found with in
- Run
docker-compose up
- To stop:
cntrl-c
anddocker-compose down
Connecting to endpoints.
Elasticsearch
is found on port80
by runningcurl <username>:<password>@localhost
Kibana
is found by runningcurl <username>:<password>@localhost/kibana
- On ES versions 2 and 5, Kibana is found by running
curl localhost:5601
- The monitor is found by running
curl <username>:<password>@localhost/monitor
- ElasticHQ: click "connect" once it opens.
- Cerebro: can only be found by hitting
localhost:9000
. - Once in Cerebro enter your credentials (from the
.env
and choose your cluster.)
Customization and Logging
Customization of Version 1, 2, and 5 is possible via the elasticsearch.yml
that is found in the config folder of your chosen ES version. Version 6 and 7 can be modified using environment variables in the docker-compose file.
Elasticsearch logs for each node are written to its own file and located in the directories you set in the .env
file. Logs are rotated and settings can be found in the config folder under either the logging.yml
or log4j2.properties
file depending on your ES version. Non-rotated logs are overwritten on each docker-compose
start-up.
Contacting Support
Qbox support can be contacted through email or from the Support Portal.
- Legacy Qbox: support@qbox.io
- Next-Gen Qbox: next-gen@qbox.io
- Support Portal: https://qboxio.atlassian.net/servicedesk/
- You must have a Qbox user to use the Support Portal
Qbox Support will provide documentation on how to obtain and restore the most recent available copy of your data.
Elasticsearch Plugins
If your ES cluster has plugins installed, you will need to install plugins before any data is restored.
Plugins must be installed on each node and the node restarted. This can be can be done by docker exec
into your ES containers and running bin/elasticsearch-plugin install <plugin>
, then running docker restart <container>
. These commands can also be added to the elasticsearch-with-s3.Dockerfile
and will be ran on startup. The respository-s3
plugin is already added for your convenience.
Please see the plugins.md
file in the root of this repo for links to plugins. It is also recommended to read through Elastic's plugin information for your ES version as there are different methods of plugin installation that can be tried.
https://www.elastic.co/guide/en/elasticsearch/plugins/7.10/plugin-management.html
Cloning via AWS CLI rather than ES restore
You may receive information from Qbox support regarding commands to run to set up restore location, like the following:
cluster: [CLUSTER_NAME]
----------------------------------
1. Add the snapshot repository, replace and password below
curl -X PUT REPLACE_USER:REPLACE_PASS@localhost/_snapshot/recovery -H 'Content-Type: application/json' -d @- <<BODY
{
"type": "s3",
"settings": {
"bucket": "[BUCKET_WITH_REGION_NAME]",
"base_path":"[BASE_PATH]",
"access_key": "[ACCESS_KEY]",
"secret_key": "[SECRET_KEY]"
}
}
BODY
This encompasses the information needed to authenticate and connect to an S3 bucket. This can be adapted to be ran as an aws cli command. Taking the info above and adapting it gives us:
# Assumes aws cli tool not installed, using docker instead
docker run -it --entrypoint bash -v $PWD:/backup amazon/aws-cli
# replace the following exports
export AWS_ACCESS_KEY_ID=[ACCESS_KEY]
export AWS_SECRET_ACCESS_KEY=[SECRET_KEY]
export BASE_PATH=[BASE_PATH]
export REGION=[REGION_NAME (found in bucket name)]
export CLUSTER_NAME=[CLUSTER_NAME]
# Legacy Qbox
# list contents
aws s3 --region $REGION ls s3://recovery-qbox-backup-$REGION/$BASE_PATH/
# copy
aws s3 --region $REGION cp s3://recovery-qbox-backup-$REGION/$BASE_PATH /backup/$CLUSTER_NAME/. --recursive
# Next-Gen Qbox
# list contents
aws s3 --region $REGION ls s3://recovery-qbox-backup-ng-$REGION/$BASE_PATH/
# copy
aws s3 --region $REGION cp s3://recovery-qbox-backup-ng-$REGION/$BASE_PATH /backup/$CLUSTER_NAME/. --recursive
Disclaimer
By using this repo, you agree to Qbox's Terms of Service:
https://qbox.io/terms
Even though extensive testing went into the development of this repo and its files, it should not be used as an alternative to a managed production service provider or be considered a production worthy deployment and should only be used for testing or emergencies. Qbox will not offer any support for its use. Use at your own risk.