Elasticsearch 7.x.0 service and private s3 bucket setup on 'space'

1. Create ES 7.x.0 service instance (~15 minutes)

cf create-service aws-elasticsearch <service_plan_name> <fec_instance_name>
on prod: cf create-service aws-elasticsearch es-medium fec-api-elasticsearch
on stage: cf create-service aws-elasticsearch es-medium fec-api-elasticsearch
on dev: cf create-service aws-elasticsearch es-medium fec-api-elasticsearch

2. Setup private S3 bucket (one time setup)

Ref: https://cloud.gov/docs/services/aws-elasticsearch/#managing-backups

1. Create private s3 bucket (It will store the ES snapshot files)
   cf create-service s3 basic <s3_bucket_instance_name>
   ex: cf create-service s3 basic fec-s3-snapshot
   
   
2. Get S3 bucket name
   a)Create a service key to access S3 credentials
   cf create-service-key <service_instance_name> <service_key_name>
   ex: cf create-service-key fec-s3-snapshot fec-s3-snapshot-key
   
   b)Get s3 credentials
   cf service-key <service_instance_name> <service_key_name>
   ex: cf service-key fec-s3-snapshot fec-s3-snapshot-key
   
   c)Connect to new S3 bucket :
   run: aws configure and enter the bucket credentials
   
   d)Create a cors.json on local and copy below rules into cors.json. Upload the CORS policy to private s3 bucket.
   {"CORSRules": [
{
"AllowedOrigins": ["*"],
"AllowedHeaders": ["*"],
"AllowedMethods": ["HEAD", "GET"],
"ExposeHeaders": ["ETag"]
}
]
}
   
   upload cors.json to s3 bucket on space
   ['dev' space]:
   aws s3 ls s3://cg-55a0b774-46d0-41ca-bdbd-39c931854856/
aws s3api put-bucket-cors --bucket cg-55a0b774-46d0-41ca-bdbd-39c931854856 --cors-configuration file://cors.json --profile s3-dev-snapshot
   
   ['stage' space]:
   aws s3 ls s3://cg-52c16ed3-879b-4163-951a-50b1c8b79f8b/
aws s3api put-bucket-cors --bucket cg-52c16ed3-879b-4163-951a-50b1c8b79f8b --cors-configuration file://cors.json --profile s3-stage-snapshot
   
   ['feature' space]:
   aws s3 ls s3://cg-1a47b3a6-7351-4414-9ad8-cbe8a8949569/
aws s3api put-bucket-cors --bucket cg-1a47b3a6-7351-4414-9ad8-cbe8a8949569 --cors-configuration file://cors.json --profile s3-feature-snapshot
   
   ['prod' space]:
   aws s3 ls s3://cg-a779d3fe-b4a5-4f36-ac11-704d858e0032/
aws s3api put-bucket-cors --bucket cg-a779d3fe-b4a5-4f36-ac11-704d858e0032 --cors-configuration file://cors.json --profile s3-prod-snapshot

3. Bind private S3 bucket to ES7 service and get snapshotRoleARN

1. Assign the new s3 bucket to Elasticsearch instance (keep in the same terminal)
   a)run: export s3_bucket=<bucket_name>
   on dev: export s3_bucket=cg-55a0b774-46d0-41ca-bdbd-39c931854856
   on stage: export s3_bucket=cg-52c16ed3-879b-4163-951a-50b1c8b79f8b
   on prod: export s3_bucket=cg-a779d3fe-b4a5-4f36-ac11-704d858e0032
   on feature: export s3_bucket=cg-1a47b3a6-7351-4414-9ad8-cbe8a8949569
   
   b)run: cf create-service-key <fec_es_instance_name> <fec_es_instance_name_key> -c '{"bucket":"${s3_bucket}"}'
   ex: cf create-service-key fec-api-elasticsearch fec-api-elasticsearch-key -c '{"bucket":"${s3_bucket}"}'
   
   
2. Get service key for fec es instance
   cf service-key <fec_es_instance_name> <fec_es_instance_name_key>
   ex: cf service-key fec-api-elasticsearch fec-api-elasticsearch-key
   
   
3. Get Elasticsearch service snapshotRoleARN from step (2)
   
   
4. Add/Update 'ES_SNAPSHOT_ROLE_ARN' env variable on
   follow this instruction: Switch out cf environment variables

4. Create repository and test snapshot, restore

follow this:Elasticsearch 7.x.0 management instruction

5. How to put back ES56 service

1. Re-name s3 service
   cf rename-service <org_instance_name> <new_instance_name>
   ex(maybe need type commend manually):
   cf rename-service fec-s3-api fec-s3-dev
   
   
2. Rebuild the latest api to circleci
   this will bind-service api, celery-beat, celery-worker to fec-api-search56
   
   
3. unbind app from ES74 (manually)
   cf unbind-service api <fec_es_instance_name>
   ex: cf unbind-service api fec-api-elasticsearch
   ex: cf unbind-service celery-beat fec-api-elasticsearch
   ex: cf unbind-service celery-worker fec-api-elasticsearch
   
   
4. Delete ES7 service
   Check service key: cf service-keys <fec_es_instance_name>
   ex: cf service-keys fec-api-elasticsearch
cf delete-service-key <fec_es_instance_name> <fec_es_instance_key>
   ex: cf delete-service-key fec-api-elasticsearch fec-api-elasticsearch-key
   ex: cf delete-service fec-api-elasticsearch