HC-447: Added capability to rollover old job_status indices in Mozart ES

[mcayanan]

Overview

Dependent on this related PR: hysds/hysds#149

• Updated the sds update mozart function such that it will install the ES ILM policy as well as the templates for:
  • job_status
  • worker_status
  • event_status
  • task_status
• Index and template creation was previously done in the hysds repo, but it was discovered that the indices needed to be created beforehand in order to properly attach the ILM policy to these indices. Note that this ticket only focuses on attaching ILM to job_status. The rest have been moved for convenience since they eventually will have an ILM per HC-451.
• The new ILM policy has a default value of the following:
  • warm phase: 90 days from date of creation
  • cold phase: 97 days from date of creation
  • delete phase: 104 days from date of creation
• The ILM policy is configurable by placing your own custom file under ~/.sds/files

Note that when these templates get created, the following message is seen in the logs:

[2023-02-01T17:25:24,439][WARN ][o.e.c.m.MetadataIndexTemplateService] [master] index template [job_status] has index patterns [job_status*] matching patterns from existing older templates [index_defaults] with patterns (index_defaults => [*]); this template [job_status] will take precedence during new index creation
[2023-02-01T17:25:24,557][INFO ][o.e.c.m.MetadataIndexTemplateService] [master] adding index template [job_status] for index patterns [job_status*]
[2023-02-01T17:25:25,093][WARN ][o.e.c.m.MetadataIndexTemplateService] [master] index template [worker_status] has index patterns [worker_status*] matching patterns from existing older templates [index_defaults] with patterns (index_defaults => [*]); this template [worker_status] will take precedence during new index creation
[2023-02-01T17:25:25,103][INFO ][o.e.c.m.MetadataIndexTemplateService] [master] adding index template [worker_status] for index patterns [worker_status*]
[2023-02-01T17:25:25,649][WARN ][o.e.c.m.MetadataIndexTemplateService] [master] index template [task_status] has index patterns [task_status*] matching patterns from existing older templates [index_defaults] with patterns (index_defaults => [*]); this template [task_status] will take precedence during new index creation
[2023-02-01T17:25:25,658][INFO ][o.e.c.m.MetadataIndexTemplateService] [master] adding index template [task_status] for index patterns [task_status*]
[2023-02-01T17:25:26,192][WARN ][o.e.c.m.MetadataIndexTemplateService] [master] index template [event_status] has index patterns [event_status*] matching patterns from existing older templates [index_defaults] with patterns (index_defaults => [*]); this template [event_status] will take precedence during new index creation
[2023-02-01T17:25:26,203][INFO ][o.e.c.m.MetadataIndexTemplateService] [master] adding index template [event_status] for index patterns [event_status*]

This is because according to the ES Index Template Docs,

When a composable template matches a given index it always takes precedence over a legacy template. If no composable template matches, a legacy template may still match and be applied.

Legacy templates, which is what appears that Core has been using historically, is deprecated: https://www.elastic.co/guide/en/elasticsearch/reference/7.10/indices-templates-v1.html

Testing

See comments in https://hysds-core.atlassian.net/browse/HC-447