Modify Deployment for multi data source searchengine

[khaledk2]

This PR contains the required changes to deploy the multi-data source searchengine ome/omero_search_engine#102

[sbesson]

A few comments. Also two general process questions:

• what happens if the searchengine_backup volume is created but not cloned from a previous volume? Will the playbook still initialise an empty cluster or will it fail during restore_elasticsearch_data command? This will be immediately relevant for the creation of prod128 as the volume will not be cloned from a previous version.
• what is the process for creating a new backup of the search cluster? Does that happen automatically after the indexer is executed? Or is that a separate command?

We will probably need to 2 companion PRs:

• one against submission workflow with the indexing/backup commands if they are different from the current one
• one against the deployment workflow to update the creation of new volumes and include the new volume to clone

[khaledk2]

I have added two new playbooks to backup and restore the searching data

• restore_searchengine_data.yml

This one should run after the deployment playbooks have completed successfully. It will check for the existence of the snapshot before running

• backup_searchengine_data.yml

This playbook should run just before releasing the production server.

[sbesson]

This one should run after the deployment playbooks have completed successfully. It will check for the existence of the snapshot before running

Shouldn't it run automatically as part of the search engine deployment then? Is the task asynchronous? If not how long will it typically take?

This playbook should run just before releasing the production server.

So the current indexing process is unchanged to update for new studies. Should this be executed by the person running the indexer using the same approach docker run rather than requiring an Ansible set-up?

[khaledk2]

The search engine data restore process is asynchronous, and it runs automatically after the last commit.
Indexing and cache updates for newly added studies are performed manually.
Backups can also be run manually using the following command

docker run -v /data/searchengine/searchengine/:/etc/searchengine/ -v /data/searchengine/searchengine/logs/:/opt/app-root/src/logs/ --network searchengine-net openmicroscopy/omero-searchengine:0.7 backup_elasticsearch_data

[sbesson]

Backups can also be run manually using the following command

docker run -v /data/searchengine/searchengine/:/etc/searchengine/ -v /data/searchengine/searchengine/logs/:/opt/app-root/src/logs/ --network searchengine-net openmicroscopy/omero-searchengine:0.7 backup_elasticsearch_data

@jburel @francesw @will-moore @dominikl what are your thoughts about the best place to integrate this into the IDR lifecycle?

[sbesson]

Let's discuss the state of this PR tomorrow at the IDR weekly meeting. I think we need to clarify two questions before spinning up prod128 with this included:

• the process for creating a dump of the search engine as raised in #448 (comment). My opinion is that this task is outside the scope of these playbooks
• re-reading the diff, what would happen if the playbooks and notably idr-02-services.yml was re-run against an environment previously deployed? Would the restore command be re-executed and what would be the expected state of the indexer?

[sbesson]

From the earlier discussion, the agreement was to:

• merge this in preparation of the deployment of prod128 with a new search engine container
• following creation, immediately run the indexer and backup the database (from the state of prod127)
• load new studies as usual
• run the indexer for the new studies and update the backup for prod129

I will open 2 issues to capture the two concerns raised in #448 (review)