Don't fill up cluster if there is no manual intervention

[fsteeg]

New indexes should be switched to production automatically, one old index retained to switch back manually if something is wrong. I seem to remember this was our original setup. Currently it's reversed: new index is switched to production manually if everything is ok. This leads to the cluster filling up and failing if indexes are not switched manually in time (see hbz/nwbib#302, hbz/nwbib#303, hbz/lobid#299).

[dr0i]

New indexes should be switched to production automatically

This can be easily done by setting the alias parameter to "resources" instead of setting it to "resources-staging". Although, I wouldn't recommend that for obvious reasons (at least there should be a test proving if the new index is plausible (i.e. more than 66M documents etc)).

This leads to the cluster filling up and failing if indexes are not switched manually in time

No. The underlying cause is a full HD. In https://github.com/hbz/lobid-resources/blob/master/src/main/java/org/lobid/resources/ElasticsearchIndexer.java#L297 it's specified that if there are 3 (or more) indices all old ones will be removed (as long as they have no alias set) so that only 2 indices are left. Thus, normally the HD wouldn't fill up.

A full HD also depends on other things (overflowing logs etc). This was probably the cause of hbz/nwbib#302.

So I would suggest to close this issue.