Automatically clean up after failed indexing runs

[aldenstpage]

When an indexing job fails (such as if a node in our Elasticsearch cluster has a full disk, or a bug in indexer-worker halts the process), the incomplete index is left inside of the Elasticsearch cluster, requiring someone to manually delete it. The indexer should detect this condition when the job starts and handle it.

The production index is determined by the image alias. The indexer should delete any index NOT pointed to by this alias following the naming scheme image-<uuid>.

[hedonhermdev]

Can I work on this issue?

[CodeMonk263]

Can i work on this issue?

[kgodey]

@hedonhermdev go ahead. @CodeMonk263 please find another issue to work on since @hedonhermdev commented first.

[DantrazTrev]

@hedonhermdev are still working on this issue?

[hedonhermdev]

No.

…

On Sat, 29 Feb 2020 at 8:07 PM, Dantraz ***@***.***> wrote: @hedonhermdev <https://github.com/hedonhermdev> are still working on this issue? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#402>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFRPNJ5THFLEUXK4JO5K42TRFEORJANCNFSM4KGJ2ZYQ> .

[DantrazTrev]

Can i take it over?
@aldenstpage

[kgodey]

Go ahead @DantrazTrev

[tushar912]

@DantrazTrev are u still working on this?

[kgodey]

@tushar912 it's been a few months since @DantrazTrev's post, I think you can go ahead and work on this.

[tushar912]

Ok

[tushar912]

The way i understood this issue is as follows. The main indexing job is done by indexer.py in ingestion_server . The TableIndexer class contains a method _index_table which checks if the database is in sync with index and replicates if not.There are two methods of indexing. reindex which creates a new index and makes it live alias and update which updates the index. Currently during reindex if the index is not created successfully it still persists in the cluster so the job is to delete the index if indexing fails . @kgodey or @aldenstpage please tell if i have understood correctly.

[tushar912]

Also i am thinking of modifying the already existing consistency_check method and add it to the reindex to delete the index if it is not indexed properly. Am i on the right track?