chore(indexes): remove in progress only when a matching index shows up in real indexes list; cleanup rolling on a timeout

[gribnoysup]

Based on the recent Slack discussion, we're adjusting the logic here a bit:

• "In progress" indexes are now removed not based on finishing the "create index" op, but on a matching index appearing in regular or rolling indexes lists
• To work around a corner case where rolling index might never show up when listed (see the notes in the code), we are setting up a timeout that will just remove the in progress rolling index from the list if enough time has passed and fetching the indexes haven't removed it before

[paula-stacho]

how long does it normally take for the rolling index to be created? 15s doesn't sound like much

[gribnoysup]

It's a bit hard to wrap the head around (tried to do it here but not sure if I did a good job to be honest), but we don't need to wait for the index to be created before this timeout check kicks in, we are only waiting for the rolling index build to start showing up in the list returned from the backend which corresponds to the index starting to be created on any of the nodes in the cluster, which happens way faster. As soon as it starts showing up in the list returned from the rolling indexes service, it will be removed from the in progress list and this action firing will be a no-op basically

[paula-stacho]

ah I see, thanks!

[paula-stacho]

Looks good! Are tests planned as a follow up?

[lerouxb]

Wondering if it is worth fussing over the timeout and clearing it. Probably not. Super rare to be adding rolling indexes and then there's only one and if the in-progress index disappeared in the meantime then it will just do nothing.

[gribnoysup]

That was my reasoning for not doing it, but I probably should put this in a comment, it's a rarely used feature and this action is harmless when it fires, so adding more code to store the reference seemed like an overkill

[lerouxb]

Tests would be nice, but it is probably fine as is too.

[gribnoysup]

@lerouxb @paula-stacho thanks for holding me accountable, will definitely add some tests 😄

[gribnoysup]

Added tests, also drive-by refactored a bit the polling mechanism so that it can be cleaned up when plugin unmounts (writing a functional test for this started to make the tests "hang" until timeout resolves, so was easy to catch the missing cleanup issue)