Regularly restart GitPython-based services

[HebaruSan]

Problems

The bot has been skipping a lot of runs for months now based on the checks from #112 and #115:

And the current pass is taking an extremely long time to complete, over 10 hours to get from 'A' to 'P' (where it used to be 10-20 minutes for 'A' to 'Z'):

This has meant a very long delay in the post-merge handling of KSP-CKAN/NetKAN#7831 (and who knows what else).

Cause

We still don't know exactly why this happens, but based on graphs that @techman83 has shared previously, it seems to be something that starts out mild and then gets gradually worse the longer the containers run. May be due to some aspect of GitPython, but we don't quite have proof of that, and rewriting to use another tool is a big project (see #148).

Changes

There's already a RestartWebhooks service to handle changes in certs once a week.

Now some new duplicates of that service are created to automatically restart the Indexer, Adder, and Mirrorer every 3 days. This should reset whatever the problem is with the Python-based services and hopefully give us better service availability.

[techman83]

The only thing is, we'd have to to restart these much more frequently than every 3 days. But we can implement it and now that things have settled at work, I should be able to get back to looking at pygit.

I'll check this out in the next couple of days if I don't think of something else that might give us some breathing space. I can tell you though that it's the Indexer where this problem lay and believe it is to do with all the branching as the other containers do lots of git work without issue and that's one of the key differences. Though the reflog may be also an issue, but I can't see how it could be.

[HebaruSan]

KSP-CKAN/CKAN#3021 and KSP-CKAN/CKAN#3031 seem to have mostly solved the skipped runs problem by reducing the amount of staging, which @techman83 reports is important because git branch operations are costly and performed regardless of whether the files have changed. I think we can set this aside for now. Even if the problem returns, we know we can try to attack it by reducing usage of staging.

[techman83]

I don't know that we should reduce our use of staging, but maybe get a bit smarter about creating branches. Though staging implies manual intervention, so being smart about when we stage is likely just as useful for less technical reasons.

[HebaruSan]

being smart about when we stage is likely just as useful for less technical reasons

Yup, my motivation for those PRs was not having to deal with unnecessary decisions. It was sheer dumb luck that it happened to also help with other problems. But now we know that if any future change increases the use of staging, there's a trade-off to be considered.