Port script for finding/deleting unreferenced state groups into Synapse

[erikjohnston]

There is a really simple rust script here: https://github.com/erikjohnston/synapse-find-unreferenced-state-groups for finding unreferenced state groups. Unlike the compress state script there isn't really a reason for it to be in rust, and given a) its small and mostly just SQL and b) needs a lot of rework anyway, we may as well port it o Python and add it to Synapse.

For now it should just be a script that can run independently (though part of the synapse code), but we may wish to look into running it from within synapse later.

c.f. #3364

[clokep]

needs a lot of rework anyway

@erikjohnston Can you expand upon what this means?

Were the steps you were thinking for this:

1. Pull out the SQL and port those to Python somewhere under synapse.*
2. Add a script that actually goes ahead and find/deletes the unreferenced state.

I think you also mentioned to be careful that this doesn't get applied to "new" state that might not be referenced yet but will soon.

[erikjohnston]

Yeah, exactly. We'd want all the functional bits of the script in synapse.* so that we can later add it to the admin API or run it automatically or whatever.

We probably also want to make sure that it can work incrementally, so that it doesn't have to do it all at once

I think you also mentioned to be careful that this doesn't get applied to "new" state that might not be referenced yet but will soon.

Yeah. The current script just takes a parameter for the maximum state group to look at, which you can get by looking for a state group that was persisted before the last restart

[clokep]

but we may wish to look into running it from within synapse later.

Now that we have support for running background tasks on a separate worker, it might be a logical place to run it. I'm unsure of the impact on smaller deploys that don't have this worker though.