Clean up webhook events

[AlexTugarev]

by running a periodic garbage collector. This is a pattern we use for other resources.

fixes #12430

Description

The way GC works is admittedly tricky, as it's an interplay between the periodic GC task and the db-sync. Periodically this will mark a bunch of entries as deleted, and we rely on the db-sync to actually delete the entries.

Because it's proven unreliable to mark entries as delete in the thousands, leaving them in the db for ever after a failure, the settings to mark as deleted periodically are: 600 entries every 6 minutes, which is a rate of ~200K/day and should not produce a lot of DB CPU usage.

Related Issue(s)

Fixes #12430

How to test

Release Notes

NONE

Documentation

Werft options:

• /werft with-preview

[werft-gitpod-dev-com]

started the job as gitpod-build-at-events-gc.1 because the annotations in the pull request description changed
(with .werft/ from main)

[AlexTugarev]

back to draft. just tested the query and explain showed me a full table scan :-P

[AlexTugarev]

I removed the problematic delete <> 0 predicate. Also adding more description on how this will work in the end.

[AlexTugarev]

@geropl, maybe you want to have a look at this one?

[geropl]

@AlexTugarev Will review now.

[AlexTugarev]

/werft run

👍 started the job as gitpod-build-at-events-gc.5
(with .werft/ from main)

[AlexTugarev]

build was failing for an unrelated issue with db-sync replication. restarted.

[AlexTugarev]

/werft run

👍 started the job as gitpod-build-at-events-gc.6
(with .werft/ from main)

[geropl]

Code LGTM, tested and saw the GC doing it's job. Thank you! 🎉