Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
The amount of data generated by webhooks is huge because the requests are huge and we save them all.
For example, on the mainnet demo server it reached 15GB... which is more than 10 times the rest of the data. (For example the invoices table, which cover all invoices created on demo mainnet since 2017 is 400MB)
Much of this data is never actually used, it is only useful for redelivering webhooks. So I decided to prune the deliveries older than 60 days. Those are unlikely to be useful to anybody.
Implementation
services.AddScheduledTask
.CleanupWebhookDeliveriesTask
is dealing with the pruning. I keep only the type of the delivery so we can still see the timing of events in the invoice details list.I am using Dapper in
CleanupWebhookDeliveriesTask
because we are using covering indexes onTimestamp
and entity framework doesn't generate a request that can take advantage of this index.