Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prune webhook data from database #5005

Merged
merged 1 commit into from May 29, 2023
Merged

Conversation

NicolasDorier
Copy link
Member

@NicolasDorier NicolasDorier commented May 28, 2023

Motivation

The amount of data generated by webhooks is huge because the requests are huge and we save them all.
For example, on the mainnet demo server it reached 15GB... which is more than 10 times the rest of the data. (For example the invoices table, which cover all invoices created on demo mainnet since 2017 is 400MB)
Much of this data is never actually used, it is only useful for redelivering webhooks. So I decided to prune the deliveries older than 60 days. Those are unlikely to be useful to anybody.

Implementation

  • I introduce a new way to register periodic tasks via services.AddScheduledTask.
  • CleanupWebhookDeliveriesTask is dealing with the pruning. I keep only the type of the delivery so we can still see the timing of events in the invoice details list.
  • It is unfortunate, but there isn't any other way than dropping the old Delivery table and recreate fresh tables. Any attempt to create an index or queries on existing tables is overly complicated and time out. I think it is fine to do this, as webhook delivery data is only useful when somebody wants to replay a webhook delivery.
  • I skip the deliveries table in the migration from sqlite/mysql

I am using Dapper in CleanupWebhookDeliveriesTask because we are using covering indexes on Timestamp and entity framework doesn't generate a request that can take advantage of this index.

@NicolasDorier NicolasDorier marked this pull request as ready for review May 29, 2023 06:36
@Kukks Kukks merged commit 4e03c25 into btcpayserver:master May 29, 2023
4 checks passed
@NicolasDorier NicolasDorier deleted the foiqntq branch May 29, 2023 07:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants