Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Performance: Asynchronous garbage collection #94

Closed
cjpatton opened this issue Aug 16, 2022 · 1 comment
Closed

Performance: Asynchronous garbage collection #94

cjpatton opened this issue Aug 16, 2022 · 1 comment

Comments

@cjpatton
Copy link
Contributor

Whenever a DO instance is created, the name of the instance is sent to GarbageCollector so that the instance can be scheduled for deletion.

Centralizing garbage collection in this fashion has two major downsides:

  1. DO->DO communication incurs noticeable latency for the request that initiated it
  2. There is a risk that that the garbage collector won't keep up with the rate at which garbage is created.

@oliy suggests a different approach, which ought to scale better. Once cloudflare/workers-rs#189 lands, we can have new DO instances schedule an alarm to delete themselves when they're no longer needed:

  • HelperStateStore is needed only for the aggregation sub-protocol. This would be safe to expire after, say, an hour.
  • ReportStore is probably safe to expire after, say, a few days. We'll have to coordinate expiration with the Leader/Helper behavior, since we'll need to reject reports that correspond to expired ReportStores.
  • AggregateStore will require some thought. Certainly this should not retire before ReportStore. It may be safe to keep these around for months, given the relatively low volume. Perhaps it's acceptable to garbage collect these as usual?
  • LeaderAggregationJobQueue, LeaderCollectionJobQueue -- We should probably leave as-is until we've thought through Scaling up the Leader #25.
@cjpatton
Copy link
Contributor Author

cjpatton commented Jun 6, 2023

Closing this ticket since we basically know what our plan is here.

@cjpatton cjpatton closed this as completed Jun 6, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant