Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Batch processing of notifications #193

Open
c0c0n3 opened this issue May 16, 2019 · 1 comment
Open

Batch processing of notifications #193

c0c0n3 opened this issue May 16, 2019 · 1 comment

Comments

@c0c0n3
Copy link
Member

c0c0n3 commented May 16, 2019

PR #191 added support to store multiple entities in notification messages---btw, kudos to @ohylli. But we still need to think about how to handle:

  1. Partial errors. What to do if we encounter an error for some of the entities in the batch? e.g. what if 2 out of 20 entities fail validation? should we reject the whole batch (probably not a good idea!) or try saving the 18 that passed validation? What would the HTTP response look like in this case? How do we accumulate errors?
  2. Message size. What to do with large batches---e.g. 100s of entities? Ideally we should process them in constant space to avoid eating up all available RAM. This applies both to the received HTTP message and the DB call we issue.

For now this is just a reminder that at some point in time we'll have to consider the points above. Also, we should probably split (1) and (2) in two separate PRs.

Initial thoughts about a possible way to implement

  1. Leverage monoids/monads/monoidal functors as suggested on stackoverflow
  2. Use stream processing techniques to process a batch of batches of configurable size.
@github-actions
Copy link
Contributor

Stale issue message

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants