Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reliability - use MongoDB $push operator on push notification dispatch logging #36

Closed
f-w opened this issue Jun 6, 2021 · 0 comments · Fixed by #44
Closed

Reliability - use MongoDB $push operator on push notification dispatch logging #36

f-w opened this issue Jun 6, 2021 · 0 comments · Fixed by #44
Labels
enhancement New feature or request

Comments

@f-w
Copy link
Collaborator

f-w commented Jun 6, 2021

Currently the statuses of push notification dispatches to respective subscribers are aggregated at main request before writing to database once. Aggregation is vulnerable to node failures. It also makes embracing k8s HPA not feasible.

A more reliable logging method is to use MongoDB $push operator to log notification dispatch outcome for each subscription at sub-request level.

Ref implemention ideas.

This is the first step to achieve sub-request failure resilient dispatching. Next steps are

  1. save all subscription candidates before dispatching; use the candidate list instead of polling every time from database in chunk sub-request (Reliability - create and use subscription candidate list on broadcast push notification #37)
  2. exclude all subscriptions in successful and failed dispatch list when dispatching (Reliability - use successful and failed dispatch list in broadcast push notification dispatching #38)
  3. re-submit chunk sub-request when master request got http response error from sub-request (Reliability - re-submit chunk sub-requests upon failure #39)

Sub-request failure resilient dispatching is only half of node failure resilient dispatching. The other half is handling main request failure. To achieve

  1. Have the main request log a heart-beat checkpoint every pre-determined interval, say 1 min, to notification db record during the dispatching. The status of the record should be sending (Reliability - log heartbeat checkpoint during broadcast push notification dispatching #40)
  2. Create a cronJob to resend all broadcast push notifications in sending state and last modified timestamp much older than pre-determined interval. (Reliability - re-try broadcast push notification upon failure #41)
  3. Terminate chunk sub-request processing upon main request connection failure (Reliability - terminate chunk sub-request processing upon main request connection failure  #42)

Achieving node failure resilience doesn't guarantee notification will be dispatched to every intended subscriber, however. Dispatching can still be rejected by smtp/sms server or the receiving end. Rather, node failure resilience guarantees every subscription candidate will be processed by NotifyBC.

@f-w f-w added the enhancement New feature or request label Jun 6, 2021
@f-w f-w changed the title Use MongoDB $push operator on notification dispatch outcome logging Use MongoDB $push operator on push notification dispatch logging Jun 6, 2021
@f-w f-w changed the title Use MongoDB $push operator on push notification dispatch logging Reliability - use MongoDB $push operator on push notification dispatch logging Jun 7, 2021
@f-w f-w closed this as completed in #44 Jun 20, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant