Native Notifications #1428

ansh · 2023-08-03T01:54:17Z

Types of notifications handled

Misc items

Separate AppView and PDS concerns
Move notification sending out of processor.ts into indexing service
Duplicate code in AppView
Add tests
Rate limiting and queueing

Future:

Show image previews in notification (need to do some CDN resizing/compression stuff for this) -- not very important
Handle notifs for web (gorush does not support web notifs)

…push_token` table

packages/bsky/src/services/indexing/processor.ts

…ions

dholms · 2023-08-22T16:41:24Z

packages/bsky/src/services/indexing/processor.ts

@@ -230,6 +233,17 @@ export class RecordProcessor<T, S> {
      notifs = this.params.notifsForInsert(op.inserted)
    }
    for (const chunk of chunkArray(notifs, 500)) {
+      if (this.notifServer) {


seems nicer to do this out in the indexing service instead of within the processor which I view as a "raw record -> DB indexes" translator 🤔

any reason to prefer it here?

I put it here in order mostly to make use of the existing code paths for notification handling, make it simpler to answer "where are notifications sent?", ensure the same logic was being applied no matter the destination of the notifications. Previously notifications that may be generated from a deletion weren't being handled, and the added notifsForInsert() calls down in insertRecord()/updateRecord() felt a bit leaky/special-cased to me.

I do see what you're saying, though. I may just have a slightly different perspective on it—that the destination of the notifications (in the db or off to some push service) isn't all that significant. And that having one code path for notifications is useful.

My guess is that we could get the best of both worlds with a little refactor! But now might not be the time for it. Happy to switch it back to the way it was if you think that's the move 👍

nah that's cool I dig the justification 👍

if we end up maturing this system, that might be a good time to move it around a bit

dholms · 2023-08-22T16:42:49Z

packages/bsky/src/notifications.ts

+    const allUris = [...subjectUris, ...recordUris]
+
+    // gather potential display data for notifications in batch
+    const [authors, posts] = await Promise.all([


i still wonder if we can avoid most of these db hits by passing the record into the notification service?

i will see - these bulk gets give a decent justification for processing notifs in batches

I believe that does help with the post record lookup for mentions and replies, which do show the text from the post that was just indexed. The author display name lookups for all notifs and post record lookups for likes, quotes, reposts will still need to happen at some point. The changes I made here tried to batch work together into two queries rather than a few queries per notification. Batching more notifications together as you're suggesting will probably be what gives the biggest win here.

…ions

* pseudocode for sending notification to user * add notification push token table * lexicon codegen * `pds` and `api` codegen * update lexicon * add simple function to `putNotificationPushToken` into `notification_push_token` table * reorgnaize imports * add unspecced `putNotificationPushToken` to pds * add `notification-push-tokens` table to PDS * check if token exists before adding it to db * add endpoint and appId to PDS table * setup notification server * fix logic for inserting token into db * fix NotificationServer methods by making them static * fix merge conflicts * add comments on how sending ntoifications work * remove dead code * move notifServer to AppContext * refactor code to handle notif sending in `Indexer` service * add additional data when sending notifs * clean up code * move notif logic to indexer * add `appId` and `endpoint` optional params to putNotificationPushToken * clone notification code to AppView * add endpoint to register push token with app view * disable pds and enable app view notification server * clean up code * simply logic to check if a token already exists * remove NotificationServer from PDS * remove notification-push-token table from PDS * remove `putNotificationPushToken` endpoint * clean up code * let `axios` throw error if `gorush` has an error * let `kysely` throw error if notif cannot be registered by client * rename `registerPushNotificationEndpoint` to `registerPushNotifications` * delete `putNotificationPushToken` from AppView * rename putPushNotificationToken to registerPushNotification * remove dead notification code from pds * remove sanitizeDisplayName from NotificationSever * move `pushNotificationEndpoint` to config * temp add `pushNotificationEndpoint` to dev-env setup * remove example test from feed-generation.test.ts * add test for registerPushNotification and clean up error handling * move notifications test to its own file * add test for NotificationServer to check if tokens are retrieved correctly * remove unused functions from NotificationServer * add additional tests for NotificationServer * add return type to getNotificationDisplayAttributes function * remove unnecessary comments * remove dead NotificationServer code from PDS * clean up code to prepareNotifsToSend * put sending notifications as part of the backgroundQueue * log instead of throwing error if notification attributes don't exist * remove logs * add more tests to `notification-server.test.ts` * show replied with text for reply notifs * better error handling when sending notifications via backgroundQueue * add rate limit and batching to sending notifications * add comments to NotificationServer * merge with main * use redis for rate limiting instead of normal rate limits * move `notificationBatchSize` into config * usePrimary db in test * hoist push notif migration to present, update model to remove endpoint * update push notif lexicon * pare down lex for unregistering push * helpers for working with service endpoints from did doc * service-authed register/unregister push notifs * add well-known endpoint to appview * update bsky notif service tests * fix to batching logic, misc tidy * batch display notifications * colocate all notification sending logic * tidy tests * remove unregister token for now * fix registerPush lexicon, make a procedure * fix registerPush impl, test pds proxy to notif service * fix tests, make notif server optional when not configured * fix notif server config for bsky app service * move notif server rate limiting in-mem for now, add sending retry * codegen tidy * only push notifs on commit * build * fix notif rate limiter check * send notifs from users w/o a profile * remove build --------- Co-authored-by: dholms <dtholmgren@gmail.com> Co-authored-by: Devin Ivy <devinivy@gmail.com>

ansh and others added 6 commits August 2, 2023 16:20

pseudocode for sending notification to user

af8d372

add notification push token table

15dbbcf

lexicon codegen

30d5d06

pds and api codegen

9bceb8a

update lexicon

f193cfc

add simple function to putNotificationPushToken into `notification_…

553e1ce

…push_token` table

ansh mentioned this pull request Aug 3, 2023

[APP-786] Native notifications bluesky-social/social-app#1095

Merged

3 tasks

ansh added 4 commits August 3, 2023 10:41

reorgnaize imports

4cb75d3

add unspecced putNotificationPushToken to pds

7c58dd2

add notification-push-tokens table to PDS

7dd4837

check if token exists before adding it to db

7dd60c9

dholms reviewed Aug 4, 2023

View reviewed changes

packages/bsky/src/services/indexing/processor.ts Outdated Show resolved Hide resolved

ansh added 18 commits August 4, 2023 17:17

add endpoint and appId to PDS table

31b63b3

setup notification server

654fea8

fix logic for inserting token into db

ae5dd0b

fix NotificationServer methods by making them static

fec348f

Merge branch 'main' into ansh/native-notifications

28c5671

fix merge conflicts

fec52ef

add comments on how sending ntoifications work

41703fa

remove dead code

261052d

move notifServer to AppContext

7185ff6

refactor code to handle notif sending in Indexer service

9b8a91b

Merge branch 'main' into ansh/native-notifications

7cb30eb

add additional data when sending notifs

e91c86d

clean up code

fb4849a

move notif logic to indexer

38ed02f

add appId and endpoint optional params to putNotificationPushToken

470059f

clone notification code to AppView

871b03b

add endpoint to register push token with app view

2b1d67d

disable pds and enable app view notification server

6876820

devinivy added 14 commits August 17, 2023 18:11

helpers for working with service endpoints from did doc

6106410

service-authed register/unregister push notifs

0da644a

add well-known endpoint to appview

29ccc99

update bsky notif service tests

06b2f70

fix to batching logic, misc tidy

5abe623

batch display notifications

a188482

colocate all notification sending logic

6c56645

tidy tests

79cd44b

remove unregister token for now

2beeca7

fix registerPush lexicon, make a procedure

3b61fcf

fix registerPush impl, test pds proxy to notif service

10abfc3

Merge remote-tracking branch 'origin/main' into ansh/native-notificat…

22024f0

…ions

fix tests, make notif server optional when not configured

557ea82

fix notif server config for bsky app service

cc37403

dholms reviewed Aug 22, 2023

View reviewed changes

devinivy and others added 9 commits August 22, 2023 13:12

move notif server rate limiting in-mem for now, add sending retry

410afd5

Merge remote-tracking branch 'origin/main' into ansh/native-notificat…

7f2421e

…ions

codegen tidy

ab6b81e

only push notifs on commit

1b93f33

build

1515b8f

fix notif rate limiter check

378276c

merge main

f61e292

send notifs from users w/o a profile

5b1e20a

remove build

12e01d4

devinivy merged commit eae682f into main Aug 23, 2023

devinivy deleted the ansh/native-notifications branch August 23, 2023 22:37

devinivy mentioned this pull request Sep 13, 2023

Push notifications #484

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Native Notifications #1428

Native Notifications #1428

ansh commented Aug 3, 2023 •

edited

Loading

dholms Aug 22, 2023

devinivy Aug 22, 2023 •

edited

Loading

dholms Aug 22, 2023

dholms Aug 22, 2023

devinivy Aug 22, 2023

Native Notifications #1428

Native Notifications #1428

Conversation

ansh commented Aug 3, 2023 • edited Loading

dholms Aug 22, 2023

Choose a reason for hiding this comment

devinivy Aug 22, 2023 • edited Loading

Choose a reason for hiding this comment

dholms Aug 22, 2023

Choose a reason for hiding this comment

dholms Aug 22, 2023

Choose a reason for hiding this comment

devinivy Aug 22, 2023

Choose a reason for hiding this comment

ansh commented Aug 3, 2023 •

edited

Loading

devinivy Aug 22, 2023 •

edited

Loading