Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RFC] Push Notifications API #469

Open
omarroth opened this issue Apr 6, 2019 · 7 comments

Comments

@omarroth
Copy link
Owner

commented Apr 6, 2019

Related issues: #195, #427

Currently, in order for an application to receive updates from a channel, it needs to send a request to /api/v1/channels/latest/:ucid or similar endpoint. This is inefficient when the number of channels being requested reaches even moderate size. This propsal is for a /api/v1/notifications that uses EventSource to push notifications in near real-time.

The EventSource protocol pushes updates using a persistent connection to an HTTP server. It is supported in all major browsers with a polyfill available for IE. The EventSource API can automatically reconnect to the source on errors, and provides a simple API for handling events. There is also a wrapper reconnecting-eventsource for handling more challenging network conditions.

The EventSource protocol is quite simple and can be implemented easily in any language where it isn't already supported. It sends comments (lines prefixed with :), or events containing: one or more data fields, an optional id, and an optional type. After each message is sent a trailing newline. Example from the spec:

: test stream

data: first event
id: 1

data:second event
id

data:  third event

/api/v1/notifications

The endpoint /api/v1/notifications would support a topics parameter containing a list of comma-separated resources. Each resource would be a UCID or any other supported ID. /api/v1/notifications would support a maximum of 1000 topics per call.

Opening a connection to /api/v1/notifications creates a persistent connection between the server and the client, with :keepalive #{TIMESTAMP} messages sent every 20-30 seconds to ensure the connection is not dropped.

Each event would contain a data field with the updated resource as JSON. Schema would be the same as /api/v1/channels/latest/:ucid.

Example output from cURL:

$ curl https://invidio.us/api/v1/notifications?topics=UCAAAA,UCBBBB
:keepalive 1554427320

id: 0
data: {"title":"...","videoId":"AAAAAAAAAAA","author":"...","authorId":"UCAAAA","authorUrl":"/channel/UCAAAA","videoThumbnails":[{"quality":"...","url":"...","width":120,"height":90}],"description":"...","descriptionHtml":"<div>...</div>","viewCount":1,"published":1554474212,"publishedText":"1 days ago","lengthSeconds":10,"liveNow":false,"paid":false,"premium":false}

:keepalive 1554427333

:keepalive 1554427335

id: 1
data: {"title":"...","videoId":"BBBBBBBBBBB","author":"...","authorId":"UCBBBB","authorUrl":"/channel/UCBBBB","videoThumbnails":[{"quality":"...","url":"...","width":120,"height":90}],"description":"...","descriptionHtml":"<div>...</div>","viewCount":1,"published":1554474212,"publishedText":"1 days ago","lengthSeconds":10,"liveNow":false,"paid":false,"premium":false}

Example handling in JavaScript:

var notifications = new EventSource('https://invidio.us/api/v1/notifications?topics=UCAAAA,UCBBBB');
notifications.onmessage(event) {
    var notification = JSON.parse(event.data); // => {title: "...", videoId: "AAAAAAAAAAA", ...

    // Update feed, notify user, or any other necessary handling for the event
}

Polling for notifications after being offline

In order for an application to receive notifications after being offline , a ?since=TIMESTAMP should be added to /api/v1/channels/latest/:ucid. Using ?since= would return up to 15 videos uploaded after the specified timestamp. If there are more than 15 videos that have been uploaded since the given date, /api/v1/channels/latest/:ucid will return the 15 latest.

It may also make sense to support an ?until=TIMESTAMP to create a "window" where specific videos can be queried, although this would require authentication as mentioned below.

Final thoughts

Notifications would only be supported for channels tracked by the instance, which means that one or more users on that instance are subscribed to them. Requesting that a new channel be tracked by an instance must be authenticated to prevent abuse. Authentication will be handled in a second RFC.

Important to note is that notifications will be sent when a channel changes a video's description or title, not just when it is uploaded.

In accordance with this spec, a "protocols":["eventsource"] field should be added to /api/v1/stats, example. See also #356.

I would like to discuss if there are any issues with this approach, or any other issues that haven't been considered to ensure this is provides as much utility as possible.

Thanks!

@omarroth omarroth added the enhancement label Apr 6, 2019
@omarroth

This comment has been minimized.

Copy link
Owner Author

commented Apr 7, 2019

A couple thoughts on the "Polling for notifications after being offline" section after some discussion in the Matrix server:

The ?since=#{TIMESTAMP} param should be added to /api/v1/notifications, not (or in addition to) /api/v1/channels/latest/:ucid. This would provide all published events for the subscribed topics up to a maximum of 15 since the specified TIMESTAMP, then continue serving events as normal. If there are more than 15 videos that have been uploaded since the given date, /api/v1/notifications will return the 15 latest from each channel.

Example output from cURL:

$curl https://invidio.us/api/v1/notifications?topics=UCAAAA,UCBBBB&since=1554597448
:keepalive 1554597449

id: 0
data: {"title":"...","videoId":"AAAAAAAAAAA","author":"...","authorId":"UCAAAA","authorUrl":"/channel/UCAAAA","videoThumbnails":[{"quality":"...","url":"...","width":120,"height":90}],"description":"...","descriptionHtml":"<div>...</div>","viewCount":1,"published":1554474212,"publishedText":"1 days ago","lengthSeconds":10,"liveNow":false,"paid":false,"premium":false}

id: 1
data: {"title":"...","videoId":"BAAAAAAAAAA","author":"...","authorId":"UCAAAA","authorUrl":"/channel/UCAAAA","videoThumbnails":[{"quality":"...","url":"...","width":120,"height":90}],"description":"...","descriptionHtml":"<div>...</div>","viewCount":1,"published":1554474212,"publishedText":"1 days ago","lengthSeconds":10,"liveNow":false,"paid":false,"premium":false}

id: 2
data: {"title":"...","videoId":"CAAAAAAAAAA","author":"...","authorId":"UCAAAA","authorUrl":"/channel/UCAAAA","videoThumbnails":[{"quality":"...","url":"...","width":120,"height":90}],"description":"...","descriptionHtml":"<div>...</div>","viewCount":1,"published":1554474212,"publishedText":"1 days ago","lengthSeconds":10,"liveNow":false,"paid":false,"premium":false}

:keepalive 1554597470

:keepalive 1554597493

id: 3
data: {"title":"...","videoId":"BBBBBBBBBBB","author":"...","authorId":"UCBBBB","authorUrl":"/channel/UCAAAA","videoThumbnails":[{"quality":"...","url":"...","width":120,"height":90}],"description":"...","descriptionHtml":"<div>...</div>","viewCount":1,"published":1554474212,"publishedText":"1 days ago","lengthSeconds":10,"liveNow":false,"paid":false,"premium":false}

Although this would need to run an expensive query against the DB depending on the number of topics, so calls using ?since=TIMESTAMP should be authenticated.

@cloudrac3r

This comment has been minimized.

Copy link

commented Apr 9, 2019

It's unclear how authentication ties in to all of this.

@omarroth

This comment has been minimized.

Copy link
Owner Author

commented Apr 9, 2019

In order for an instance to provide notifications for a channel, said channel needs to be tracked. To track a channel means the instance is subscribed to a channel's RSS feed and/or has a job running to pull all videos/update older videos from a channel.

Tracking channels can be resource intensive so currently it is only done when at least one user has previously subscribed to a channel (it will still track after unsubscribing). To ensure that a channel is being tracked, a client should send a POST /subscriptions/:ucid as mentioned here, which can then be followed by a DELETE /subscriptions/:ucid if there isn't a need to generate a corresponding subscription feed.

Authentication is needed so that a client can ensure it will receive notifications for a channel, while not allowing an easy vector for abuse.

@omarroth

This comment has been minimized.

Copy link
Owner Author

commented Apr 9, 2019

Additionally, supporting a ?since= parameter would need to run a query against all the specified topics, so that would also require authentication, since the query could get quite expensive.

@omarroth

This comment has been minimized.

Copy link
Owner Author

commented Apr 20, 2019

Added /api/v1/auth/notifications with fb7068d. ?since=TIMESTAMP has not yet been implemented.

Since each message contains the entire /api/v1/videos/:id response, it is recommended to use the fields API (see #429) to request only necessary data for providing notifications. For example &fields=videoId,title,author,authorId,published,isLive.

Important to note is that notifications will also be sent whenever a video is updated for example when a description or title is changed. It is recommended to use logic similar to here to provide consistent notifications for new videos.

@omarroth

This comment has been minimized.

Copy link
Owner Author

commented Apr 29, 2019

Added ?since=TIMESTAMP with 54d250b.

@omarroth

This comment has been minimized.

Copy link
Owner Author

commented May 21, 2019

Added POST /api/v1/auth/notifications with b3e083d.

Although there isn't technically a limit on the size of a GET request, things can start to fail in weird ways after a certain limit. This provides a way to access the advertised 1000 topics per call by putting the topics field into the body.

If you're using something like sse.js, you can do:

notifications = new SSE(
    '/api/v1/auth/notifications?fields=videoId,title,author,authorId,publishedText,published,authorThumbnails', {
        withCredentials: true,
        payload: 'topics=' + subscriptions.join(','),
        headers: { 'Content-Type': 'application/x-www-form-urlencoded' }
    });

notifications.onmessage = function (event) {
    if (event.data === '') {
        return;
    }

    var notification = JSON.parse(event.data);
    // ...
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.