New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow pagination in subscriptions #591

Open
avoulk opened this Issue Oct 22, 2014 · 9 comments

Comments

Projects
None yet
4 participants
@avoulk
Copy link

avoulk commented Oct 22, 2014

When issuing a subscription request, Orion responds among others with a number of discovered entities, depending on the subscription specifics. However, the received entities seem to follow the general pagination rule, meaning that the first sets of received data are limited to 20 entities. Data updates are normal from this point on, without being paginated.

Would it be possible to allow for controlling (initial) pagination in subscription requests?

@fgalan fgalan added bug P3 labels Oct 23, 2014

@fgalan

This comment has been minimized.

@njoylab

This comment has been minimized.

Copy link

njoylab commented Apr 20, 2015

Hi, is there any plan on this? It would be nice to have a parameter in the config file

@fgalan

This comment has been minimized.

Copy link
Member

fgalan commented Apr 20, 2015

Probably the best approach should be to include the limit parameter on the subscribeContext request.

@kzangeli

This comment has been minimized.

Copy link
Member

kzangeli commented May 7, 2015

What if we have 1,000,000 entities of id 'Car' and somebody makes a subscription of 'Car' ...
For queries it's OK, we just respond with 20 'Cars' (or more if URI param limit is used'), but, for subscriptions, we can't really send all those one million entities. This would practically stall the broker for a few minutes ...

One way to deal with this problem would be to detect this kind of situation and respond with an error saying something like:

'Your search gives too many responses - please narrow it down and try again'.

@kzangeli

This comment has been minimized.

Copy link
Member

kzangeli commented May 7, 2015

Also, when creating the subscription the URI param limit should be supported and saved in the database. If the subscription is accepted, all notifications should be sent, 20 at a time ('limit' at a time).

@kzangeli

This comment has been minimized.

Copy link
Member

kzangeli commented May 7, 2015

Now, what if the broker accepts a subscription of entityId 'Car', as we only have 5 entities of that id right now in the data base, and then a million entities 'Car' are added ...

Perhaps we need to detect this situation as well, and avoid sending a million notifications, but ... in case we decide not to send notifications, how do we inform the subscriber about this?

@fgalan

This comment has been minimized.

Copy link
Member

fgalan commented May 7, 2015

Also, when creating the subscription the URI param limit should be supported and saved in the database. If the subscription is accepted, all notifications should be sent, 20 at a time ('limit' at a time).

For ONCHANGE subscriptions only the changed entity is notified, so probably we don't need storing it.

@fgalan

This comment has been minimized.

Copy link
Member

fgalan commented May 28, 2015

After internal discussion, we conclude that sending an "unlimited" initial notification at initial time (either in one message or pagginated) is not a good idea. For example, let's image an ContextBroker managing all the cars in a big city (each car modeled as an entity). We can have easily a couple of million entities. If an application wants so subscribe to all cars (.* pattern), the notification would include million of entities, which could impact seriously in the broker perfomance.

Thus, this issue will be about implementing the same limit URL argument we have in query operations also in subscriptions operations, so the user could chose a different limit that 20 (but with a hard limit of 1000). If that doesn't suffices, the receptor has always the option of using queryContext before subscribing.

@fgalan

This comment has been minimized.

Copy link
Member

fgalan commented Oct 23, 2017

initial_notification.md#additional-considerations should be adjusted once this issue gets completed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment