Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

limit amount of messages received by a consumer #892

Open
aaleksandrov opened this issue Jan 14, 2020 · 4 comments
Open

limit amount of messages received by a consumer #892

aaleksandrov opened this issue Jan 14, 2020 · 4 comments

Comments

@aaleksandrov
Copy link

aaleksandrov commented Jan 14, 2020

Hi!
We're using this library with https://github.com/cometd/cometd-nodejs-client for fetching server-to-server from Salesforce. And when Salesforce publishes many messages (5k+) our consumer can't process it, runs out of memory etc.
Is it possible to limit amount of messages received by consumer? I cannot find anything regarding limits in cometd

@sbordet
Copy link
Member

sbordet commented Jan 14, 2020

CometD is notifying message listeners synchronously: it reads from the network, it calls the listeners and does not read from the network again until the listeners return.

This naturally applies backpressure so that the client does not explode.

If the server sends down a single enormous message, or a single array of messages of an enormous length, then it's the server assuming too much of what the client can do, and I would see that as a server-side application mistake (it should split the enormous message, or split the array into smaller arrays).

I'm not that familiar with SalesForce message stream, but you should look into whether SalesForce allows you to configure how messages are sent from the server. For example, you may subscribe to less channels, or ask to be notified less frequently, etc.

lmcalpin added a commit to lmcalpin/cometd that referenced this issue May 6, 2020
…uffering it.

Signed-off-by: Lawrence McAlpin <lmcalpin@gmail.com>
@sbordet
Copy link
Member

sbordet commented Oct 6, 2020

FTR, in the case of many small messages, it may be possible to write an extension where the client declares its demand, similar to reactive streams.

This will require collaboration of the server, and delivery via /meta/connect only.

The client will send demand: 10 the server will have an extension similar to the ack extension where the messages are kept around and only 10 are delivered via /meta/connect. The client receives the 10, and sends a new /meta/connect with demand: 15, etc.

This moves the queuing problem to the server, where it's already handled by queue listeners, although it needs to be carefully coded - if the client is really slow, what can the server do? disconnecting the client will just make the client connect again, making it even slower. But again this is logic that only applications can implement, not CometD.

Needs some experimentation to see if there are unforeseen issues, but looks promising.

@surajitgit
Copy link

surajitgit commented Dec 1, 2020

Hi.. I'm using CometD-Java Client 4.0.9 version. I'm stuck with something..
We have microservice which subscribes and continuously listens to Salesforce cometD channel, once a message is available in the channel, listener microservice immediately consumes it but the issue is, if we scale up listener MS instance more than one, both the instances would consume copy of the same message hence duplicate records are being produced wrongly.
Is there anything we could perform which would restrict other instance to consume the same message i.e. already consumed by other instance?

@sbordet
Copy link
Member

sbordet commented Dec 1, 2020

You have 2 subscribers to the same channel, so for the server these are 2 different subscribers and it will send the same message to both.

It's only on the client that you know that the 2 subscribers are in fact the same subscriber.

There are plenty solutions for this architectural problem, from shared in-memory queues, to shared RDBMS, to shared storages such as Redis and such, etc. but it's in general outside the scope of CometD -- there are too many variations due to the application behavior (may or not tolerate duplicates), the architecture, etc. that CometD cannot cover them all.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants