Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Signal backpressure? #1469

Closed
durban opened this issue May 15, 2019 · 5 comments

Comments

Projects
None yet
2 participants
@durban
Copy link
Contributor

commented May 15, 2019

Is Signal (or SignallingRef) intended to have backpressure?

I'm not actually sure that backpressure is the correct term to use here. I'm thinking about the fact, that Topic is documented as having "built-in back-pressure support", and indeed, it seems to semantically block publishers if subscribers are slower. The implementation of SignallingRef seems to have a similar thing underlying it (PubSub), however "publish" seems to be forked (see here for example). Could this cause a problem if, e.g., someone modifies the signal much faster than a reader can process the discrete stream?

@SystemFw

This comment has been minimized.

Copy link
Contributor

commented May 15, 2019

PubSub underlies every abstraction (Queue, Topic, Signal, Broadcast).
There is no problem in this case because the semantics of Signal are that it represents the current value.
If you write too fast, you are going to lose updates and only see the most recent ones, which makes sense in certain use cases, and Signal is what you need for those.
If you don't want to lose updates, then you need Topic, which was designed exactly as a Signal that does not lose updates, as opposed to a Queue with broadcast, and that's why Topic requires an initial value like Signal (we might want to change the name and provide the "Queue with broadcast" semantics as well, since the initial value on Topic trips people up).
Because Topic does not lose updates, you need back pressure (and Topic has it), and because Signal only guarantees you will see the most up to date value, you don't need back pressure.

I hope this clarifies things :)

@durban

This comment has been minimized.

Copy link
Contributor Author

commented May 15, 2019

Yes, I definitely want to lose updates, and that's why I wanted to use Signal. I probably shouldn't have used the term "backpressure", because it seems that's not what I actually want.

What I'm wondering about is this: if I use Signal, and I have an arbitrarily big number of updates, while the consumer still consumes the first one, I will "lose" some updates, in the sense that when the consumer "catches up", it will receive the most recent one. (Right?) However, I don't "lose" them, in the sense that they will still "exist" as "forked fibers" (is that correct terminology?). And at the end of the day, those "tasks" will be in the queue of a thread pool somewhere. So at least they will consume memory, and possibly CPU too(?).

So my question is, basically, shouldn't we cancel those fibers somewhere/somehow?

@SystemFw

This comment has been minimized.

Copy link
Contributor

commented May 15, 2019

well, first of all "a lot of updates" can mean different things, it could be a single producer updating the signal sequentially very fast, or a lot of producers updating the signal concurrently.

I do think however you have an incorrect model for what "losing updates" means. You seem to think that losing updates is due to fibers being queued up, i.e you lose newer updates, whereas you lose older ones.

However, I don't "lose" them, in the sense that they will still "exist" as "forked fibers" (is that correct terminology?)

No, if you lost them it means that they have successfully updated the Signal (so they go on doing whatever else they do, they are not stuck waiting), and then newer ones have overriden that value. Does that make sense?

@durban

This comment has been minimized.

Copy link
Contributor Author

commented May 15, 2019

Yeah, sorry, I've completely misunderstood how this works. I'm now reading the PubSub.Strategy.Discrete implementation, and starting to understand. Thanks for the explanation, it helps somewhat.

@SystemFw

This comment has been minimized.

Copy link
Contributor

commented May 15, 2019

Cool, I'm going to close this for now, but feel free to ask other questions either here or on gitter :)

@SystemFw SystemFw closed this May 15, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.