added a limits field to the stream configuration #254
Conversation
tulshi
added
the
v1Final
appsdesh
left a comment
appsdesh
left a comment
There was a problem hiding this comment.
I am still trying to figure out how would the receiver use this value and what change of behavior would it have?
In case of a PUSH receiver, imo this has no impact as the function of pushing still lies on the transmitter. Receiver will have no say what so ever.
In case of a POLL receiver, this will have minimal impact as well, does this require the receiver to poll more frequently? how would that help? When to poll is receiver's descretion.
This will not be a knob for backstopping as well.
I think we should punt this and think holistically on what are we solving.
| > | ||
| > If the Transmitter decides to pause or disable the stream, it MUST send a Stream Updated Event to the Receiver as described in {{status}}. | ||
|
|
||
| limits |
There was a problem hiding this comment.
| limits | |
| retention_limits |
To disambiguate a little
| > | ||
| > If the `limits` field is present, the value of this field MUST be a JSON object that contains one or more of the following members: | ||
| > | ||
| > max_events |
There was a problem hiding this comment.
max_events, max_time
A slightly different approach to this that I was thinking about yesterday is to leave out the exact values and instead use a coarse-grained priority system.
E.g. the limits or priority of the stream will be a value between 1-10 inclusive, 1 being the min and 10 the max, and 5 being the default.
The exact backend limits are then left up to the transmitter to define in their documentation, but this provides a way to simplify limit management (e.g. for a receiver, there is minimal difference between a limit of 1M max_events and 1.1M max_events) and negotiate stream priorities (e.g. streams would start out at 5 by default, receiver could request a higher or lower priority if needed, transmitter could agree or reject).
This solves the priority problem, the downside is that the limit definition is in documentation instead of the stream config - but that might give more flexibity to the transmitters, e.g. no need to send stream updated events if the limit changes.
Let me know what you think.
There was a problem hiding this comment.
I feel that we are adding constructs of the queuing service on this "delivery" mechanism.
IMO, all priority determinations should be handled by the receiver subsystem and not added to the notion of event delivery.
|
Great points, @appsdesh and @TusharR-google . Clearly, this is not as easy as I thought it would be. The idea behind this PR was exactly this - to understand how easy or difficult it will be to address this need. Let's discuss in the meeting on May 13th. |
|
One more item to consider in our discussion: How does the Transmitter indicate to the Receiver that the stream is discarding events? Should there be a new SSF event that the Transmitter generates, (e.g. "stream overflow") that indicates that the stream is dropping events? |
Outside the purpose of the FYI, how does this event affect the receiver's behavior? |
I'm not sure that sending an event for this purpose will be helpful. If the stream was paused by the receiver,it is not clear if it will be able to receive such events. If the pull is used and the transmitter is dropping events, it is likely because the receiver is not pulling events from the stream. It is also not clear when the transmitter will send the event. Only after the first time it dropped an event? Every single time an event is dropped? Etc. An alternative to sending an event is to add some output to the event stream status response for that purpose. |
I see the limits as setting expectations and not necessary as something that the receiver need to do something about. In normal conditions there is no need to the limits as there is no build up of events in the queue (beside potential temporary spikes). In all three reasons Atul mentioned, the receiver is behaving abnormally (pausing the stream, not pulling, and not being able to process the pushed events). The limits are to set the expectations of the behavior of the transmitter in those cases. |
tulshi
removed
the
v1Final
If this is purely for the FYI and no receiver behavior change is needed, then why can't we communicate this out of band? |
|
More than reacting to limits, I guess a more useful case would be the receiver negotiating priorities / limits for the stream when it is first created (or updated). Since the transmitter necessarily has limited capacity, this allows the receiver to hint at which events can be dropped with minor inconvenience vs which must be delivered no matter what. |
No description provided.