Issue using FlowControlSettings and setting setMaxOutstandingElementCount = 1

[meredithslota]

Opening a new issue to debug the comment from user @dgrudenic in #17:

"We actually experienced same behavior mentioned in example above.

So if you have 3 subscribers on same subscription/topic and you want that each subscriber is processing 1 message at the time which is kind of standard point to point workflow it is not really possible with FlowControlSettings and setting setMaxOutstandingElementCount=1 because as soon as first subscriber gets message it will buffer more than 1 message.

Let me know if this is not the case and example above is actually working."

Originally posted by @dgrudenic-qumu in #17 (comment)

[meredithslota]

@anguillanneuf The user comment above references some challenges they are having using flow control settings — the older ticket was a feature request so I opened a new issue instead of reopening the older ticket. The relevant example is in the original ticket.

[anguillanneuf]

if you have 3 subscribers on same subscription/topic and you want that each subscriber is processing 1 message at the time

@dgrudenic-qumu Setting max outstanding element count to 1 while using 3 subscribers to pull messages doesn't sound like a good use case for StreamingPull. I would suggest using synchronous pull instead.

setMaxOutstandingElementCount=1 does not guarantee that there's exactly 1 message between the Pub/Sub service and your subscriber. The subscriber is responsible for 1) holding only one message in its memory 2) making sure that this message has its acknowledgement deadline extended and 3) returning further messages to the server. This subscriber, however, has no control over whether Pub/Sub will redeliver the returned messages to itself or the other two subscribers that have also open streaming connections to the Pub/Sub service. It will be helpful to keep in mind that the gRPC buffer between Pub/Sub service and the subscriber is roughly 10 MiB, so up to that many messages can be delivered back and forth in the gRPC streams.

To effectively load-balance across all your subscribers when the message load is small, or to achieve the goal of never starving a subscriber when you have a small message load, I would recommend using synchronous pull. Here's an example,
https://github.com/googleapis/java-pubsub/blob/HEAD/samples/snippets/src/main/java/pubsub/SubscribeSyncExample.java