Support sendTimeout

[wuYin]

Motivation

Support SendTimeout and DisableBlockIfQueueFull for producer.

Modifications

• Add 2 configurations into ProducerOptions
• Check and fail timeout messages before flushing batch.
• Check DisableBlockIfQueueFull before acquiring publish semaphore while sending messages.
• Add TestSendTimeout to verify sendTimeout effective whille backlog exceeded on topic.

Others

client-go and client-java have different ways to disable sendTimeout

• client-java: set sendTimeout to 0 explicitly, to override default 30s configuration.
• client-go: set sendTimeout to -1, to bypass unset detection.

Documentation

configure-producer

Verifying this change

• Make sure that the change passes the CI checks.

[merlimat]

Shouldn't this be taken the ctx passed in by the application at the moment of Send() ?

[wuYin]

Thanks for review.
I see PR#252 implementing sendTimeout with context but progress has stalled.
Personally think there are 2 ways to implement sendTimeout:

message level

• Perpose: precisely control the sendTimeout for each message, so that different messages can set different timeout, like consumer.Receive() did.
• Implement: set timeout context for Send, and check every message before dequeue, or something else.

producer level

• Perpose: sendTimeout for producer instance is sufficient for most situation, like client-java did.
• Implement: set timeout timer to periodic fail pending queue.
• Trade-off: optional big change, we may need remove context parameter from Send method.

I will try context If context way is better, looking forward to reply, thanks.

[merlimat]

The logic should be negated here. The default should be to block if queue full.

[wuYin]

Thanks for review.
I intented to maintain the same default behavior as client-java, but this does change the default behavior of current client-go.
I replaced NonBlockIfQueueFull with BlockIfQueueFull, just like DisableBatching for enable batch by default.

[cckellogg]

Instead of having a separate go routine checking for timeouts can the message expiry check be performed here?

I don't think the timeout check needs to be exact? We just need to expires old messages at some point? This could simplify the code and reduces the need for another lock. We can just call this method failTimeoutMessages and expire messages if needed.

[wuYin]

Thanks for review, good idea.
In some case producer.eventLoop() will be stucked such as producer is reconnecting to broker, in this case:

• Check at independent goroutine: sendTimeout still effective, periodic check and fail pending messages.
• Check before flushing: sendTimeout not effective in this case.

If no need to check message timeout precisely, the above case will be ignored.
This is reasonable, since sendTimeout now implemented at producer level, no need to be that precise, also avoided cost of pending queue lock.

[wuYin]

@cckellogg
I used sentAt for checking timeout of messasges.
sentAt always be set before puting into pending queue, so I removed this predicate.
Like sendRequest.publishTime did.

[cckellogg]

Looks ok to me

[cckellogg]

How can this be disabled? Setting it to -1?

[wuYin]

Yes, negative to disable, such as -1 typically.
https://github.com/apache/pulsar-client-go/blob/master/pulsar/producer.go#L81