New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(pubsub): retry temporary failures for ack/modacks #6485
feat(pubsub): retry temporary failures for ack/modacks #6485
Conversation
599fa5d
to
2c0979e
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Overall looked good, just one nit/question.
for k := range m { | ||
ackIDs = append(ackIDs, k) | ||
} | ||
log.Printf("automatic lease modack retry failed for following IDs: %v", ackIDs) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: does pubsub log today? I don't think most(any?) of our clients do right now
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Summarizing our direct conversation, we don't log right now and most Go clients don't. There should be a better way to do logging and we should expose hooks for users to do this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
* feat(pubsub): prepare iterator for exactly once (#6040) * feat(pubsub): read exactly once for SubscriptionProperties * rename vars to be specific this is exactly once delivery * feat(pubsub): send stream ack deadline seconds on exactly once change #6157 (#6162) * add RWMutex for guarding exactly once bool * feat(pubsub): send stream ack deadline seconds on exactly once change * remove extra test * feat(pubsub): add AckWithResult and NackWithResult to message (#6201) * add AckResult and related methods * feat(pubsub): add AckWithResult and NackWithResult to message * feat(pubsub): add AckWithResult and NackWithResult to message * add comments for AckResult and bring over AcknowledgeStatus from internal * update function definition for IgnoreExported in tests * temporarily update internal/pubsub for samples test * change enum naming to AcknowledgeStatus * remove extra enums in temp internal message.go * remove internal/pubsub/message.go * fix style issues with variadic function options * add back comment format to exported const * keep track of AckResults if exactly once is enabled * feat(pubsub): add helper method for parsing ErrorInfos (#6281) * add AckResult and related methods * feat(pubsub): add AckWithResult and NackWithResult to message * feat(pubsub): add AckWithResult and NackWithResult to message * add comments for AckResult and bring over AcknowledgeStatus from internal * update function definition for IgnoreExported in tests * temporarily update internal/pubsub for samples test * add process results * change enum naming to AcknowledgeStatus * remove extra enums in temp internal message.go * remove internal/pubsub/message.go * add process results * update process info with new enum names * add tests to process error info * add process results * update process info with new enum names * add process results * add tests to process error info * clean up iterator from merge * cleanup comments * add list of retriable errors to test * simplify testing of completed/retry slice lengths * remove getStatus/ackErrors methods * address code review comments * remove error string conversion step * feat(pubsub): complete AckResult for exactly once (#6387) * refactor sendAck to pipe errors to AckResult map * rewrite sendAck/sendModAck for exactly once * add AckResult to list of uncompared methods * use ackResultWithID in all locations * feat(pubsub): retry temporary failures for ack/modacks (#6485) * retry acks in goroutine * retry acks/modacks with transient errors * add retry test * add nack tests and support shorter timeouts * add integration tests * remove extra comment * add commnets to ack/modack methods in iterator * remove transient invalid ack id error string * reduce number of mutex locks * pass in StreamAckDeadline seconds for streaming pull requests in fake_test * fix lint issues * add changes to internal/pubsub/message * implement default ack handler functions in lite * use pubsub package ack result * use pinned library for pubsublite * resolve all lite Ack/NackWithResult to success
This PR adds the logic for retrying temporary ack/modack (including nack) failures for exactly once delivery.
Some other changes captured in this PR
subscription_test.go
andintegration_test.go
to support exactly once delivery for publish/subscribing.(Git diff is not great here, but a lot of changes just involves moving a lot of the publish/receive test code into a sub-test function so they can be invoked in parallel. Not much logic change.)
ackResultWithID
to just beAckResult
. Passing around the map of ackID->AckResult works best and we just needed to be consistent.