Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Filebeat] Update Google Pub/Sub Input to wait for publisher ACK #13346

Closed
andrewkroh opened this issue Aug 26, 2019 · 2 comments · Fixed by #14715
Closed

[Filebeat] Update Google Pub/Sub Input to wait for publisher ACK #13346

andrewkroh opened this issue Aug 26, 2019 · 2 comments · Fixed by #14715
Assignees
Labels
enhancement Filebeat Filebeat x-pack Issues and pull requests for X-Pack features.

Comments

@andrewkroh
Copy link
Member

The Google Pub/Sub input needs to be updated to wait for the ACK from the internal libbeat publisher before ACK'ing the pub/sub message.

The current implementation ACKs the message after the message has been accepted by the internal queue, but not when the internal publisher notifies the publisher client's ACKed callback.

@andrewkroh andrewkroh added enhancement Filebeat Filebeat x-pack Issues and pull requests for X-Pack features. labels Aug 26, 2019
@andrewkroh
Copy link
Member Author

As an example the S3 input is ACK'ing its message after the callback from the libbeat publisher:

ACKEvents: func(privates []interface{}) {
for _, private := range privates {
if s3Context, ok := private.(*s3Context); ok {
s3Context.done()
}
}
},

@andrewkroh
Copy link
Member Author

@adriansr Can you please remove any beta labels from code/docs for the input type after you address this issue.

adriansr added a commit to adriansr/beats that referenced this issue Nov 25, 2019
Update the google-pubsub input to only ACK received events when an ACK
from the internal publisher is received.

Closes elastic#13346
adriansr added a commit that referenced this issue Dec 2, 2019
Update the google-pubsub input to only ACK received events when an ACK
from the internal publisher is received.

Also removes the Beta label from the input docs.

Closes #13346
adriansr added a commit to adriansr/beats that referenced this issue Jan 16, 2020
Update the google-pubsub input to only ACK received events when an ACK
from the internal publisher is received.

Also removes the Beta label from the input docs.

Closes elastic#13346

(cherry picked from commit 2ec17cc)
adriansr added a commit that referenced this issue Jan 17, 2020
Update the google-pubsub input to only ACK received events when an ACK
from the internal publisher is received.

Also removes the Beta label from the input docs.

Closes #13346

(cherry picked from commit 2ec17cc)
adriansr added a commit to adriansr/beats that referenced this issue Jan 17, 2020
Update the google-pubsub input to only ACK received events when an ACK
from the internal publisher is received.

Also removes the Beta label from the input docs.

Closes elastic#13346

(cherry picked from commit 2ec17cc)
adriansr added a commit that referenced this issue Jan 17, 2020
Update the google-pubsub input to only ACK received events when an ACK
from the internal publisher is received.

Also removes the Beta label from the input docs.

Closes #13346

(cherry picked from commit 2ec17cc)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Filebeat Filebeat x-pack Issues and pull requests for X-Pack features.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants