-
Notifications
You must be signed in to change notification settings - Fork 3.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support fetching metadata from entry data in publish callback #9257
Merged
sijie
merged 4 commits into
apache:master
from
BewareMyPower:bewaremypower/expose-entry-data
Jan 25, 2021
Merged
Support fetching metadata from entry data in publish callback #9257
sijie
merged 4 commits into
apache:master
from
BewareMyPower:bewaremypower/expose-entry-data
Jan 25, 2021
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
BewareMyPower
force-pushed
the
bewaremypower/expose-entry-data
branch
from
January 22, 2021 07:28
61b4a18
to
4265494
Compare
BewareMyPower
changed the title
[WIP] Support fetching metadata from entry data in publish callback
Support fetching metadata from entry data in publish callback
Jan 22, 2021
/pulsarbot run-failure-checks |
BewareMyPower
force-pushed
the
bewaremypower/expose-entry-data
branch
from
January 23, 2021 05:01
4265494
to
9f7809e
Compare
/pulsarbot run-failure-checks |
2 similar comments
/pulsarbot run-failure-checks |
/pulsarbot run-failure-checks |
Now all tests passed, PTAL @sijie @jiazhai @codelipenghui |
jiazhai
approved these changes
Jan 25, 2021
sijie
approved these changes
Jan 25, 2021
BewareMyPower
added a commit
to streamnative/kop
that referenced
this pull request
Apr 19, 2021
Fixes #332 The offset in produce callback may be not accurate when the messages are sent in batch, see #332 (comment) for detail explanation. Since apache/pulsar#9257 introduced a new API that supports fetching some metadata from the entry data, we can use this API to get the accurate offset.
1 task
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
PIP-70 introduced lightweight broker entry metadata which is added by related interceptor like
AppendBrokerTimestampMetadataInterceptor
before being written to bookie asynchronously. However, in the callback when entries are written to bookie successfully, the caller side can only get the ledger id and entry id, seePublishContext#completed
:Currently there's no way to get the broker entry data in publish callback. We can access the interceptor's internal field but it may be overwritten by the next entry's broker entry metadata because the interceptor changes the internal fields when the entry was sent while the callback is invoked when the send is done.
This PR intends to expose the entry's data to callbacks, including managed ledger's add callback and the publish context's complete callback. If we want to make use of broker entry metadata or the old message metadata in future, we can use the new callbacks.
Modifications
ByteBuf
argument toAddEntryCallback#addComplete
to expose the entry data.setMetadataFromEntryData
toPublishContext
interface to allow implementer side retrieve some metadata from entry data.AddEntryCallback#addComplete
.PublishContext#setMetadataFromEntryData
.Verifying this change
This change added tests and can be verified as follows:
ManagedLedgerTest#asyncAddEntryWithoutError
verifies the read-only buffer argumententryData
.PersistentTopicTest#testPublishMessage
verifies the new method ofPublishContext
interface.