Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support fetching metadata from entry data in publish callback #9257

Merged
merged 4 commits into from
Jan 25, 2021

Conversation

BewareMyPower
Copy link
Contributor

@BewareMyPower BewareMyPower commented Jan 21, 2021

Motivation

PIP-70 introduced lightweight broker entry metadata which is added by related interceptor like AppendBrokerTimestampMetadataInterceptor before being written to bookie asynchronously. However, in the callback when entries are written to bookie successfully, the caller side can only get the ledger id and entry id, see PublishContext#completed:

void completed(Exception e, long ledgerId, long entryId);

Currently there's no way to get the broker entry data in publish callback. We can access the interceptor's internal field but it may be overwritten by the next entry's broker entry metadata because the interceptor changes the internal fields when the entry was sent while the callback is invoked when the send is done.

This PR intends to expose the entry's data to callbacks, including managed ledger's add callback and the publish context's complete callback. If we want to make use of broker entry metadata or the old message metadata in future, we can use the new callbacks.

Modifications

  • Add an extra ByteBuf argument to AddEntryCallback#addComplete to expose the entry data.
  • Add a default method setMetadataFromEntryData to PublishContext interface to allow implementer side retrieve some metadata from entry data.
  • Add tests for AddEntryCallback#addComplete.
  • Add tests for PublishContext#setMetadataFromEntryData.

Verifying this change

  • Make sure that the change passes the CI checks.

This change added tests and can be verified as follows:

  1. ManagedLedgerTest#asyncAddEntryWithoutError verifies the read-only buffer argument entryData.
  2. PersistentTopicTest#testPublishMessage verifies the new method of PublishContext interface.

@BewareMyPower BewareMyPower changed the title [WIP] Support fetching metadata from entry data in publish callback Support fetching metadata from entry data in publish callback Jan 22, 2021
@BewareMyPower
Copy link
Contributor Author

/pulsarbot run-failure-checks

@BewareMyPower
Copy link
Contributor Author

/pulsarbot run-failure-checks

2 similar comments
@BewareMyPower
Copy link
Contributor Author

/pulsarbot run-failure-checks

@BewareMyPower
Copy link
Contributor Author

/pulsarbot run-failure-checks

@BewareMyPower
Copy link
Contributor Author

Now all tests passed, PTAL @sijie @jiazhai @codelipenghui

@sijie sijie merged commit b4ee611 into apache:master Jan 25, 2021
@BewareMyPower BewareMyPower deleted the bewaremypower/expose-entry-data branch January 25, 2021 02:48
BewareMyPower added a commit to streamnative/kop that referenced this pull request Apr 19, 2021
Fixes #332 

The offset in produce callback may be not accurate when the messages are sent in batch, see #332 (comment) for detail explanation.

Since apache/pulsar#9257 introduced a new API that supports fetching some metadata from the entry data, we can use this API to get the accurate offset.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants