This repository has been archived by the owner on Jan 24, 2024. It is now read-only.
[transaction] Producer state manager snapshot recovery - Part-1: Add snapshot I/O buffer #1906
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report
@@ Coverage Diff @@
## master #1906 +/- ##
============================================
- Coverage 18.76% 18.35% -0.41%
- Complexity 742 743 +1
============================================
Files 185 190 +5
Lines 13266 13584 +318
Branches 1213 1257 +44
============================================
+ Hits 2490 2494 +4
- Misses 10594 10909 +315
+ Partials 182 181 -1
|
|
@BewareMyPower Can you help take a look at this PR? |
.../streamnative/pulsar/handlers/kop/storage/PulsarTopicProducerStateManagerSnapshotBuffer.java
Show resolved
Hide resolved
BewareMyPower
suggested changes
Jun 20, 2023
| /** | ||
| * Use memory to store the latest snapshot. | ||
| */ | ||
| public class MemoryProducerStateManagerSnapshotBuffer implements ProducerStateManagerSnapshotBuffer { |
There was a problem hiding this comment.
We should remove this class and related tests. Because it is only used in tests, which is meaningless.
There was a problem hiding this comment.
Here is a similar example: #1805
There is no bug with MemorySchemaStorage because it's protected by tests. However, it is never used out of tests. PulsarSchemaStorage is actually and always used and it's buggy.
There was a problem hiding this comment.
Good point, addressed.
BewareMyPower
approved these changes
Jun 20, 2023
Demogorgon314
deleted the
transaction/Support_producer_state_manager_recovery_part-1
branch
BewareMyPower
pushed a commit
that referenced
this pull request
Jun 29, 2023
…snapshot I/O buffer (#1906) This PR is a part of the Producer state manager snapshot recovery implementation, it cherry-picks from datastax/starlight-for-kafka#29. ### Motivation Introduce the `ProducerStateManagerSnapshotBuffer`. It can write and read the latest snapshot from a specified topic. This PR introduces two different `ProducerStateManagerSnapshotBuffer` implementations: * `PulsarTopicProducerStateManagerSnapshotBuffer`: Store the snapshot into the topic. * `PulsarPartitionedTopicProducerStateManagerSnapshotBuffer` : Store the snapshot in a partitioned topic. ### Modifications * Introduce `ProducerStateManagerSnapshotBuffer` interface. * Provide two diffident implementations for `ProducerStateManagerSnapshotBuffer`. * Add units test to cover the `Buffer`. (cherry picked from commit a6eb7a6)
BewareMyPower
pushed a commit
that referenced
this pull request
Jul 3, 2023
…snapshot I/O buffer (#1906) This PR is a part of the Producer state manager snapshot recovery implementation, it cherry-picks from datastax/starlight-for-kafka#29. ### Motivation Introduce the `ProducerStateManagerSnapshotBuffer`. It can write and read the latest snapshot from a specified topic. This PR introduces two different `ProducerStateManagerSnapshotBuffer` implementations: * `PulsarTopicProducerStateManagerSnapshotBuffer`: Store the snapshot into the topic. * `PulsarPartitionedTopicProducerStateManagerSnapshotBuffer` : Store the snapshot in a partitioned topic. ### Modifications * Introduce `ProducerStateManagerSnapshotBuffer` interface. * Provide two diffident implementations for `ProducerStateManagerSnapshotBuffer`. * Add units test to cover the `Buffer`. (cherry picked from commit a6eb7a6)
Demogorgon314
added a commit
to Demogorgon314/kop
that referenced
this pull request
Aug 14, 2023
…snapshot I/O buffer (streamnative#1906) This PR is a part of the Producer state manager snapshot recovery implementation, it cherry-picks from datastax/starlight-for-kafka#29. ### Motivation Introduce the `ProducerStateManagerSnapshotBuffer`. It can write and read the latest snapshot from a specified topic. This PR introduces two different `ProducerStateManagerSnapshotBuffer` implementations: * `PulsarTopicProducerStateManagerSnapshotBuffer`: Store the snapshot into the topic. * `PulsarPartitionedTopicProducerStateManagerSnapshotBuffer` : Store the snapshot in a partitioned topic. ### Modifications * Introduce `ProducerStateManagerSnapshotBuffer` interface. * Provide two diffident implementations for `ProducerStateManagerSnapshotBuffer`. * Add units test to cover the `Buffer`. (cherry picked from commit a6eb7a6)
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR is a part of the Producer state manager snapshot recovery implementation, it cherry-picks from datastax/starlight-for-kafka#29.
Motivation
Introduce the
ProducerStateManagerSnapshotBuffer.It can write and read the latest snapshot from a specified topic.
This PR introduces two different
ProducerStateManagerSnapshotBufferimplementations:PulsarTopicProducerStateManagerSnapshotBuffer: Store the snapshot into the topic.PulsarPartitionedTopicProducerStateManagerSnapshotBuffer: Store the snapshot in a partitioned topic.Modifications
ProducerStateManagerSnapshotBufferinterface.ProducerStateManagerSnapshotBuffer.Buffer.Documentation
Check the box below.
Need to update docs?
doc-required(If you need help on updating docs, create a doc issue)
no-need-doc(Please explain why)
doc(If this PR contains doc changes)