-
Notifications
You must be signed in to change notification settings - Fork 201
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Merged by Bors] - feat: added topic-level deduplication mechanism #3385
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
first pass
} else { | ||
error!("follower replica {} didn't exists!", old_replica.id); | ||
} | ||
outputs | ||
} | ||
|
||
pub async fn apply_replica_update( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is these formatting changes only? If so, would prefer to move out of this PR since it's not related to this feature
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is not formatting. This is propagating errors up from replica initialization.
use std::{ | ||
ops::{Deref}, | ||
}; | ||
use std::ops::Deref; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same comment as previously. let's move formatting changes out of this PR
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If I modify a file where there are warnings from the rust-analyzer, I fix them. In that way, such warnings will be gradually fixed. Nobody ever will fix those warnings separately unless all tasks on the project are gone.
This is a rather large PR. Can this break it into smaller chunks? Suggest
|
Ok, this is big, indeed. I am going to split it into parts. |
First part #3392 |
This is the first part of [Dedup Impl PR](#3385) split for a more convenient review process.
This PR only contains SPU/SC changes |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. Great work interleaving Smartmodule in existing replica lifecycle
bors r+ |
Upd.: Split into two parts: [first](#3392) Added topic-level deduplication mechanism. Key impacted components: - Topic Config - Fluvio CLI (topic creation, topic list, topic describe) - SPU (replica and producer handler) - SC - K8s CRDs (topic and partition) The feature requires for Dedup SmartModule filter to be provided. It is coming, but the one from the examples can be used to try it out: ```bash cd smartmodule/examples/filter_hashset smdk build smdk load ``` then create a file `topic.yaml`: ```yaml version: 0.1.0 meta: name: dedup-topic deduplication: bounds: count: 10 filter: transform: uses: infinyon/fluvio-smartmodule-filter-hashset@0.1.0 ``` and ```bash fluvio topic create -c topic.yaml ``` with such configuration the window will be last 10 records. Closes infinyon/roadmap#114
Canceled. |
bors r+ |
Upd.: Split into two parts: [first](#3392) Added topic-level deduplication mechanism. Key impacted components: - Topic Config - Fluvio CLI (topic creation, topic list, topic describe) - SPU (replica and producer handler) - SC - K8s CRDs (topic and partition) The feature requires for Dedup SmartModule filter to be provided. It is coming, but the one from the examples can be used to try it out: ```bash cd smartmodule/examples/filter_hashset smdk build smdk load ``` then create a file `topic.yaml`: ```yaml version: 0.1.0 meta: name: dedup-topic deduplication: bounds: count: 10 filter: transform: uses: infinyon/fluvio-smartmodule-filter-hashset@0.1.0 ``` and ```bash fluvio topic create -c topic.yaml ``` with such configuration the window will be last 10 records. Closes infinyon/roadmap#114
Pull request successfully merged into master. Build succeeded! The publicly hosted instance of bors-ng is deprecated and will go away soon. If you want to self-host your own instance, instructions are here. If you want to switch to GitHub's built-in merge queue, visit their help page. |
Upd.: Split into two parts: first
Added topic-level deduplication mechanism.
Key impacted components:
The feature requires for Dedup SmartModule filter to be provided. It is coming, but the one from the examples can be used to try it out:
cd smartmodule/examples/filter_hashset smdk build smdk load
then create a file
topic.yaml
:and
with such configuration the window will be last 10 records.
Closes infinyon/roadmap#114