Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[MM] image_deduplicator #63

Closed
HYLcool opened this issue Nov 2, 2023 · 0 comments · Fixed by #72
Closed

[MM] image_deduplicator #63

HYLcool opened this issue Nov 2, 2023 · 0 comments · Fixed by #72
Assignees
@chenhesen
Labels
dj:multimodal issues/PRs about multimodal data processing enhancement New feature or request
Milestone
Basic Multimodal ...

Comments

@HYLcool
Copy link
Collaborator

HYLcool commented Nov 2, 2023

A new Deduplicator image_deduplicator will be supported. It will remove duplicate images in multimodal samples. Maybe based on imagededup library

TBD: when a sample contains multiple images, remove duplicate images only or remove this whole sample?
@HYLcool HYLcool added enhancement New feature or request dj:multimodal issues/PRs about multimodal data processing labels Nov 2, 2023
@HYLcool HYLcool added this to the Multimodal Support milestone Nov 2, 2023
@chenhesen chenhesen linked a pull request Nov 14, 2023 that will close this issue
add image_deduplicator #72
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@chenhesen chenhesen

Labels
dj:multimodal issues/PRs about multimodal data processing enhancement New feature or request
Projects
None yet
Milestone
Basic Multimodal Support
Development

Successfully merging a pull request may close this issue.

add image_deduplicator modelscope/data-juicer
2 participants
@HYLcool @chenhesen