Skip to content

Conversation

SuaYoo
Copy link
Member

@SuaYoo SuaYoo commented Oct 9, 2025

Resolves #2862
Depends on #2868

Changes

  • Adds new "Deduplication" section to workflows
  • Allows users to use a collection for deduplication
  • Various refactors for consistency

Screenshots

Page Image/video
Edit Workflow / Deduplication Screenshot 2025-10-09 at 8 29 28 PM
Edit Workflow / Deduplication Screenshot 2025-10-09 at 8 32 41 PM
Edit Workflow / Deduplication Screenshot 2025-10-09 at 8 32 52 PM
Edit Workflow / Deduplication Screenshot 2025-10-09 at 8 32 57 PM
Edit Workflow / Deduplication Screenshot 2025-10-09 at 8 34 10 PM
Edit Workflow / Collections Screenshot 2025-10-09 at 8 34 35 PM
Edit Workflow / Collections Screenshot 2025-10-09 at 8 34 55 PM
Workflow / Settings Screenshot 2025-10-09 at 8 33 57 PM

@SuaYoo SuaYoo requested a review from ikreymer October 10, 2025 03:43
@SuaYoo SuaYoo marked this pull request as ready for review October 10, 2025 03:46
@SuaYoo SuaYoo requested review from emma-sg and tw4l October 10, 2025 03:46
"Browsertrix",
"btrix",
"clsx",
"dedup",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should standardize whether we call it "dedupe" or "dedup"

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ikreymer can we standardize the backend values to be dedupe? I believe that is the more common shorthand.

@SuaYoo SuaYoo marked this pull request as draft October 13, 2025 18:20
@SuaYoo
Copy link
Member Author

SuaYoo commented Oct 13, 2025

Moving to draft until #2868 (review) and #2868 (comment) is addressed.

@SuaYoo SuaYoo force-pushed the feature-dedup--frontend-dedupe-form branch from ce9d5f5 to 3190e85 Compare October 13, 2025 22:28
`)}
${this.renderHelpTextCol(
msg(
"This collection will be used as the deduplication source for all crawls of this workflow.",
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thoughts on this info text? @DaleLore @emma-sg

@SuaYoo SuaYoo marked this pull request as ready for review October 14, 2025 00:08
@SuaYoo SuaYoo requested a review from emma-sg October 14, 2025 00:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants