New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Indexer: create cache of published URLs #111

Closed

creekorful opened this issue Jan 3, 2021 · 1 comment · Fixed by #115

Labels

Milestone

Owner

creekorful commented Jan 3, 2021

Prevent duplicates between URLs publishing by creating a short lived cache to keep track of published URLs and ensure no duplicates are being pushed.

Estimate if useful by counting duplicates in crawlingQueue.
Cache could be redis, and key TTL should be < than refresh delay (if any)

creekorful added the enhancement label

creekorful added this to the 0.10.0 milestone

Owner Author

creekorful commented Jan 3, 2021

there's a lot of duplicates

creekorful modified the milestones: 0.10.0, 0.9.0

creekorful changed the title ~~Indexer: create cache of published URLs?~~ Indexer: create cache of published URLs

creekorful linked a pull request

that will close this issue

Prevent duplicates urls in crawlingQueue #115

Merged

creekorful closed this as completed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment