Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Refactor Scheduler and add BloomFilter for duplicate removing #118

Closed
code4craft opened this issue May 2, 2014 · 0 comments
Closed

Refactor Scheduler and add BloomFilter for duplicate removing #118

code4craft opened this issue May 2, 2014 · 0 comments
Assignees
@code4craft
Milestone
WebMagic-0.5.1

Comments

@code4craft
Copy link
Owner

The Scheduler use hashset for duplicate removing. It will take a lot of memory when number of urls is huge. Add BloomFilter for less memory usage.
@code4craft code4craft added this to the WebMagic-0.5.1 milestone May 2, 2014
@code4craft code4craft self-assigned this May 2, 2014
code4craft added a commit that referenced this issue May 2, 2014
@code4craft
add bloom filter for scheduler #118
d1140b9
code4craft added a commit that referenced this issue May 2, 2014
@code4craft
set setDuplicateRemover for chain api #118
56f033c
code4craft added a commit that referenced this issue May 2, 2014
@code4craft
refactor redisscheduler #118
186b905
code4craft added a commit that referenced this issue May 2, 2014
@code4craft
extension point of geturl #118
01aec7e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@code4craft code4craft

Labels
None yet
Projects
None yet
Milestone
WebMagic-0.5.1
Development

No branches or pull requests
1 participant
@code4craft