Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use pool to download items with filter #13

Open
manycoding opened this issue Dec 25, 2018 · 1 comment
Open

Use pool to download items with filter #13

manycoding opened this issue Dec 25, 2018 · 1 comment
Labels
good first issue Good for newcomers Type: Bug Something isn't working Type: Performance

Comments

@manycoding
Copy link
Contributor

The current logic of dividing on batches by start_index and count doesn't account for filter.
When using filter, returned items _key don't correspond with actual index so the data repeats.
@manycoding manycoding self-assigned this Dec 25, 2018
@manycoding manycoding removed their assignment Jan 16, 2019
@manycoding manycoding changed the title Items contain duplicates if downloading with pool Items contain duplicates if downloading with pool and filters Mar 18, 2019
@manycoding manycoding transferred this issue from another repository Mar 18, 2019
@manycoding manycoding added Type: Feature New feature or request Type: Bug Something isn't working labels Mar 18, 2019
@manycoding manycoding added Type: Performance and removed Type: Feature New feature or request labels Apr 1, 2019
@manycoding manycoding mentioned this issue Jul 11, 2019
Closed
@manycoding
Copy link
Contributor Author

The solution is to use nodata scrapinghub/python-scrapinghub#129

@manycoding manycoding changed the title Items contain duplicates if downloading with pool and filters Use pool to download items with filter Oct 22, 2019
@manycoding manycoding added the good first issue Good for newcomers label Oct 22, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
good first issue Good for newcomers Type: Bug Something isn't working Type: Performance
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@manycoding