Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add workers for pagination #3

Open
Nhoya opened this issue Dec 30, 2017 · 0 comments
Open

Add workers for pagination #3

Nhoya opened this issue Dec 30, 2017 · 0 comments

Comments

@Nhoya
Copy link
Owner

Nhoya commented Dec 30, 2017

After the implementation of pagination for GitHub and BitBucket API (thanks to the report on #1) i was thinking about using some workers to speedup the scraping process.

My idea is to spawn a worker every 5 pages.

Any idea, suggestion and contribution is well accepted.

The current solution available at paginator spawns a goroutine for each page

@Nhoya Nhoya added this to Done in gOSINT Core Mar 6, 2018
@Nhoya Nhoya moved this from Done to TODO in gOSINT Core Mar 6, 2018
@Nhoya Nhoya removed this from TODO in gOSINT Core Mar 6, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant