Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use multiprocessing #1

Open
MatthewChatham opened this issue Jun 14, 2018 · 0 comments
Open

Use multiprocessing #1

MatthewChatham opened this issue Jun 14, 2018 · 0 comments

Comments

@MatthewChatham
Copy link
Owner

The script can operate in parallel on each page of 10 reviews.

In order to do this effectively, I should determine a good number of workers (probably 2-6) and assign each worker some subset of the total pages of reviews. So we need to compute the total number of pages of reviews. I have confirmed that there is a clear mapping between page number and URL, so we can send each worker to the appropriate pages with ease.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant