New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stop using multiprocessing.dummy.Pool #126

Merged
merged 3 commits into from Oct 18, 2014

Conversation

Projects
None yet
1 participant
@untitaker
Member

untitaker commented Oct 15, 2014

This implements a custom job queue with workers based on threads.
Collection discovery is now done in a separate thread.

See #124

  • Fix tests
  • Document max_workers
  • Deprecate processes
  • Don't sync collections more than once
    • Add tests
@untitaker

This comment has been minimized.

Member

untitaker commented Oct 15, 2014

@mistotebe I think this would fix your issue. It reuses the storage classes from the discovery stage for actual syncing.

@untitaker untitaker force-pushed the issue124 branch 6 times, most recently from 7ce54d2 to 0c2bd13 Oct 16, 2014

untitaker added some commits Oct 15, 2014

Stop using multiprocessing.dummy.Pool
- Custom job queue with workers based on threads.
- Collection discovery is now done in a separate thread. Due to the
  gained flexibility, we could do the sync actions in separate threads
  too?
- The processes parameter has been removed, the related new option is
  only available on the CLI.

@untitaker untitaker force-pushed the issue124 branch from b08ffc0 to aba0a40 Oct 16, 2014

untitaker added a commit that referenced this pull request Oct 18, 2014

Merge pull request #126 from untitaker/issue124
Stop using multiprocessing.dummy.Pool

@untitaker untitaker merged commit 6ac71e0 into master Oct 18, 2014

1 check passed

continuous-integration/travis-ci The Travis CI build passed
Details

@untitaker untitaker deleted the issue124 branch Nov 2, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment