Skip to content

Improved threaded page getting #483

Closed
wants to merge 23 commits into from
Commits on Apr 1, 2013
  1. @ptone
  2. @ptone

    debug log for HTTPError

    ptone committed Mar 14, 2012
  3. @ptone
  4. @ptone
  5. @ptone

    remove unused thread lock obj

    ptone committed Apr 14, 2012
  6. @ptone

    just use a fixed 10 worker threads

    ptone committed Apr 14, 2012
  7. @ptone

    update docstring

    ptone committed Apr 15, 2012
  8. @ptone

    Move the thread queue management and variables from method to instance

    ptone committed Apr 15, 2012
    Since the pool of workers gets reused, this seems the cleanest way
    to ensure that excess threads aren't created as they no never return
    
    because the queue is only ever used for one req at a time, it seems safe
    to reuse it on the PackageFinder instance as _get_pages is called through:
    
    prepare_files - find_requirement - _get_pages
  9. @ptone
  10. @ptone

    move ALL the thread getting state onto self

    ptone committed Apr 15, 2012
    no sharing of seen and req variables
  11. @ptone

    stub of thread helper class

    ptone committed Jun 20, 2012
  12. @ptone

    better threading support for PackageFinder

    ptone committed Jun 21, 2012
    adds a helper class that manages a threads and queue for page/link scraping
  13. @ptone

    removed duplicate class stub

    ptone committed Jun 21, 2012
    another glitch of the cherry-pick merge
  14. @ptone

    verbose tests

    ptone committed Apr 1, 2013
Commits on Apr 3, 2013
  1. @ptone
  2. @ptone
  3. @ptone
  4. @ptone

    reuse the page_getter

    ptone committed Apr 3, 2013
    we were creating 10 new threads per request
Commits on Apr 5, 2013
  1. @ptone
  2. @ptone
  3. @ptone
Commits on Apr 8, 2013
  1. @ptone

    more testing for PageGetter

    ptone committed Apr 8, 2013
  2. @ptone

    remove unused imports

    ptone committed Apr 8, 2013
Something went wrong with that request. Please try again.