Performance improvements #70

wants to merge 13 commits into


None yet
3 participants
  1. improve Redis storage engine performance
  2. added expiration to Redis page storage
  3. added stop crawl functionality
  4. improved RAM usage (page queue got filled up too quickly)

efrat-safanov added some commits Nov 15, 2012

I think the major issue is bcos of the class variable @page storing all the pages. If you remove that there is no mem leak

FYI this did help a lot with memory issues. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment