This still needs testing and review. It worked on the crawl I tested it with, but quite possibly has broken or changed something with cookie handling.
performance improvement to cookie handling when a single spider is cr…
…awling a large number of domains
This patch was developed when we noticed a slow-down running a broad crawl. I think when we got into the thousands of domains it got slow, and we wanted to crawl millions. If you go a few pages deep on many domains with cookies enabled you should notice a slow down (unless something has been improved in the meantime)