Drain queue

[grafana-dee]

I have the following scenario:

1. Start crawling a large website
2. Queue 1,000 URLs
3. Need to cancel the crawl ASAP (for whatever reason)

q.Close() does not help as even though it prevents new items from getting added to the queue, no method is offered for draining the queue of the existing items.

I'm not sure how to approach doing it, otherwise I'd offer a patch, but could a q.Drain() be added... or better still a q.Cancel() which first calls q.Close() and then drains the queue before returning and releasing q.Block().

[mna]

q.Cancel sounds like a good idea, yeah. I may not get to it before a few weeks, though.

[mna]

Actually I had a little free time and I implemented an experiment on the branch cancel if you want to take a look. I may have missed a couple things, haven't looked at that code in a while, but if that's it it would be pretty simple. I will test it when I have a bit more time.

[mna]

Did you get a chance to try it out? Was it working as expected?

[grafana-dee]

I got a chance to try it out, but not to debug it (got on a plane and am now in a different city doing different things).

It appears to work (it cancels further crawling), but never actually unblocks.

Stops here:

fetchbot/fetch.go

Line 142 in 96b7aa9

q.wg.Wait()

My queue was fairly long at time of cancellation (several thousand URLs at a default rate of 2 reqs per second), but it hangs indefinitely on waiting for something else to mark the waitgroup as done.

[mna]

Cool, I'll try to look at it today.
On Thu, Jul 23, 2015 at 12:18 David Kitchen notifications@github.com
wrote:

I got a chance to try it out, but not to debug it (got on a plane and am
now in a different city doing different things).

It appears to work (it cancels further crawling), but never actually
unblocks.

Stops here:

fetchbot/fetch.go

Line 142 in 96b7aa9

q.wg.Wait()

My queue was fairly long at time of cancellation (several thousand URLs at
a default rate of 2 reqs per second), but it hangs indefinitely on waiting
for something else to mark the waitgroup as done.

—
Reply to this email directly or view it on GitHub
#10 (comment)
.

[grafana-dee]

Much appreciated, in fact if you have a PayPal email let me know. Beer will be bought.

[mna]

Thanks, I appreciate it, but I'm lucky enough to not be needing donations so feel free to give that amount to a charity of your choice. In case you have no preference, those are important to me:

https://netdonor.net/ea-action/action?ea.client.id=1770&ea.campaign.id=17989

https://greenpeace.donorportal.ca/Donation/DonationDetails.aspx?L=en-CA&G=131&F=985&T=RECUR&cscid=Homepage_Donate_Button_Eng&__utma=218051913.31581654.1437687123.1437687123.1437687123.1&__utmb=218051913.4.10.1437687123&__utmc=218051913&__utmx=-&__utmz=218051913.1437687123.1.1.utmcsr=(direct)%7Cutmccn=(direct)%7Cutmcmd=(none)&__utmv=-&__utmk=147148404

[mna]

I found the bug and added a test for the Cancel method, it should now be working as expected and finishing quickly. Let me know how it goes and I'll merge once you confirm it's good for you too (if you're able to test it, of course, otherwise I'll just merge right away).