Randomized Delay

[jodizzle]

Is it possible to create randomized delays, i.e., per-request delays selected from some range or based on some random factor? I couldn't think of a good way to do this, other than maybe cycling through several Collectors with different limit sets, which seems sub-optimal.

It seems like having LimitRule.DelayRange or LimitRule.RandomFactor options would be quite helpful.

[asciimoo]

@jodizzle this would be a handy addition. Do you want to work on it?

As a workaround you can add an OnRequest callback with random sleeps:

c.OnRequest(func(_ *colly.Request) {
    time.Sleep(time.Duration(rand.Randint(5)+1) * time.Second)
})

[jodizzle]

@asciimoo Ah, I hadn't thought of the OnRequest approach. Thanks.

And sure, I can try working on it. Do you have any thoughts about what the best implementation would be?

[asciimoo]

I like the idea of extending LimitRule with a RandomDelay time.Duration attribute which is 0 by default. If its value isn't 0, a random duration between 0 and RandomDelay could be added to LimitRule.Delay on every request. This solution doesn't break the current API and it's intuitive for me. What do you think?

[jodizzle]

That sounds pretty reasonable. I'll take a look.

[jlr52]

Not sure if its okay, but I got interested in the code base so I just implemented the enhancement.

@jodizzle let me know if you still want to fix it and I can decline my pull request.

#67

[jodizzle]

@jlr52 Fine by me -- I was being slow. Your approach seems similar to what I was thinking anyway. I'll take a closer look.

[asciimoo]

implemented in #67 - thanks @jlr52