Feature Request: retry request when error

[xg-wang]

Is there a way to do request retry inside colly? Right now I can only resend the request in onError

[asciimoo]

@xg-wang Currently this is not supported by default, but it's definitely a useful feature. It could be achieved by adding a Retry() member function to the Request object. What do you think?

[xg-wang]

Or we can do it on [http_backend](https://github.com/gocolly/colly/blob/master/http_backend.go#L156) level by adding another LimitRule.Retry.

[asciimoo]

@xg-wang yes, that is another good approach, however it gives less flexibility to manage retries imo. If it is a part of the Request object, then we are able to decide that on which returned status codes we want to perform further requests.
What are the benefits of extending LimitRule?

[xg-wang]

@asciimoo The Response has all the information that Request have. By extending LimitRule is more consistent as it already has Delay and Parallelism, it is quite natural to specify the retry attempt number here imo.

[asciimoo]

By extending LimitRule is more consistent as it already has Delay and Parallelism, it is quite natural to specify the retry attempt number here imo.

LimitRule doesn't feel like the right place for me to handle request resubmission, since it is responsible for limiting requests and not to perform requests again.
What if we want to increase the delay between retries or just print a debug message on each retry? It doesn't seem to be achievable with the LimitRule approach.

I've implemented the Request.Retry() solution on branch retry: 03c5f59 . It feels more intuitive and flexible for me so if you like this solution I'll merge it.

[xg-wang]

I'm down for it. Looks good.

[wedneyyuri]

Hi @asciimoo do you have any example using your retry solution?

[samirsss]

bump on any example - since it helps give users a clear idea on how you intend users to use it.

[olivierpons]

    c.OnError(func(resp *colly.Response, err error) {
        resp.Request.Retry()
    })

[samirsss]

@olivierpons thanks for the response. I did add retry logic once i due in a bit into the codebase. I wanted to limit the number of retries before i give up.