Concurrent Scrape Errors should include additional conext about the URL and request which failed

[twosdai]

Hey,

Currently when running a concurrent scrape if an error is thrown its simply caught and returned as is. Meaning generally there is significant context loss about the URL actually being requested, headers used etc...

Right now I don't see an easy way to get this information while running requests concurrently.

The offending bit of code is here:
https://github.com/scrapfly/typescript-scrapfly/blob/main/src/client.ts#L194

I would have made a pull request regarding this, however I see that in this repo typed errors exist, and its not clear to me exactly which typed error should be returned in the case of the requesting site returning an HTTP error. If you let me know which error class should be used, I can implement a pull request.

Additionally if I am wrong please feel free to call me out, not trying to create more work, just looking for a way to get a URL in the error response.

[Granitosaurus]

Hey thanks for opening this. Currently each ScrapflyError raised by scrape() (and concurrentScrape()) attach context to .args property which includes a lot of context details as you can see here: https://github.com/scrapfly/typescript-scrapfly/blob/main/src/client.ts#L37 in particular check .args.api_response property which should include the used scrape config details. Could you check an verify whether that's enough for your use case?

I think there's a possibility that some context could be lost if non ScrapflyError is thrown there (like client losing connection or similar). In cases like this we should attach the used scrape config to the raised exception as .scrapeConfig or something. I'll add some sort of fix for this in the upcoming release.

[Granitosaurus]

The v0.6.2 now attaches error with .scrapeConfig property which helps to identify which concurrentRequest has failed. Though for ScrapflyError type errors still see ScrapflyError.args.api_response property instead.