You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I'm having the following issue when scraping yielding ScrapflyScrapyRequest:
ERROR scraper.py:246 Error downloading <GET https://immobilienscout24.de/expose/146870274>
Traceback (most recent call last):
File "/Users/dev/scraper/.venv/lib/python3.9/site-packages/scrapfly/api_response.py", line 105, in __call__
return self.content_loader(content)
File "msgpack/_unpacker.pyx", line 194, in msgpack._cmsgpack.unpackb
File "/Users/dev/scraper/.venv/lib/python3.9/site-packages/scrapfly/api_response.py", line 51, in _date_parser
value[k] = _date_parser(v)
File "/Users/dev/scraper/.venv/lib/python3.9/site-packages/scrapfly/api_response.py", line 53, in _date_parser
value[k] = v
TypeError: 'bytes' object does not support item assignment
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/dev/scraper/.venv/lib/python3.9/site-packages/twisted/internet/defer.py", line 1697, in _inlineCallbacks
result = context.run(gen.send, result)
File "/Users/dev/scraper/.venv/lib/python3.9/site-packages/scrapy/core/downloader/middleware.py", line 75, in process_exception
response = yield deferred_from_coro(method(request=request, exception=exception, spider=spider))
File "/Users/dev/scraper/.venv/lib/python3.9/site-packages/scrapfly/scrapy/middleware.py", line 70, in process_exception
raise exception
File "/Users/dev/scraper/.venv/lib/python3.9/site-packages/twisted/internet/defer.py", line 1693, in _inlineCallbacks
result = context.run(
File "/Users/dev/scraper/.venv/lib/python3.9/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
File "/Users/dev/scraper/.venv/lib/python3.9/site-packages/scrapy/core/downloader/middleware.py", line 49, in process_request
return (yield download_func(request=request, spider=spider))
File "/Users/dev/scraper/.venv/lib/python3.9/site-packages/twisted/internet/defer.py", line 892, in _runCallbacks
current.result = callback( # type: ignore[misc]
File "/Users/dev/scraper/.venv/lib/python3.9/site-packages/scrapfly/scrapy/downloader.py", line 82, in on_body_downloaded
scrapfly_api_response:ScrapeApiResponse = spider.scrapfly_client._handle_response(
File "/Users/dev/scraper/.venv/lib/python3.9/site-packages/scrapfly/client.py", line 295, in _handle_response
api_response = self._handle_api_response(
File "/Users/dev/scraper/.venv/lib/python3.9/site-packages/scrapfly/client.py", line 453, in _handle_api_response
body = self.body_handler(response.content)
File "/Users/dev/scraper/.venv/lib/python3.9/site-packages/scrapfly/api_response.py", line 107, in __call__
raise EncoderError(content=content.decode('utf-8')) from e
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x84 in position 0: invalid start byte
This error is present in 2% of my total requests and it's completely random some URLs may hit this error in a few tries, but in most cases, they don't repeat.
Environment Setup:
python = 3.9.9
MacOS = Apple M2
scrapfly-sdk = {extras = ["all"], version = "^0.8.9"}
The text was updated successfully, but these errors were encountered:
If you can, share a minimum setup to get the same condition as yours (with a poetry lock or requirement.txt with a fixed version). Because in your stack trace, you have /twisted/internet/, I guess there is scrapy involved? (I tested regular SDK and Scrapy integration)
I'm having the following issue when scraping yielding ScrapflyScrapyRequest:
This error is present in 2% of my total requests and it's completely random some URLs may hit this error in a few tries, but in most cases, they don't repeat.
Environment Setup:
The text was updated successfully, but these errors were encountered: