TypeError: can't pickle thread.lock objects

[benjaminelkrieff]

Hi.

I don't know how much it happens or how much it already happenned but one of my crawl fell on the error below.
I ran a thousand requests and it happened to only one of them. But still my crawl was put down. Here is the stack trace:

2021-02-08 15:03:30 [scrapy.core.scraper] ERROR: Error downloading <GET https://pccomponentes-prod.mirakl.net/login>
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
result = g.send(result)
File "/usr/local/lib/python2.7/site-packages/scrapy/core/downloader/middleware.py", line 66, in process_exception
spider=spider)
File "/usr/src/app/crawler/crawling/log_retry_middleware.py", line 89, in process_exception
self._log_retry(request, exception, spider)
File "/usr/src/app/crawler/crawling/log_retry_middleware.py", line 102, in _log_retry
self.logger.error('Scraper Retry', extra=extras)
File "/usr/src/app/crawler/scutils/log_factory.py", line 244, in error
extras = self.add_extras(extra, "ERROR")
File "/usr/src/app/crawler/scutils/log_factory.py", line 319, in add_extras
my_copy = copy.deepcopy(dict)
File "/usr/local/lib/python2.7/copy.py", line 163, in deepcopy
y = copier(x, memo)
File "/usr/local/lib/python2.7/copy.py", line 257, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/local/lib/python2.7/copy.py", line 190, in deepcopy
y = _reconstruct(x, rv, 1, memo)
File "/usr/local/lib/python2.7/copy.py", line 334, in _reconstruct
state = deepcopy(state, memo)
File "/usr/local/lib/python2.7/copy.py", line 163, in deepcopy
y = copier(x, memo)
File "/usr/local/lib/python2.7/copy.py", line 257, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/local/lib/python2.7/copy.py", line 163, in deepcopy
y = copier(x, memo)
File "/usr/local/lib/python2.7/copy.py", line 264, in _deepcopy_method
return type(x)(x.im_func, deepcopy(x.im_self, memo), x.im_class)
File "/usr/local/lib/python2.7/copy.py", line 190, in deepcopy
y = _reconstruct(x, rv, 1, memo)
File "/usr/local/lib/python2.7/copy.py", line 334, in _reconstruct
state = deepcopy(state, memo)
File "/usr/local/lib/python2.7/copy.py", line 163, in deepcopy
y = copier(x, memo)
File "/usr/local/lib/python2.7/copy.py", line 257, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/local/lib/python2.7/copy.py", line 190, in deepcopy
y = _reconstruct(x, rv, 1, memo)
File "/usr/local/lib/python2.7/copy.py", line 334, in _reconstruct
state = deepcopy(state, memo)
File "/usr/local/lib/python2.7/copy.py", line 163, in deepcopy
y = copier(x, memo)
File "/usr/local/lib/python2.7/copy.py", line 257, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/local/lib/python2.7/copy.py", line 190, in deepcopy
y = _reconstruct(x, rv, 1, memo)
File "/usr/local/lib/python2.7/copy.py", line 334, in _reconstruct
state = deepcopy(state, memo)
File "/usr/local/lib/python2.7/copy.py", line 163, in deepcopy
y = copier(x, memo)
File "/usr/local/lib/python2.7/copy.py", line 257, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/local/lib/python2.7/copy.py", line 190, in deepcopy
y = _reconstruct(x, rv, 1, memo)
File "/usr/local/lib/python2.7/copy.py", line 334, in _reconstruct
state = deepcopy(state, memo)
File "/usr/local/lib/python2.7/copy.py", line 163, in deepcopy
y = copier(x, memo)
File "/usr/local/lib/python2.7/copy.py", line 257, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/local/lib/python2.7/copy.py", line 163, in deepcopy
y = copier(x, memo)
File "/usr/local/lib/python2.7/copy.py", line 230, in _deepcopy_list
y.append(deepcopy(a, memo))
File "/usr/local/lib/python2.7/copy.py", line 190, in deepcopy
y = _reconstruct(x, rv, 1, memo)
File "/usr/local/lib/python2.7/copy.py", line 334, in _reconstruct
state = deepcopy(state, memo)
File "/usr/local/lib/python2.7/copy.py", line 163, in deepcopy
y = copier(x, memo)
File "/usr/local/lib/python2.7/copy.py", line 257, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/local/lib/python2.7/copy.py", line 190, in deepcopy
y = _reconstruct(x, rv, 1, memo)
File "/usr/local/lib/python2.7/copy.py", line 334, in _reconstruct
state = deepcopy(state, memo)
File "/usr/local/lib/python2.7/copy.py", line 163, in deepcopy
y = copier(x, memo)
File "/usr/local/lib/python2.7/copy.py", line 257, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/local/lib/python2.7/copy.py", line 182, in deepcopy
rv = reductor(2)
TypeError: can't pickle thread.lock objects

Some help would be highly appreciated

[madisonb]

Your error seems to be happening in this method - perhaps you are adding a non-serializable field to your request? you can disable that logging middleware or simply change what it is logging to help you understand what is happening.

[benjaminelkrieff]

Thanks for the help, I will give it a shot