Can't disable SSL verification in Scrapy

[bigtang5]

I think it's a bug, not a feature.

The proxy provider give the demo code:

r = requests.get("https://www.baidu.com/", headers=headers, proxies=proxy, verify=False, allow_redirects=False)

It works fine, It will fail when remove "verify=False".

Now I have to use the proxy in my scrapy project, There is no way to add "verify=False" in Scrapy unless I write my own HTTPDownloadHandler.

HEW, what a huge work......

[wRAR]

What happens in your spider? As described in https://docs.scrapy.org/en/latest/topics/settings.html#downloader-clientcontextfactory Scrapy doesn't verify server certificates by default, unlike requests.

[wRAR]

Hi @bigtang5 , do you have anything to add here?

[zhuifeng7000]

@bigtang5

how do you solve this problem ?
can you share it?

[wRAR]

@zhuifeng7000 can you please answer the questions in the first comment?