fix samair parser#34
Conversation
pgaref
left a comment
pgaref
left a comment
There was a problem hiding this comment.
test_SemairProxyParser is failing since the mocked html is tailored to the previous provider format!
Need to fix that before merge!
| self.logger.debug("Using proxy: {0}".format(str(self.current_proxy))) | ||
| request = requests.request(method, url, proxies={"http": self.current_proxy}, | ||
| headers=headers, data=data, params=params, timeout=req_timeout) | ||
| headers=headers, data=data, params=params, timeout=req_timeout, allow_redirects=False) |
There was a problem hiding this comment.
What is the reason behind disabling redirects?
There was a problem hiding this comment.
The reason is to avoid requests.exceptions.TooManyRedirects raised by gathered proxy that redirect traffic to another website
There was a problem hiding this comment.
Since proxy providers could use redirects in some cases, I would recommend using the maxRedirects property instead - a default value could be provided as method argument (i.e 3)
The only downside is that in this case we should also catch the TooManyRedirects Exception.
Let me know what you think
There was a problem hiding this comment.
Ye, sounds good, maybe being able to pass allow_redirects from generate_proxied_request() would be cool too!
There was a problem hiding this comment.
Sure, do want to work on what we discussed above (maxRedicts, method arguments, and test_SemairProxyParser) or should I have the ticket reassigned?
There was a problem hiding this comment.
I could do the method argument but not test_SemaireProxyParser since I don't understand the problem, sry my first contribute on github
There was a problem hiding this comment.
In the tests/mocks.py file samair_mock content should be identical to what you see in the real web page. So when you run the test_providers all the tests should pass.
No worries at all, push your changes once you are ready and I will pick it up from there.
Cheers,
Panagiotis
| # ports[key] = value | ||
|
|
||
| table = soup.find("table", attrs={"id": "proxylist"}) | ||
| table = soup.find("div", attrs={"id": "proxylist"}) |
2 participants
No description provided.