New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.
Already on GitHub? Sign in to your account
accidental work of grab.spider #298
Comments
Probably it's related to this code part: def create_grab_instance(self, **kwargs):
g = super(ExampleSpider, self).create_grab_instance(**kwargs)
g.setup(proxy='127.0.0.1:8090', proxy_type='socks5', timeout=60, connect_timeout=15)
return g I have copied this snippet from the documentation, my goal to set proxy and other settings for grab. Without this part of code all work fine. It seems to me that need use alternative here. |
@EnzoRondo
Happens what? You've provided two log outputs. I do not see big difference between them. With these logs, I can't find what have been done incorrectly (or have not been done correctly) by your spider. |
Closing this issue untill @EnzoRondo provides additional details of what did he mean. |
Look at that:
grab trying to follow 5 same urls instead of right one |
does it work w/o socks5 proxy? as far as i remember pycurl has issue with socks5 |
@istinspring it works |
@EnzoRondo Spider does not work correctly with socks5 in multicurl mode.
|
I have translated this post and got that it's possible to use socks5 using threaded transport, but now you are saying another things, where is the truth? Tested: bot = SomeSpider(network_service='threaded`, grab_transport='urllib3'), works perfect, thanks Will that bug fixed in future? I am using grab.spider in different project and that's one place (create_grab_instance function) where I have problems with it |
Works with socks5:
Does not work with socks5:
|
works perfect
works, but we can see bug from the first post |
Code from first post does NOT use |
Yep, but I have tried: bot = ExampleSpider(thread_number=2, network_service='threaded', grab_transport='pycurl') and bug still here |
So what do you want from me? I do not know what you've tried and have not tried. |
to fix bug, but I can't reproduce it now on last dev build 馃槙 , probably some of your commits successfully fixed this issue, thanks a lot friend! now spider works more stable 馃憤 |
I am very happy to have no problems here, thanks a lot again, I appreciate it 馃槑 |
Hey there (@lorien), thanks a lot for great library 馃槂
I am learning your library and now see unexpected behavior during work, here is my code sample which is based on example in documentation:
first run:
馃槙
then I am running code again ~20 attempts and have same shit, but 21 time gives success and I see what I want to see:
why it happens?
The text was updated successfully, but these errors were encountered: