-
-
Notifications
You must be signed in to change notification settings - Fork 508
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix multi-threading issue #173
Comments
I’d like to tackle this—thinking |
I'm working on this currently but my solution deals with using Golang. If you're able to write something in Python that has similar performance then that would be fine by me. |
@KingAkeem - Golang would be easier for this for sure, but I think there may be a way to do it in Python. How were you planning on tying this project to a Golang based multithreaded URL analyzer? |
I'm going to try just using Go without any attempts at optimization since one of our bottlenecks is I/O, I think simply using Go to perform the request and print the status while provide a significant speed increase. After that, I'll run each request in a separate goroutine and take advantage of Go's built-in concurrency. |
I have a somewhat working prototype currently. You can check our this PR: #174 and this repo contains the Golang code that's being used as a plugin https://github.com/KingAkeem/display_status |
Sounds good, I'll un-assign myself from this issue. I'll take a look at the PR too 👍 |
Sounds good to me 👍 |
Describe the bug
We're using multi threading to display links and Python isn't great at handling threads due to the GIL so this causes stalling when trying to display several hundred (at minimum) links. I'm thinking that the solution in this case is to process the links asynchronously instead, it may not be as fast but it'll be stable.
To Reproduce
python3 torBot.py -u https://hiddenwikitor.org
Expected behavior
LInks are retrieved until user cancels operation or all links have been found.
The text was updated successfully, but these errors were encountered: