CPU performance optimalization #41
Comments
Did you use some software? |
|
@ad-m Thanks for the incredibly detailed bug report, I sincerely appreciate that. I thought using bencoder.pyx (written in Cython) instead of the pure Python solution would solve the performance issues, but as you have also said, my CPU usage (on 5$ Linode instance) stayed same. I'll have a look at writing it in Cython myself to solve the issue, or as a last resort in C itself (whether I write it myself or adapt from somewhere else). I would be also very much grateful if you can analyse the SQL queries in magneticow, if you can. Queries are extremely slow and I honestly don't know why. |
|
@ngosang , yes, i googled "python profiling", visited http://stackoverflow.com/questions/582336/how-can-you-profile-a-script, collect data @boramalper , I don't know a lot about C++ binding (I don't write any yet), but there might be some costs of context switching. There will be 2605497 context switch in bencoder.bencode, so there might be some bulking required. Again I emphasize, I do not know C++ binding. |
Before doing that (I think you aren’t going to improve speed enough), ask yourself if it's necessary to bencode all time practically the same data. and send the same request data to all nodes (bootstrap and neighbours). Since your id is the same and other fields are random should work...
You aren't doing anything wrong. SQLite is not designed to store many documents, performance is degraded after few thousand rows. If you want to use a relational DB use MySQL or PostgreSQL instead. SQL syntax is almost the same and your queries will be x10000 times faster. Since it's a search engine, using Elasticsearch (no relational search engine) will be awesome but you have to change too much code... |
|
@boramalper , SQL queries in magneticow is described in #8 |
@ngosang That's a great idea indeed! I added some comments on your "hack". :) |
Hello,
When application started there is 100% CPU usage, so the indexing speed is CPU-limited mostly, so if we want more effective software we need start optimising of CPU usage.
I know the software author knows that limitation, but I would like starts public discussion about possible optimisation.
Here is some logs of cProfile:
Based on them I draw some graph:
60% of script was spend in bencoder.bencode. It was executed 2605497 times.
Greetings,
The text was updated successfully, but these errors were encountered: