High CPU usage

[ocervell]

cariddi can give crazy CPU usage and decent memory usage when run on a huge list of URLs (~150 in this case).
We should probably profile it and find areas of improvements for performance. Such a library should be I/O heavy but not too CPU or memory heavy normally.

cat urls.txt | cariddi -info -s -err -e -ext 1 -json -c 5

[edoardottt]

Hi @ocervell ! Thanks for the issue.
I know it's quite resource intensive since the first commits :(
I guess the CPU and memory high load comes from pattern matching.
I'd love to profile it and understand if we can improve it, however I've never used a profiler.

Let me know if you can help in some way, I'd love to improve cariddi.

Thanks again

[jaikishantulswani]

@edoardottt High cpu usage up to 95% even without -s or -ext flag

[edoardottt]

@ocervell @jaikishantulswani Let me know what you think:

In this commit d3834a4 I've optimized pattern matching for secrets, errors and info. Now the regular expressions compilation is performed just once at the starting point and no where else. Before for every html response the regex were recalculated (regexp.MustCompile(regex) every time).

Just by doing this when those flags are active the improvements should be noticeable.

Then, I've also updated colly to the latest version v2.2.0. You can find the complete update in devel branch.

git clone --branch devel https://github.com/edoardottt/cariddi

[jaikishantulswani]

@edoardottt thanks for the update but CPU usage still goes higher, it would be good if it have to control concurrency rate limit and workers flag to set them accordingly. also have flag to show links associated with the domain like frequent domain name or root domain name and skip out of scope links

[edoardottt]

@jaikishantulswani there are the flags:

• -c for the concurrency
• -d for the delay
• -i to ignore certain paths

[edoardottt]

See https://github.com/edoardottt/cariddi/releases/tag/v1.4.0