You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Bug] Hybrid crawler has memory usage explosion for sites with "infinite loops" of links
#1710
For certain websites with links which contain randomly generated strings, Katana's memory usage explodes very quickly. An example of such a site is here:
I expect Katana to be able to crawl the site without unbounded memory usage growth, and to complete once the crawl duration limit is reached.
What actually happens
The memory usage grows quickly seemingly without any limit. After about 20secs of crawling this site I could see from the activity monitor that Katana was taking up 22GB of memory:
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
For certain websites with links which contain randomly generated strings, Katana's memory usage explodes very quickly. An example of such a site is here:
https://github.com/evanrolfe/katana_bug/tree/main/bug2
What I expect to happen
I expect Katana to be able to crawl the site without unbounded memory usage growth, and to complete once the crawl duration limit is reached.
What actually happens
The memory usage grows quickly seemingly without any limit. After about 20secs of crawling this site I could see from the activity monitor that Katana was taking up 22GB of memory:
Steps to reproduce:
Start the target website from here with:
Then run Katana against it:
All reactions