Cannot create a sitemap for www.forthea.com, my app just freezes

[Chamomile11]

Do you want to request a feature or report a bug?

BUG

What is the current behavior?

I am building a small console Node app. I successfully managed to create sitemaps for several thousands websites using this package. However, when I try to use it for www.forthea.com my app just hangs forever (even when I use it inside a promise with a timeout timer to force it to stop and even despite its own built-in timeout value of 30000 ms). My app just freezes but does not crash and there is no any errors.

If the current behavior is a bug, please provide the steps to reproduce.

const sitemapGenerator = require('sitemap-generator');
const generator = sitemapGenerator('http://www.forthea.com', {
stripQuerystring: false
});

generator.start(); // Loops forever, no errors, no sitemap, even when used with a timer.

What is the expected behavior?

The package must be capable of building a sitemap for www.forthea.com (or at least generate some error).

[lgraubner]

Will have a look at it tomorrow.

[Chamomile11]

Will have a look at it tomorrow.

Alright, thank you Lars.

[Chamomile11]

Seems like the package blocks the main event loop altogether when parsing www.forthea.com so that even timers set via setTimeout stop working. I fixed my problem by running my sitemap generation code inside a separate child process and killing it after a predetermined amount of time, as I originally needed. This way it works alright and I can kill the child process even if it hangs.

[lgraubner]

Interesting. Would still be good to know why the main event loop is blocked for this website especially.

[Rob-Rychs]

that is weird, I've never experienced this infinite loop error. Are you setting the crawler to respect robots.txt? @Chamomile11