Issues with subfolder crawling

[vladejs]

I'm having issues with subfolders. It seems like if I have this:

www.site.com/
  css/index.css
  blog/
     indexBlog.html
     css/index.css

When crawling site.com it downloads first css/index.css.
When crawling blog it finds a link with href to css/index.css insideindexBlog.html.

This last path isn't relative, and the crawler instead of downloading it into blog/css/index.css it checks for the upper /css/index.css, leading to the asset not being downloaded.

So it ends up with this:

www.site.com/
  css/index.css
  blog/
     indexBlog.html // the reference inside is wrong
     [missing index.css file]

[s0ph1e]

Hi @vladejs
Could you please provide a website url where I can reproduce this issue? (I'm not asking for real website, some test website with same problem is enough)
Or write test which demonstrates incorrect behavior?

[vladejs]

hi @s0ph1e , I can't share the URL here.
I'll be happy to send it to you via email so you can test

Edit: Sent you an email to the address provided on your profile.

Thank you in advance

[s0ph1e]

Yes, I received it. Will investigate the problem

[s0ph1e]

Hi again @vladejs
I found an issue, url is incorrectly calculated for relative resources on some redirected page (example is in #242)

I'll fix it today/tomorrow

[vladejs]

Thank you @s0ph1e , I saw the pull request.
Please, feel free to close the issue once the library gets updated on npm with the fix.

[s0ph1e]

Thanks @vladejs
Fix is released in version 3.3.1

[vladejs]

Hello @s0ph1e .
I tested out the patch and it partially works, the subpage now correctly renders the CSS, but I'm seeing now extremely long scraping times. I tell to scrape 2h ago and it hasn't finished yet.

Also, I got a bunch of unrelated html files, like news-20070730, news-20070731 .....

I tested with the same sample project I sent you via email.

I'm seeing a bunch of 404 errors on console regarding images. Maybe it got stuck downloading them, I'm not sure.

Any feedback would be greatly appreciated.
Thank you

[s0ph1e]

Hi @vladejs
You can use logs to determine where it got stucked (see log and debug section)

export DEBUG=website-scraper*; node app.js

Probably something you don't want to load is loading?
If you are using configuration mentioned in #238 - I'd suggest to:

• decrease maxRecursiveDepth (20 may be too much)
• and/or filter urls, see urlFilter option (recursive downloading with depth 20 and without urlFilter may try to download whole internet)

Hope it helps