urlFilter terminates recursive scraping

[jkanel]

See this discussion thread for more detail.

ISSUE: When the rootUrl does not match the urlFilter criteria, recursive scraping terminates at the root page.

RECOMMENDED SOLUTION: Do not apply the urlFilter to the rootUrl; or make it an option to ignore the rootUrl from scraping.

DESIRED: I'm specifying a rootUrl and would like the scraper to recurse through all hyperlinks. The rootUrl will not be downloaded in this scenario. When the scraper finds a hyperlink ending in .abc it should download the file.

ACTUAL: The rootUrl (see code below) does not meet the urlFilter criteria and the scraper stops with no recursion. The scraper should find a hyperlink to http://trillian.mit.edu/~jc/music/book/SCD/Book45.abc in the rootUrl among other .abc urls, but it does not. Note that when I set the rootUrl equal to an .abc url, e.g. the example above, the file downloads as expected.

const rootUrl = "http://trillian.mit.edu/~jc/music/book/SCD"

scrape({
  urls: rootUrl,
  recursive: true,
  maxRecursiveDepth: 5,
  urlFilter: function(url) {
    let match = url.match(/\.abc$/);
    return (match && match[0]);
  },
  directory: savePath
});

[s0ph1e]

Fix for the issue will be released in next version, most probably 5.0.0

[jkanel]

Great to hear! Thanks, Sophia. Jeff Kanel Mobile | 614.778.5562 ***@***.***

…

On Mon, Oct 18, 2021 at 11:52 AM Sophia Antipenko ***@***.***> wrote: Fix for the issue will be released in next version, most probably 5.0.0 — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#460 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAPDG45VNWAYYWJ544XRPTTUHQ7ETANCNFSM5EJLOQEQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.