Find resources in <meta> redirects #216
Conversation
| // If the RegExp doesn't contain a global flag, we expect to find the | ||
| // resources in the capture groups, so we strip away the match as a | ||
| // whole (ie. the first entry in the matches array). | ||
| if (regex.toString().split("/").slice(-1)[0].indexOf("g") === -1) { |
There was a problem hiding this comment.
Whoah, this is meta. I wonder if there's a simpler way to do this.
There was a problem hiding this comment.
To avoid the entire split and slice part, there's https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/RegExp/flags, but that's not supported in node.
But I just realized that to avoid this entire logic (which I admit might not be the most solid) I could actually just wrap the entire regexp in a non-capturing group! I'll fix that later today
We only discover those resources, we don't treat them as an actual redirect (eg. emit a "fetchredirect" event). Fixes simplecrawler#190
fredrikekelund
force-pushed
the
respectHtmlRedirects
branch
from
7eed94f
to
fc23bdb
Compare
|
I went for a slightly more generalized solution: allowing functions in the discoverRegex array. What do you think about that, @cgiffard? |
|
Looks good. In future I think a good philosophy might be to make a simpler but more general regex, which might hit false positives, but which should be validated and filtered out later. :) |
Find resources in <meta> redirects
We only discover those resources, we don't treat them as an actual
redirect (eg. emit a "fetchredirect" event). Fixes #190