Is it possible to extract links using a custom pattern?

[bruce-genhot]

For some reasons, I have to extract links having javascript as href, like below. is it possible ?
<a href="javascript:__doPostBack('MoreInfoList1$Pager','2')" title=""><img src="/sczw/images/page/2n.gif" align="Baseline" border="0"></a>

[essiembre]

Right now, anything that is not a regular URL (absolute or relative) is ignored by the GenericLinkExtractor. At this moment, to be able to extract the Javascript, you would need to create your own implementation of ILinkExtractor.

I will make this a feature request to add the ability to provide custom regex to extract just about anything you like as URLs.

[essiembre]

I take back my last comment. It turns out you should be able to get your URLs by adding "javascript" as a supported scheme. Like this:

<extractor class="com.norconex.collector.http.url.impl.GenericLinkExtractor">
    <schemes>http,https,ftp,javascript</schemes>
</extractor>

Please give it a try and confirm.

I will still keep this as a feature request since I think supporting custom regex is a nice to have.

[bruce-genhot]

OK, will have a try, thanks.

[MohamedElsakka]

please can you said to me if it worked or not , because i have the same issue with Javascript.
for me i edited in class GenericLinkExtractor
private static final String[] DEFAULT_SCHEMES =
new String[] { "http", "https", "ftp","javascript" }; that's ok or not .

[essiembre]

Maybe you can paste your config and/or sample HTML code that does not get extracted how you would expect?

[MohamedElsakka]

I want to extract links having javascript as href, like below. is it possible ?

<a href="javascript:removeFiltersProc();" class="btn btn-success btn-binder-success btn-sm">Remove Selected</a>

and there is another situation for loading page , i can't catch any links because the page not loaded yet , it possible ?

[essiembre]

The crawler does not interpret the JavaScript, so it is currently not possible to do what you want out-of-the-box. There is already a feature request for interpreting JavaScript in issue #95. We are working on a solution for the next feature release (no set time-frame for it).

[essiembre]

The latest snapshot has a new RegexLinkExtractor that makes it possible to use custom patterns to extract URLs. For JavaScript generated URLs, you should have a look at the PhantomJSDocumentFetcher, tracked in #95.