A Crawler to find a text content in the web page
- Node and npm to be installed in the machine
- User should have admin or elivated access to install the module
- Install the related packages.
npm install- Install the module globally
npm install -gnpm uninstall my_crawlermy_crawler website_url search_termExamples :
my_crawler https://www.apple.com Applemy_crawler https://www.apple.com "Apple TV"| Name | Option | Description | Default | Type | Example |
|---|---|---|---|---|---|
| Depth | --depth |
The depth to which the links from the original page will be crawled. | Default is 1 | number |
my_crawler website_url search_term --depth=2 |
| Ignore Case | --ignoreCase |
Ignores the character case of the search term. | Default is false | boolean |
my_crawler website_url search_term --ignoreCase or my_crawler website_url search_term --ignoreCase="true" |
| Show Failed URL | --showFailedURL |
Shows the list of links that were unable to reach. | Default is false | boolean |
my_crawler website_url search_term --showFailedURL or my_crawler website_url search_term --showFailedURL="true" |
| Show Result | --showResult |
Displays the list of string that matched the search term. | Default is true | boolean |
my_crawler website_url search_term --showResult="false" |
- If the crawler encounters any css link with import in the body, it fails to parse.
- A loader [user is not provided with any feedback] is not provided when the crawler is parsing or crawling.