CLI Support

Sometimes we need to start our scraping work automatically or periodically.

You can setup a bat or sh file containing the following command and setup a task scheduler or even start your scraping work programmatically.

Useage:

chrome "chrome-extension://[ExtensionID]/devtools/devtools_scraper_panel.html?run=[sitemapid]&[anti=true|false]&[distinct=true|false]"

Example:

chrome.exe "chrome-extension://fedhdngpddfjodmnfopokanlgnapnmeb/devtools/devtools_scraper_panel.html?run=test3&anti=false"

Parameters:

ExtensionID:

This is the ID of your Web Scraper Plus. Right click the icon and select option, then you can find it in your address bar.

The address will be like chrome-extension://fedhdngpddfjodmnfopokanlgnapnmeb/options_page/options.html and the ExtensionID is fedhdngpddfjodmnfopokanlgnapnmeb

sitemapid:

This is the sitemap id of which the task you want to start.

anti:

Enable the Anti Lazy-Loading or not. Default value is true.

distinct:

Enable the Distinct or not. Default value is true.

Use cases:

This feature is very useful if you want to controll the scraping by program.

For example, you can create some sitemaps by program and store it into MySQL just like you store anything else.

Indicate the parameters and construct a URL.

Call the chrome application and open the URL you just created.