Skip to content

2.1 The crawler polling service

jackpay edited this page Dec 10, 2019 · 2 revisions

The CrawlerPollingManager is an instance of IndependentCrawlerPollingManager which monitors crawler jobs running on a JQM instance and restarts those that have completed.

This program also allows for the killing of all running jobs on the queue.

The main classes for the CrawlerPollingManager can be found in the project m52-norconex-crawler.

This is a service that once begun will periodically poll (based on user param or a default of 2hrs) the JQM queue and restart those which have completed.

The main parameters for the CrawlerPollingManager are as follows:

  1. casm.jqm.polling.props = The crawler properties file
  2. casm.jqm.polling.cache = The path to store the pollcache
  3. (Optional) casm.jqm.polling.kill = set only if you intend to kill all crawler jobs in JQM

Clone this wiki locally