General Notes

Setting up the environment

• It can be useful to setup an individual screen session for each component of managing the crawling/scraping architecture.

e.g. screen -S <INDICATIVE_SCREEN_NAME>

It is suggested you create five screens for the processes:

1. Seed submission - submitting multiple seeds to JQM
2. Crawl polling - incremental crawl monitoring and submission).
3. JEF - starting and running the JEF crawler monitoring service.
4. Configuring JEF - updating its configuration file with new crawl indexes.
5. JQM nodes - starting, stopping creating and otherwise managing JQM nodes (most other operations can be performed in the JQM admin U.I.).