WebCrawler is an utility software written in Java.
n_threads=4 delay=100 root_dir=C://root log_level=3 depth=100 It can crawl using robots.txt file
For Sitemap, a Sitemaps folder should be created. For Filter and Search, an existing index.json with a specific structure needed.
Search <site_name> Filter <site_name> <file_type> Help Search Sitemap <absolute_path_to_site_file> Crawl <use_robots_yes_or_no> <size_limit>