Skip to content
This repository has been archived by the owner on Jun 10, 2021. It is now read-only.

Easily extensible library which provides straightforward way to create Crawler-like (this one focuses rather on time limited crawling) programs in few variations (Concurrent, Normal, One-time [one page]).

michjak-szymanski/michal.szymanski.crawler

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

25 Commits
 
 
 
 
 
 

Repository files navigation

Example of usage:

    FlexibleOneShotCrawler crawler = new FlexibleOneShotCrawler();
    SentencesCollector c = new SentencesCollector();
    c.setTarget("Learn"); // tells crawler which words he should looking for.
    ImagesCollector i = new ImagesCollector();

    crawler.configure()
            .initUrl("https://www.w3schools.com/") // tells crawler where he should start his work.
            .addCollector(c) 
            .addCollector(i);
			
    crawler.start(); 

About

Easily extensible library which provides straightforward way to create Crawler-like (this one focuses rather on time limited crawling) programs in few variations (Concurrent, Normal, One-time [one page]).

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages