Factory instead of hardcoded class.newInstance()

[rzo1]

I would like to suggest, that adding a the possibility to use a factory to create new web-crawlers would be of great value.

Since a web-crawler could hold a few custom services (e.g. classifiers, database services) a factory would be a very nice thing to make crawler4j usable for example via Spring.

A few years ago an issue was created on googlecode (https://code.google.com/p/crawler4j/issues/detail?id=144), which is a duplicate of mine request - but nothing happend. Is there a reason for not including a factory approach in the code-base?

Thanks in advance.

[yasserg]

It is already implemented but not well documented yet. See WebCrawlerFactory in https://github.com/yasserg/crawler4j/blob/master/src/main/java/edu/uci/ics/crawler4j/crawler/CrawlController.java

[rzo1]

A i see - thanks for your answer - but it is not included in the current release of crawler4j?
So i probably have to use the SNAPSHOT, if i like to use this feature?

[yasserg]

Yes

[s17t]

+1 for factory or any way that allow dependency injection. Any scheduled for new stable that includes this feature ?

[rzo1]

@s17t Just get the source code and build the current SNAPSHOT by yourself. The snap implements a WebCrawlerFactory, which you can use for DI e.g. with Spring.

[epubreader]

I know you have added WebCrawlerFactory, but How to use spring DI? can you give an example? thanks

[s17t]

@eimhee, something like this:

public class CrawlerCrawlerControllerFactory implements CrawlController.WebCrawlerFactory {

   public CrawlerCrawlerControllerFactory(...) {
    ...
   }

    @Override
    public WebCrawler newInstance() {
        return new edu.uci.ics.crawler4j.crawler.WebCrawler(...) // Or new instance of your WebCrawler children class
}

Then in spring:

@Bean
public CrawlController crawler() {

     CrawlerCrawlerControllerFactory factory = new CrawlerCrawlerControllerFactory(...);

    // Configure CrawController accordingly...
    CrawlController controller = new CrawlController(config, pageFetcher, robotstxtServer);
    controller.startNonBlocking(factory, numberOfCrawlers);

   return controller 
}

[rzo1]

As an alternative, you could implement ApplicationContextAware and get a WebCrawler-Bean from your ApplicationContext.

public class CustomCrawlerFactory<T extends WebCrawler> implements CrawlController.WebCrawlerFactory<T>, ApplicationContextAware {

    private ApplicationContext context;

    @Override
    public T newInstance() throws Exception {
        return (T) context.getBean(MyCrawler.class);
    }

    @Override
    public void setApplicationContext(ApplicationContext context) {
        this.context = context;
    }
}