Skip to content
Your preferred open source focused crawler for the deep web.
Java
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.github/ISSUE_TEMPLATE
src Experimental SOCKS4a proxy support added, checkstyle changed, v4.2.5 Nov 20, 2019
.gitignore
.mvn.xml
.travis.yml
LICENSE
README.md
checkstyle.xml
key.gpg.enc
pom.xml

README.md

Venom

Your preferred open source focused crawler for the Deep Web.

Maven Central Build Status Coverage Status Javadoc

Overview

Our aim is to create a blazing fast, fully customizable and robust crawler that is simple and handy to use.

Quick links

Website | API Reference | Wiki | Examples | Tutorial | PreferredAI

Features

  • Multi-threaded out of the box
  • Structured crawling with JSoup integration
  • Page Validation
  • Automatic Retries
  • Proxy support
  • Auto-save raw html pages

Getting started

Getting started with Venom is quick and easy. There are two ways to get started.

Clone our examples or tutorial

If you are starting out in a new project, you can consider cloning our Examples:

git clone https://github.com/PreferredAI/venom-examples.git

or, if you would like a more guided package, you can check out our Tutorial:

git clone https://github.com/PreferredAI/venom-tutorial.git

Add a dependency

If you already have a project then just add Venom as a dependency to your pom.xml:

<dependency>
    <!-- Venom: Your preferred focused crawler @ https://venom.preferred.ai/ -->
    <groupId>ai.preferred</groupId>
    <artifactId>venom</artifactId>
    <version>[4.2,4.3)</version>
</dependency>

Tutorial

If you are new to Venom, we have created a set of exercises to get you up and sprinting. The exercises are bundled in venom-tutorials. More information can be found on this page.

Wiki

If you would like more information check out our Wiki.

Example

Think you are beyond exercises? Get started quickly with these few lines of code.

public class Example {
 
    private static class VenomHandler implements Handler {
 
        @Override
        public void handle(Request request,
                           VResponse response,
                           Scheduler scheduler,
                           Session session,
                           Worker worker) {
 
            String about = response.getJsoup().select(".sub-title b").text();
            System.out.println("ABOUT: " + about);
 
        }
 
    }
 
    public static void main(String[] args) throws Exception {
        try (Crawler c = Crawler.buildDefault().start()) {
            Request r = new VRequest("https://venom.preferred.ai");
            c.getScheduler().add(r, new VenomHandler());
        }
    }
 
}

License

Apache License 2.0

You can’t perform that action at this time.