A tutorial based on your preferred open source focused crawler for the deep web.
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
src
.gitignore
LICENSE
README.md
pom.xml

README.md

Venom Tutorial

Your preferred focused crawler based on Venom. Now in a convenient package to quickly get your projects up to speed!

Bundled in this pack is a tutorial package to help get you up and sprinting. If you don't require a tutorial, you can access a fully functional example here. For more information, read the tutorial guide below.

Maven Central Build Status Coverage Status Javadocs

Overview

Check out our main Venom page for more information.

Quick links

Website | API Reference | PreferredAI

Tutorial

venom-tutorial includes a set of tutorial designed to quickly get you from 0 to 100.

There are 7 exercises located in the package ai.preferred.crawler.example.tutorial. Alongside the exercises are a set of test suite that automatically checks your code and provide hints on errors, so you do not have to worry about not knowing whether your code works.

TutorialCrawler.java

  • Exercise 1: Creating a crawler with default settings.
  • Exercise 2: Creating a fetcher that includes three (3) validators.
  • Exercise 3: Creating a session store with PAPER_LIST_KEY.
  • Exercise 4: Creating a crawler that uses a specified fetcher and session.

TutorialValidator.java

  • Exercise 5: Creating a validator that validate a page.

TutorialHandler.java

  • Exercise 6: Parsing the response from the crawl.

TutorialCrawler.java

  • Exercise 7: Putting it all together.

Test Suite

Easily find out what went wrong by running the tests included with the exercises.

You can run this command in the project folder

mvn test

Or use your IDE to run JUnit tests Test Suite

License

Apache License 2.0