Your preferred focused crawler based on Venom. Now in a convenient package to quickly get your projects up to speed!
Bundled in this pack is a tutorial package to help get you up and sprinting. If you don't require a tutorial, you can access a fully functional example here. For more information, read the tutorial guide below.
Check out our main Venom page for more information.
venom-tutorial includes a set of tutorial designed to quickly get you from 0 to 100.
There are 7 exercises located in the package ai.preferred.crawler.example.tutorial. Alongside the exercises are a set of test suite that automatically checks your code and provide hints on errors, so you do not have to worry about not knowing whether your code works.
- Exercise 1: Creating a crawler with default settings.
- Exercise 2: Creating a fetcher that includes three (3) validators.
- Exercise 3: Creating a session store with PAPER_LIST_KEY.
- Exercise 4: Creating a crawler that uses a specified fetcher and session.
- Exercise 5: Creating a validator that validate a page.
- Exercise 6: Parsing the response from the crawl.
- Exercise 7: Putting it all together.
Easily find out what went wrong by running the tests included with the exercises.
You can run this command in the project folder