Skip to content
This repository

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Crawlista is a support library for Clojure applications that crawl the Web

Fetching latest commit…

Octocat-spinner-32-eaf2f5

Cannot retrieve the latest commit at this time

Octocat-spinner-32 src
Octocat-spinner-32 test
Octocat-spinner-32 .gitignore
Octocat-spinner-32 .travis.yml
Octocat-spinner-32 ChangeLog.md
Octocat-spinner-32 README.md
Octocat-spinner-32 project.clj
README.md

What is Crawlista

Crawlista is a support library for Clojure applications that crawl the Web.

Continuous Integration status

Features

  • Detection of crawlable content (using Pantomime)
  • Extraction of links with normalization (using Urly)
  • Detection of link presence
  • Extraction of content (like article text) from pages (using Boilerpipe Core)

Usage

Installation

With Leiningen

[clojurewerkz/crawlista "1.0.0-alpha17"]

Artifacts are published to clojars.org.

Supported Clojure versions

Crawlista is built from the ground up for Clojure 1.3 and up.

Documentation & Examples

Crawlista is a work in progress. Please see our test suite for code examples. Once APIs and core functionality stabilizes, we will begin writing documentation guides and update this document.

Crawlista Is a ClojureWerkz Project

Crawlista is part of the group of Clojure libraries known as ClojureWerkz, together with Neocons, Langohr, Elastisch, Welle, Monger, Quartzite and several others.

Continuous Integration

Continuous Integration status

CI is hosted by travis-ci.org

Development

Crawlista uses Leiningen 2. Make sure you have it installed and then run tests against all supported Clojure versions using

lein2 all test

Then create a branch and make your changes on it. Once you are done with your changes and all tests pass, submit a pull request on Github.

Regenerating robots.txt Parser

If you make changes to the Ragel-based robots.txt parser in Crawlista, you need to regenerate it:

ragel -J -o src/java/clojurewerkz/crawlista/robots/Parser.java src/rl/clojurewerkz/crawlista/robots/Parser.rl
lein2 javac

and then run robots.txt parser test suite with

lein2 test :robots

License

Copyright (C) 2011-2014 Michael S. Klishin

Distributed under the Eclipse Public License, the same as Clojure.

Something went wrong with that request. Please try again.