robots.txt parser for Rust.
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
src
tests
.gitignore
.travis.yml
Cargo.toml
LICENSE
README.md
appveyor.yml
rustfmt.toml

README.md

robotparser-rs

Build Status Build status Coverage Status Crates.io Dependency status

robots.txt parser for Rust.

Installation

Add it to your Cargo.toml:

[dependencies]
robotparser = "0.10"

Add extern crate robotparser to your crate root and your're good to go!

Examples

extern crate robotparser;

use robotparser::RobotFileParser;

fn main() {
    let parser = RobotFileParser::new("http://www.python.org/robots.txt");
    parser.read();
    assert!(parser.can_fetch("*", "http://www.python.org/robots.txt"));
}

License

This work is released under the MIT license. A copy of the license is provided in the LICENSE file.