Skip to content
/ kit Public

๐Ÿ“š Libraries for retrieving and parsing robots.txt & sitemaps.

License

Notifications You must be signed in to change notification settings

spire-rs/kit

Folders and files

NameName
Last commit message
Last commit date

Latest commit

ย 

History

15 Commits
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 

Repository files navigation

Kit

Build Status Crate Coverage

Crates:

  • countio: The wrapper struct to enable byte counting for std::io::Read and std::io::Write and its asynchronous variants from futures and tokio.
  • robotxt: The implementation of the Robots.txt (or URL exclusion) protocol with the support of crawl-delay, sitemap and universal match extensions.
  • sitemapo: The implementation of the Sitemap (or URL inclusion) protocol with the support of txt, xml formats and video, image, and news extensions.

About

๐Ÿ“š Libraries for retrieving and parsing robots.txt & sitemaps.

Topics

Resources

License

Stars

Watchers

Forks

Languages