We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
When parsing https://www.deviantart.com/robots.txt
https://www.deviantart.com/robots.txt
User-agent: * Disallow: /*q= Disallow: /users/*? Disallow: /join/*? Disallow: /morelikethis/ Disallow: /download/ Disallow: /checkout/ Disallow: /global/ Disallow: /api/ Disallow: /critiques/ Sitemap: http://sitemaps.deviantart.net/sitemap-index.xml.gz
the parser fails with
thread 'main' panicked at 'assertion failed: !val.is_empty()', /home/me/.local/share/cargo/registry/src/github.com-1ecc6299db9ec823/robotstxt-0.2.0/src/parser.rs:207:17
Reproduction:
use robotstxt::DefaultMatcher; fn main() { let robots_content = r#"User-agent: * Disallow: /*q= Disallow: /users/*? Disallow: /join/*? Disallow: /morelikethis/ Disallow: /download/ Disallow: /checkout/ Disallow: /global/ Disallow: /api/ Disallow: /critiques/ Sitemap: http://sitemaps.deviantart.net/sitemap-index.xml.gz"#; let mut matcher = DefaultMatcher::default(); matcher.one_agent_allowed_by_robots(&robots_content, "oldnews", "https://www.deviantart.com/"); }
I'm assuming it is because of the line between the Disallows and the Sitemap, which only contains a single space.
Disallow
Sitemap
The text was updated successfully, but these errors were encountered:
It seems someone else patched this bug on their fork: scascketta@ffe972d
Sorry, something went wrong.
67475a1
Hi @iyzana. Hugely thanks for your feedback. 👍 This has been fixed via 67475a1.
No branches or pull requests
When parsing
https://www.deviantart.com/robots.txt
the parser fails with
Reproduction:
I'm assuming it is because of the line between the
Disallow
s and theSitemap
, which only contains a single space.The text was updated successfully, but these errors were encountered: