Skip to content
Browse files

Use correct parameter to is_allowed. Thanks to Thomas Bikeev.

  • Loading branch information...
1 parent 15ae16b commit e74c04d8006340a2e4d2d8398c083c52c523785c Andrew Wilkinson committed
Showing with 1 addition and 1 deletion.
  1. +1 −1 crawler/
2 crawler/
@@ -34,7 +34,7 @@ def update(self):
parse = urlparse(self.url)
robotstxt = RobotsTxt.get_by_domain(parse.scheme, parse.netloc)
- if not robotstxt.is_allowed(parse.netloc):
+ if not robotstxt.is_allowed(self.url):
return False
while cache.get(parse.netloc) is not None:

0 comments on commit e74c04d

Please sign in to comment.
Something went wrong with that request. Please try again.