Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Browse files

Use correct parameter to is_allowed. Thanks to Thomas Bikeev.

  • Loading branch information...
commit e74c04d8006340a2e4d2d8398c083c52c523785c 1 parent 15ae16b
Andrew Wilkinson authored
Showing with 1 addition and 1 deletion.
  1. +1 −1  crawler/models.py
View
2  crawler/models.py
@@ -34,7 +34,7 @@ def update(self):
parse = urlparse(self.url)
robotstxt = RobotsTxt.get_by_domain(parse.scheme, parse.netloc)
- if not robotstxt.is_allowed(parse.netloc):
+ if not robotstxt.is_allowed(self.url):
return False
while cache.get(parse.netloc) is not None:
Please sign in to comment.
Something went wrong with that request. Please try again.