Permalink
Browse files

This change avoids to get this error:

...
exceptions.TypeError: Request url must be str or unicode, got NoneType:

Obtained when an "IMAGE" type field is defined, but no image url is retrieved, once the field is set as not mandatory the scraping process should go on and not throw errors.

This way non mandatory IMAGE scrape run flawless if the image xpath is parsed unsucessfully.
  • Loading branch information...
1 parent 43c3701 commit f216e02725dfe05ce5fe0eabd57a77536e7f7700 @PaoloC68 committed May 14, 2012
Showing with 1 addition and 1 deletion.
  1. +1 −1 dynamic_scraper/pipelines.py
@@ -24,7 +24,7 @@ def get_media_requests(self, item, info):
img_elem = info.spider.scraper.get_image_elem()
if img_elem.scraped_obj_attr.name in item:
return Request(item[img_elem.scraped_obj_attr.name])
- except ScraperElem.DoesNotExist:
+ except (ScraperElem.DoesNotExist, TypeError):
pass
def image_key(self, url):

2 comments on commit f216e02

Hi Paolo, feel free to send a direct pull request to the django-dynamic-scraper repository if you've got general improvements to the code or if you have found any bugs. I will definitely have a look at it and give a comment if further discussion is needed or integrate the code directly, then everyone can benefit from your improvements. Happy scraping! :-)

Hi Paolo, actually I know fixed this bug as well in the DDS repository. Greetings!

Please sign in to comment.