Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Check obj type using xml_url #8

Merged
merged 3 commits into from Oct 7, 2013
Merged

Check obj type using xml_url #8

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
94c994f
Crawler returns a Error when crawling failed
teddychoi Oct 5, 2013
1859208
Check obj type using xml_url
teddychoi Oct 7, 2013
e2cd1c2
Remove test print codes
teddychoi Oct 7, 2013
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
17 changes: 13 additions & 4 deletions libearth/crawler.py
View file
Open in desktop
Expand Up @@ -5,6 +5,7 @@

"""
import multiprocessing.pool
import sys
try:
import urllib.request as urllib2
except ImportError:
Expand Down Expand Up @@ -35,7 +36,15 @@ def crawl(feeds, pool_size):


def get_feed(feed_url):
f = urllib2.urlopen(feed_url)
feed_xml = f.read()
parser = get_format(feed_xml)
return feed_url, parser(feed_xml, feed_url)
try:
f = urllib2.urlopen(feed_url)
feed_xml = f.read()
parser = get_format(feed_xml)
return feed_url, parser(feed_xml, feed_url)
except:
return CrawlError(
'Crawling, {0} failed: {1}'.format(feed_url, sys.exc_info()[0]))


class CrawlError(IOError):
'''Error which rises when crawling given url failed.'''
2 changes: 1 addition & 1 deletion libearth/feedlist.py
View file
Open in desktop
Expand Up @@ -474,7 +474,7 @@ def make_feed(self, type, title, xml_url, html_url=None, text=None,
return feed

def convert_from_outline(self, outline_obj):
if outline_obj.children:
if not outline_obj.xml_url:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If outline element has children(s) and doesn't have xmlUrl attribute, that is a FeedCategory, not a Feed.
So it shoud be

if outline_obj.children or not outline.xml_url:

title = outline_obj.title or outline_obj.text
type = outline_obj.type
text = outline_obj.text
Expand Down