Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Link extraction from RSS feeds fails on XML entities #204

Closed
sebastian-nagel opened this issue Apr 11, 2018 · 0 comments · Fixed by #205
Closed

Link extraction from RSS feeds fails on XML entities #204

sebastian-nagel opened this issue Apr 11, 2018 · 0 comments · Fixed by #205
Labels
sitemaps

Comments

@sebastian-nagel
Copy link
Contributor

If the content of elements in RSS feeds contain XML entities the link extraction does not the correct link: eg. <link>http://get2ch.net/?category=it&amp;pickup_id=2426594</link> (from http://get2ch.net/feed/news/?category=it). Need a solution same as in 8467517.
sebastian-nagel added a commit to sebastian-nagel/crawler-commons that referenced this issue Apr 11, 2018
@sebastian-nagel
Improve sitemap parsing
06e2601 
- ignore query part of URL to determine sitemap location prefix
  for URL validation, fixes crawler-commons#202
- resolve relative links in RSS feeds, fixes crawler-commons#203
- allow non-continuous content (containing XML entities or CDATA)
  when parsing links in RSS feeds, fixes crawler-commons#204
- extract links from <guid> elements in RSS feeds, fixes crawler-commons#201
@sebastian-nagel sebastian-nagel mentioned this issue Apr 11, 2018
Merged
sebastian-nagel added a commit to sebastian-nagel/crawler-commons that referenced this issue Apr 25, 2018
@sebastian-nagel
Improve sitemap parsing
0ef7cf8 
- ignore query part of URL to determine sitemap location prefix
  for URL validation, fixes crawler-commons#202
- resolve relative links in RSS feeds, fixes crawler-commons#203
- allow non-continuous content (containing XML entities or CDATA)
  when parsing links in RSS feeds, fixes crawler-commons#204
- extract links from <guid> elements in RSS feeds, fixes crawler-commons#201
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
sitemaps
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Improve sitemap parsing sebastian-nagel/crawler-commons
1 participant
@sebastian-nagel