-
Notifications
You must be signed in to change notification settings - Fork 8.3k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
FEATURE: Replace SimpleRSS with Ruby RSS module (#5311)
* SPEC: PollFeedJob parsing atom feed * add FeedItemAccessor It is to provide a consistent interface to access a feed item's tag content. * add FeedElementInstaller to install non-standard and non-namespaced feed elements * FEATURE: replace SimpleRSS with Ruby RSS module * get FinalDestination and download with Excon * support namespaced element with FeedElementInstaller
- Loading branch information
1 parent
410994b
commit 5f318a5
Showing
10 changed files
with
284 additions
and
79 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,52 @@ | ||
require 'rexml/document' | ||
require 'rss' | ||
|
||
class FeedElementInstaller | ||
private_class_method :new | ||
|
||
def self.install(element_name, feed) | ||
# RSS Specification at http://cyber.harvard.edu/rss/rss.html#extendingRss | ||
# > A RSS feed may contain [non-standard elements], only if those elements are *defined in a namespace* | ||
|
||
new(element_name, feed).install if element_name.include?(':') | ||
end | ||
|
||
attr_reader :feed, :original_name, :element_namespace, :element_name, :element_accessor | ||
|
||
def initialize(element_name, feed) | ||
@feed = feed | ||
@original_name = element_name | ||
@element_namespace, @element_name = *element_name.split(':') | ||
@element_accessor = "#{@element_namespace}_#{@element_name}" | ||
end | ||
|
||
def element_uri | ||
@element_uri ||= REXML::Document.new(feed).root&.attributes&.namespaces&.fetch(@element_namespace, '') || '' | ||
end | ||
|
||
def install | ||
install_in_rss unless installed_in_rss? | ||
install_in_atom unless installed_in_atom? | ||
end | ||
|
||
private | ||
|
||
def install_in_rss | ||
RSS::Rss::Channel::Item.install_text_element(element_name, element_uri, '?', element_accessor, nil, original_name) | ||
RSS::BaseListener.install_get_text_element(element_uri, element_name, element_accessor) | ||
end | ||
|
||
def install_in_atom | ||
RSS::Atom::Entry.install_text_element(element_name, element_uri, '?', element_accessor, nil, original_name) | ||
RSS::Atom::Feed::Entry.install_text_element(element_name, element_uri, '?', element_accessor, nil, original_name) | ||
RSS::BaseListener.install_get_text_element(element_uri, element_name, element_accessor) | ||
end | ||
|
||
def installed_in_rss? | ||
RSS::Rss::Channel::Item.method_defined?(element_accessor) | ||
end | ||
|
||
def installed_in_atom? | ||
RSS::Atom::Entry.method_defined?(element_accessor) || RSS::Atom::Feed::Entry.method_defined?(element_accessor) | ||
end | ||
end |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,25 @@ | ||
class FeedItemAccessor | ||
attr_accessor :rss_item | ||
|
||
def initialize(rss_item) | ||
@rss_item = rss_item | ||
end | ||
|
||
def element_content(element_name) | ||
try_attribute_or_self(element(element_name), :content) | ||
end | ||
|
||
def link | ||
try_attribute_or_self(element(:link), :href) | ||
end | ||
|
||
private | ||
|
||
def element(element_name) | ||
rss_item.respond_to?(element_name) ? rss_item.send(element_name) : nil | ||
end | ||
|
||
def try_attribute_or_self(element, attribute_name) | ||
element.respond_to?(attribute_name) ? element.send(attribute_name) : element | ||
end | ||
end |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,40 @@ | ||
require 'feed_element_installer' | ||
require 'rails_helper' | ||
|
||
describe FeedElementInstaller do | ||
describe '#install_rss_element' do | ||
let(:raw_feed) { file_from_fixtures('feed.rss', 'feed').read } | ||
|
||
it 'creates parsing for a non-standard, namespaced element' do | ||
FeedElementInstaller.install('discourse:username', raw_feed) | ||
feed = RSS::Parser.parse(raw_feed) | ||
|
||
expect(feed.items.first.discourse_username).to eq('xrav3nz') | ||
end | ||
|
||
it 'does not create parsing for a non-standard, non-namespaced element' do | ||
FeedElementInstaller.install('username', raw_feed) | ||
feed = RSS::Parser.parse(raw_feed) | ||
|
||
expect { feed.items.first.username }.to raise_error(NoMethodError) | ||
end | ||
end | ||
|
||
describe '#install_atom_element' do | ||
let(:raw_feed) { file_from_fixtures('feed.atom', 'feed').read } | ||
|
||
it 'creates parsing for a non-standard, namespaced element' do | ||
FeedElementInstaller.install('discourse:username', raw_feed) | ||
feed = RSS::Parser.parse(raw_feed) | ||
|
||
expect(feed.items.first.discourse_username).to eq('xrav3nz') | ||
end | ||
|
||
it 'does not create parsing for a non-standard, non-namespaced element' do | ||
FeedElementInstaller.install('username', raw_feed) | ||
feed = RSS::Parser.parse(raw_feed) | ||
|
||
expect { feed.items.first.username }.to raise_error(NoMethodError) | ||
end | ||
end | ||
end |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,33 @@ | ||
require 'rss' | ||
require 'feed_item_accessor' | ||
require 'rails_helper' | ||
|
||
describe FeedItemAccessor do | ||
context 'for ATOM feed' do | ||
let(:atom_feed) { RSS::Parser.parse(file_from_fixtures('feed.atom', 'feed'), false) } | ||
let(:atom_feed_item) { atom_feed.items.first } | ||
let(:item_accessor) { FeedItemAccessor.new(atom_feed_item) } | ||
|
||
describe '#element_content' do | ||
it { expect(item_accessor.element_content('title')).to eq(atom_feed_item.title.content) } | ||
end | ||
|
||
describe '#link' do | ||
it { expect(item_accessor.link).to eq(atom_feed_item.link.href) } | ||
end | ||
end | ||
|
||
context 'for RSS feed' do | ||
let(:rss_feed) { RSS::Parser.parse(file_from_fixtures('feed.rss', 'feed'), false) } | ||
let(:rss_feed_item) { rss_feed.items.first } | ||
let(:item_accessor) { FeedItemAccessor.new(rss_feed_item) } | ||
|
||
describe '#element_content' do | ||
it { expect(item_accessor.element_content('title')).to eq(rss_feed_item.title) } | ||
end | ||
|
||
describe '#link' do | ||
it { expect(item_accessor.link).to eq(rss_feed_item.link) } | ||
end | ||
end | ||
end |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,30 @@ | ||
<?xml version="1.0" encoding="UTF-8"?> | ||
<feed | ||
xmlns="http://www.w3.org/2005/Atom" | ||
xmlns:thr="http://purl.org/syndication/thread/1.0" | ||
xmlns:discourse="http://discourse.org/rss/modules/discourse/" | ||
xml:lang="en-US" | ||
xml:base="https://blog.discourse.org/wp-atom.php" | ||
> | ||
<title type="text">Discourse</title> | ||
<subtitle type="text">Official blog for the open source Discourse project</subtitle> | ||
<updated>2017-10-23T23:45:37Z</updated> | ||
<link rel="alternate" type="text/html" href="https://blog.discourse.org" /> | ||
<id>https://blog.discourse.org/feed/atom/</id> | ||
<link rel="self" type="application/atom+xml" href="https://blog.discourse.org/feed/atom/" /> | ||
<generator uri="https://wordpress.org/" version="4.8.2">WordPress</generator> | ||
<entry> | ||
<discourse:username><![CDATA[xrav3nz]]></discourse:username> | ||
<author> | ||
<name>xrav3nz</name> | ||
</author> | ||
<title type="html"><![CDATA[Poll Feed Spec Fixture]]></title> | ||
<link rel="alternate" type="text/html" href="https://blog.discourse.org/2017/09/poll-feed-spec-fixture/" /> | ||
<id>https://blog.discourse.org/?p=pollfeedspec</id> | ||
<updated>2017-09-14T15:22:33Z</updated> | ||
<published>2017-09-14T15:22:33Z</published> | ||
<category scheme="https://blog.discourse.org" term="design" /> | ||
<summary type="html"><![CDATA[Here are some random descriptions... […]]]></summary> | ||
<content type="html" xml:base="https://blog.discourse.org/2017/09/poll-feed-spec-fixture/"><![CDATA[<p>This is the body & content. </p>]]></content> | ||
</entry> | ||
</feed> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
5f318a5
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This commit has been mentioned on Discourse Meta. There might be relevant details there:
https://meta.discourse.org/t/autobot-automatic-content-creator/71756/70
5f318a5
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This commit has been mentioned on Discourse Meta. There might be relevant details there:
https://meta.discourse.org/t/configure-the-discourse-rss-polling-plugin/156387/6