Generalized feed reader #31

semmons99 · 2012-02-16T19:14:30Z

Here's the beginnings of a generalized RSS plugin for Mendibot. @gjp what do you think?
#30

gjp · 2012-02-16T19:25:45Z

👍 for "epic saga"

How to test this? With one feed and one channel, I set the pull interval much smaller than min_time and confirmed that the posts were periodically sent to the channel. With many-to-many I could see this getting fussy.

semmons99 · 2012-02-16T19:39:35Z

I'm currently playing around in #semmons99-test1 and #semmons99-test2 with a handful of different RSS feeds and configurations.

…verwritten

semmons99 · 2012-02-16T20:48:59Z

I think I've worked out the kinks. I'm going to leave my test bot sitting in #semmons99-test1 and #semmons99-test2 for a few hours to make sure things go as planned. If everything looks good I'll merge later tonight.

gjp · 2012-02-16T21:35:45Z

Sorry I didn't have anything to contribute here. I should be around for most of the day if there is something else I can help with.

jordanbyron · 2012-02-16T21:39:26Z

Sweet. Let me know once this is merged so I can take care of #26

semmons99 · 2012-02-16T21:42:38Z

@gjp actually there's a couple things you could help with.

Test more feeds to see if there's any more errors.
Add author back to the message and default it to "a mysterious stranger"

It would also be fun to see multiple templates so it's not always "an epic saga".

jordanbyron · 2012-02-16T21:51:57Z

It would also be fun to see multiple templates so it's not always "an epic saga".

👍 I would love to see that!

gjp · 2012-02-16T22:55:35Z

@semmons99 what sort of errors were you running across earlier? You'd mentioned every test feed was failing. Was that due to the RSS namespace?

alternatives to "epic saga": hmm, should we drop that into #mendicant and see how many suggestions we can gather?

semmons99 · 2012-02-16T23:04:38Z

@gjp mostly missing pubDates.

Yeah ask #mendicant for suggestions.

semmons99 · 2012-02-17T03:14:44Z

@gjp, @jordanbyron, @sandal

I've been playing around with this for the past couple hours and have come to the conclusion that most site's RSS feeds are a pile of 💩. We can't rely on much of anything to be there other than <link>. So we have two options as I see it.

Only allow well formed RSS feeds to be monitored.
Use PStore to store all the links we've announced and use that to determine what's new.

Thoughts?

jordanbyron · 2012-02-17T03:53:01Z

@semmons99 the two main RSS feeds I want to support are Community and GitHub. It looks like GitHub is using updated and we can easily add an updated attribute to the Community RSS feed.

Would that be enough to get this working?

gjp · 2012-02-17T05:43:24Z

I'm pretty new to RSS formats but I think I see what's going on here. There's RSS, there's Atom, and then there's a site like Hacker News which strips out every byte they can to save bandwidth, standards be damned, and calls it RSS anyway.

I'd suggest checking both the pubdate and updated attributes to handle both RSS and Atom, but to ignore invalid feeds. Any site which is so busy that they need to strip all dates is probably too spammy for IRC anyway. Or they just suck.

unrelated: @sandal suggested combining synonyms of "epic" and "saga". I'll make a list of overused words from book blurbs. It'll be a stunning tour de force.

semmons99 · 2012-02-17T12:11:01Z

@jordanbyron @gjp sounds good.

@gjp think you can update this branch to use pubDate and updated when you're adding our synonyms feature?

wicz · 2012-02-17T13:21:54Z

Since we want to support just a few feeds, can't we use a 3rd party tool, e.g. feedburner, to normalize this feeds for us? Or maybe Yahoo! pipes?

semmons99 · 2012-02-17T14:32:30Z

@wicz I'd rather avoid thirdparty tools and just write the logic to handle the feed types we need. The more we can rely on ourselves at mendicant the better.

jordanbyron · 2012-02-17T14:36:17Z

@semmons99 👍 I totally agree. Keep it simple

practicingruby · 2012-02-17T15:56:52Z

@wicz: With the exception of Github, Mendicant University does not use third party services unless they are based on open source and easily replaceable. There are a ton of reasons for this, which reminds me that @jordanbyron and I really need to write up our guidelines...

wicz · 2012-02-17T15:57:33Z

@semmons99 @jordanbyron Agreed, I don't like to add dependencies either, unless they're extremely necessary. But just to make sure we're on the same page here, I was talking about using the feedburner.com service and not any other library with the same name.

Just pointing it out cause I've used it before when needed to aggregate news from a bunch of shitty feeds.

EDIT: Anyway, it won't work cause it seems it doesn't support HTTPS :boo:

practicingruby · 2012-02-17T15:59:52Z

@wicz: A service dependency is an even heavier dependency than a third party library. We don't mind using libraries, because we can maintain them ourselves if/when they have problems. Services we have no control over.

wicz · 2012-02-17T16:02:08Z

@sandal You're absolutely right!

ps: comments ordering got a bit clumsy, but I'm glad we agreed anyway 😉

gjp · 2012-02-17T17:05:02Z

@semmons99 sure, I can roll up some small additions. Would you like to merge this now or wait for me to experiment on this branch? My schedule is uncertain today.

semmons99 · 2012-02-20T17:06:38Z

@gjp got any time to work on this?

gjp · 2012-02-20T17:23:49Z

I had open tickets on 4 projects before I drove out of state for the weekend. By the time I was home, 💤.

I'll background MoM and Community and start here with pubdate/updated and author. Do you mind if we make wordplay on the announcements a separate patch? I don't want to cause further delay to this one.

semmons99 · 2012-02-20T17:24:41Z

@gjp sure, that's fine.

semmons99 · 2012-02-20T17:25:05Z

@gjp feel free to work on MoM and Community first too.

jordanbyron · 2012-02-20T17:31:23Z

@semmons99 @gjp both are blocked by me at this point, so feel free to work on this first :wink2:

gjp · 2012-02-20T18:35:56Z

Working on this one. It's trial and error with different test feeds, so I might narrow it down to Community and GitHub and not worry about weird variations.

Do you think the refresh interval should remain universal, or set per-feed?

semmons99 · 2012-02-20T18:41:32Z

@gjp Just focus on Community and Github. They're all we care about right now.

Also I can see the value in having a per-feed interval. It would allow us to follow github more actively.

wicz · 2012-02-20T18:57:40Z

Hi guys, sorry for not responding earlier.

This week I'll be working on puzzlenode, but you can count on me to help with this and #32 as well.

semmons99 · 2012-02-20T19:02:10Z

@wicz awesome!

gjp · 2012-02-20T23:20:28Z

I'm feeling the temptation to try a different parser. Doing this on multiple fields feels wrong
item.respond_to?(:pubDate) && item.pubDate

and item.updated is not being cast to a Time but item.pubDate is. 😒

@semmons99 How were you using Pry to help here? I've only noodled around with it. Given how long it's taking me to make one change and test it, I must be doing something wrong.

gjp · 2012-02-20T23:41:21Z

Feedzirra is making me happy.

semmons99 · 2012-02-21T00:45:01Z

@gjp awesome, the description alone warrants inclusion into mendibot.

I was using pry by inserting:

require "pry"
binding.pry

where I kept having trouble and inspecting the items as they came through. I really don't use any of it's fancy features. Most of the time I just inspect the object, look at methods, etc. If you're stuck at all and want to try some remote pairing tomorrow, let me know. I'm pretty sure I can setup my vps to function for remote pairing.

gjp · 2012-02-21T02:02:53Z

Feedzirra normalizes the feeds nicely. Now I'm stuck on something which is making me feel brain-dead. With a single global timer, we just iterate through all feeds on each refresh. With per-feed timers, we need to know which feed to refresh, when the timers are set up.

      Mendibot::Config::RSS_SETTINGS[:feeds].each do |feed|
        timer feed_interval(feed), method: :pull
      end

Will Cinch accept arguments to the pull method? How?

Also I'd prefer to be able to do feed.interval, but feeds are just a hash.

semmons99 · 2012-02-21T02:35:54Z

Let's just define each of the pull methods programmatically.

require 'cinch'
require 'open-uri'
require 'rss'

module Mendibot
  module Plugins
    class RSSRelay
      include Cinch::Plugin

      Mendibot::Config::RSS_SETTINGS[:feeds].each do |feed|
        method = "#{feed[:name]}_pull".to_sym

        timer feed[:interval], method: method

        define_method method do
          open(feed[:url]) do |rss|
            #... regular rss stuff
          end
        end
      end
    end
  end
end

wicz · 2012-02-21T02:37:28Z

Let's go by parts 😉

@jordanbyron The thing about one using pubDate and the other updated is that Community is RSS 2.0 and GitHub is Atom 1.0, both almost valid. That said, I don't think adding updated is the solution cause it will mix the two formats in a wrong way. Also, I think we could take care to make Community RSS valid. I'll open an issue and can take a look at it later.

@gjp I don't think we need a different parser here. The stdlib's RSS is capable of parsing both formats, all we need is to be able to distinguish between them.

gh = RSS::Parser.parse(open('https://github.com/jordanbyron/mission_of_mercy/commits/master.atom'))
gh.feed_type # => "atom"

cm = RSS::Parser.parse(open('http://community.mendicantuniversity.org/articles.rss'))
cm.feed_type # => "rss"

gh_entry = gh.items.first
cm_entry = cm.items.first

cm_entry.title.class # => String
cm_entry.title # => "Tempr - no fussin' temporal expressions"

cm_entry.pubDate.class # => Time
cm_entry.pubDate # => 2012-02-16 16:07:15 -0200

gh_entry.title.class # => RSS::Atom::Feed::Title
gh_entry.title.content # => "Now that loading zipcodes is fast, we don't need progress_bar"

gh_entry.updated.content.class  # => Time
gh_entry.updated.content # => 2012-02-16 18:32:34 -0200

Note that we have to call content in the Atom feed. But it automatically casts to Time.

About Feedzirra, it seems a nice project, and even though it can make our life easier, there is a heavy tradeoff in game: it depends on ActiveSupport. I think we can cast our own solution based only in the standard RSS parser. Maybe we can create an Entry class to encapsulate the feed_type conditionals?

@gjp @semmons99 You might want take a closer look at pry-nav and pry-stack_explorer

Thoughts?

semmons99 · 2012-02-21T02:49:46Z

@wicz and @gjp can you work together to build something using stdlib rss to function as we need it? Perhaps craft a simple gem and release it?

gjp · 2012-02-21T03:21:01Z

@semmons99 so I was not missing something? Dynamically creating N methods is the simplest solution?

@wicz after a couple hours of seemingly getting a different failure mode each time I changed feeds (grrr), dropping in Feedzirra and having it just work with less code (specific to Mendibot) across community, github, and CNN (a test for missing author) felt like a win. 👍 for abstracting away trivia.

        items.each do |item|
          next unless post_time = item.published

          if post_time >= Time.now - RSSRelay.feed_interval(feed)
            author = item.author || 'a mysterious stranger'

            feed[:channels].each do |chan|
              Channel(chan).send <<-MESSAGE
via #{feed[:name]} comes the epic saga "#{item.title}", by #{author}! #{item.url}
MESSAGE
            end
          end
        end

Note how pubDate and updated are normalized to published, author is author, and the url is url, without any respond_to? checks. Of course that's not free, but this is code that will only run perhaps once every 15 minutes. That leaves the dependency stack, which as you've pointed out, may be unnecessarily large. There are a lot of features we're not using.

How about if I get at least one working version committed to this branch with Feedzirra before breaking it again, and then we can attempt to reach the same or better level of brevity without it?

semmons99 · 2012-02-21T03:34:18Z

@semmons99 so I was not missing something? Dynamically creating N methods is the simplest solution?

Simplest that comes to mind.

gjp · 2012-02-21T03:39:38Z

This amuses me way more than it probably should.

(09:35:01 PM) mendibot-test:   via teh githubs comes the epic saga "gjp pushed to generalize-rss at gjp/mendibot", by gjp! https://github.com/gjp/mendibot/compare/4d79d3de85...6b61370f10

...using the code to monitor changes to itself ✨

gjp · 2012-02-21T03:41:37Z

infinite regression!

(09:39:58 PM) mendibot-test:   via teh githubs comes the epic saga "gjp commented on pull request 31 on mendicant-university/mendibot", by gjp! https://github.com/mendicant-university/mendibot/issues/31#issuecomment-4066922

gjp · 2012-02-21T03:56:30Z

I should probably explain why feed_interval exists, since my config file is not tracked here. I experimented with having both a global (default) interval and a per-feed interval. It may make more sense to remove this and always set the interval per-feed.

    RSS_SETTINGS = {
      interval: 23,
      feeds: [
        {
              name: "community feed",
               url: 'http://community.mendicantuniversity.org/articles.rss',
          channels: ["##gjp-mendibot1", "##gjp-mendibot2"]
        },
        {
          interval: 27,
              name: "teh githubs",
               url: 'https://github.com/gjp.atom',
          channels: ["##gjp-mendibot2"]
        },
        {
          interval: 29,
              name: "CNN",
               url: 'http://rss.cnn.com/rss/cnn_topstories.rss',
          channels: ["##gjp-mendibot1"]
        }
      ]
    }

practicingruby · 2012-02-21T04:05:32Z

tl;dr on the thread but @jordanbyron @semmons99 @gjp, is there any reason why we wouldn't want to switch community to output Atom 1.0 and then not have to consider multiple types of feeds?

semmons99 · 2012-02-21T04:12:33Z

tl;dr on the thread but @jordanbyron @semmons99 @gjp, is there any reason why we wouldn't want to switch community to output Atom 1.0 and then not have to consider multiple types of feeds?

If it's easy enough for Jordan, that'd be fine with me.

semmons99 · 2012-02-21T04:13:34Z

It may make more sense to remove this and always set the interval per-feed.

@gjp I think it makes more sense to just require each feed to specify it's own interval.

semmons99 · 2012-02-21T04:16:22Z

(09:39:58 PM) mendibot-test:   via teh githubs comes the epic saga "gjp commented on pull request 31 on mendicant-university/mendibot", by gjp! https://github.com/mendicant-university/mendibot/issues/31#issuecomment-4066922

I think I might have just awoken my wife having literally lol'd 😂

jordanbyron · 2012-02-21T14:52:57Z

@sandal: is there any reason why we wouldn't want to switch community to output Atom 1.0 and then not have to consider multiple types of feeds?

Hum ... that sounds familiar to something I suggested a few days ago 😏

@jordanbyron: It looks like GitHub is using updated and we can easily add an updated attribute to the Community RSS feed.

Switching the RSS to Atom or adding an additional Atom feed to Community makes sense to me.

semmons99 · 2012-02-21T15:15:11Z

Switching the RSS to Atom or adding an additional Atom feed to Community makes sense to me.

@jordanbyron Think you can do that this week? Or do you want to keep the focus on PuzzleNode?

jordanbyron · 2012-02-21T15:17:23Z

Think you can do that this week? Or do you want to keep the focus on PuzzleNode?

I don't want to overcommit, so let's say I won't have time to do that until next week.

Created the ticket for adding an Atom feed in community if anyone is interested in picking it up: https://github.com/mendicant-university/community/issues/73

jordanbyron · 2012-02-23T14:47:24Z

Thanks to @wicz community now has an Atom feed:

http://community.mendicantuniversity.org/articles.atom

Let me know if there is anything else we need to do on the community site to get this patch merged 😄

semmons99 added 2 commits February 16, 2012 14:13

first stab at generalized rss. in reference to issue #30

9477cae

replace CommunityRSS references with RSS, testing ftw

40d16be

semmons99 added 6 commits February 16, 2012 14:42

namespace fix

74fdc8e

update .gitignore

be65f9e

stop relying on default_environment, no more stupid CONSTANTS being o…

4eb2e9f

…verwritten

rename to rss_relay, sick of colliding with top-level RSS namespace

08cc658

use pry in dev/test

6244022

cleanup some bad naming and drop author since it's not reliable

e472370

more pubDate checks

7c8b6da

semmons99 mentioned this pull request Feb 17, 2012

Github Commit Tracker #32

Open

Generalized feed reader #31

Are you sure you want to change the base?

Generalized feed reader #31

Conversation

semmons99 commented Feb 16, 2012

gjp commented Feb 16, 2012

semmons99 commented Feb 16, 2012

semmons99 commented Feb 16, 2012

gjp commented Feb 16, 2012

jordanbyron commented Feb 16, 2012

semmons99 commented Feb 16, 2012

jordanbyron commented Feb 16, 2012

gjp commented Feb 16, 2012

semmons99 commented Feb 16, 2012

semmons99 commented Feb 17, 2012

jordanbyron commented Feb 17, 2012

gjp commented Feb 17, 2012

semmons99 commented Feb 17, 2012

wicz commented Feb 17, 2012

semmons99 commented Feb 17, 2012

jordanbyron commented Feb 17, 2012

practicingruby commented Feb 17, 2012

wicz commented Feb 17, 2012

practicingruby commented Feb 17, 2012

wicz commented Feb 17, 2012

gjp commented Feb 17, 2012

semmons99 commented Feb 20, 2012

gjp commented Feb 20, 2012

semmons99 commented Feb 20, 2012

semmons99 commented Feb 20, 2012

jordanbyron commented Feb 20, 2012

gjp commented Feb 20, 2012

semmons99 commented Feb 20, 2012

wicz commented Feb 20, 2012

semmons99 commented Feb 20, 2012

gjp commented Feb 20, 2012

gjp commented Feb 20, 2012

semmons99 commented Feb 21, 2012

gjp commented Feb 21, 2012

semmons99 commented Feb 21, 2012

wicz commented Feb 21, 2012

semmons99 commented Feb 21, 2012

gjp commented Feb 21, 2012

semmons99 commented Feb 21, 2012

gjp commented Feb 21, 2012

gjp commented Feb 21, 2012

gjp commented Feb 21, 2012

practicingruby commented Feb 21, 2012

semmons99 commented Feb 21, 2012

semmons99 commented Feb 21, 2012

semmons99 commented Feb 21, 2012

jordanbyron commented Feb 21, 2012

semmons99 commented Feb 21, 2012

jordanbyron commented Feb 21, 2012

jordanbyron commented Feb 23, 2012