Issue: Duplicated Feed Entries

This issue started happening earlier today...

I have cleared out my unread list several times this afternoon, and the duplicates come back every time there are new, unread entries. I'm not sure it's always the same duplicates - but I do know that the ones pictured below do recur often...

Also, the last time I read through all unread entries, I clicked "Mark all as read" to see if that would help. It apparently did not.

[benubois]

Thanks, seeing the duplicate entry issue some feeds. Looking into it...

[benubois]

Fixed.

There will be many duplicates from high volume feeds today, but no more going forward.

[recurser]

Seeing quite a few doubled-up posts pointing at the same URL:

[benubois]

Yeah, this is a major problem on Hacker News. An entry on HN looks like

<item>
  <title>Portal Released For Steam On Linux</title>
  <link>http://www.phoronix.com/scan.php?page=news_item&amp;px=MTM2Mzk</link>
  <comments>https://news.ycombinator.com/item?id=5647914</comments>
  <description><![CDATA[<a href="https://news.ycombinator.com/item?id=5647914">Comments</a>]]></description>
</item>

So there isn't much to uniquely identify items by. In cases where a publisher does not have and <id> or <guid> or <published>, Feedbin uses a combination of the link and title to attempt to uniquely identify items.

If your example a period was added to the headline later on, which makes it look like a duplicate to a human, but it looks unique to the id generator.

[recurser]

Aha I see what you mean... I didn't notice that the titles are slightly different. Curious since it's the official HN feed, and they have the same HN post ID in the link, so it's the same canonical 'article' so to speak. I guess these are cases of the title being edited by HN mods after feedbin has already picked them up?

So there isn't much to uniquely identify items by. In cases where a publisher does not have 
and <id> or <guid> or <published>, Feedbin uses a combination of the link and title to attempt
to uniquely identify items.

In the case of hacker news, item?id=5647914 uniquely identifies it, though I realise it's a slippery slope once you start customizing things on a feed-by-feed basis.

Thanks for the explanation!

[benubois]

In the case of hacker news, item?id=5647914 uniquely identifies it, though I realise it's a slippery slope once you start customizing things on a feed-by-feed basis.

Hehe exactly. As I was posting the example I noticed that the <description> on Hacker News would make for a great ID, but that would be totally unique to them.

If I do start customizing the strategy for certain feeds this is first on my list.

[nissimk]

Use a hash of the item:

http://swik.net/RSS/RSS+Item+Uniqueness

[recurser]

@nissimk

Hash – most common method, simply hashing the entire item results in a somewhat unique id. This is however vulnerable to repeated feed items.

The problem is that the title of articles is changing over time, which makes hashing pretty difficult in the absence of a canonical id (?)

[benubois]

Here's the id strategy for Feedbin, definitely open to suggestions, although any changes would have to maintain backward compatibility so duplicates of old entries are not created:

def build_public_id(entry, feedzirra, saved_feed_url = nil)
  if saved_feed_url
    id_string = saved_feed_url.dup
  else      
    id_string = feedzirra.feed_url.dup
  end

  if entry.entry_id
    id_string << entry.entry_id.dup
  else
    if entry.url
      id_string << entry.url.dup
    end
    if entry.published
      id_string << entry.published.iso8601
    end
    if entry.title
      id_string << entry.title.dup
    end
  end
  Digest::SHA1.hexdigest(id_string)
end

[nissimk]

What about just excluding the title field from the item before hashing?

[benubois]

What about just excluding the title field from the item before hashing?

Definitely something I considered.

There are feeds that link to the same story multiple times, so the the link is not necessarily unique.

At the time I was thinking it would be better to create a duplicate than not import the item at all and potentially have missing unique items. I'm not sure which is the more common case.

Certainly with HN this is a bigger problem than other feeds.

Note: This has also been happening to me pretty often on Engadget's feed, if you need another test case...

[Zegnat]

@roomanitarian that’s weird, are you subscribed to http://www.engadget.com/rss.xml? That one includes guid elements for Feedbin to use so there should not be any duplicates. If there are any duplicates it’s either because Feedbin is broken or because Engadget is changing their own unique IDs (which would be completely silly and probably means a broken CMS on their part).

@Zegnat

Actually, I just removed that feed last week, due mainly to lack of interest on my part...

However, I just added it back in to see if I could find any duplicates... I didn't have to look very hard to find these 2 sets: