Superfeedr rewriting GUID in feed (according to dlvr.it), which causes dupes

[archon810]

Hi guys,

We started using Superfeedr when working on the APKMirror.com Pushbullet integration (you may have heard from them while we were working on it), but now facing issues with dlvr.it which pings all social services with the link.

Basically, it boils down to this:

• We publish feeds with guids that have ampersands encoded as &.
• According to dlvr.it, Superfeedr rewrites these guids and replaces & with & - which is another valid way to write this character.
• The Superfeedr ping with & gets published by dlvr.it instantly.
• 15 minutes later, they do a manual pull and find the link with & in it, don't properly dedupe it, and treat it as a new link they'd never seen before, thereby posting a dupe.

This has been going on for days, and I've been deleting every dupe manually, hoping they'd resolve it. But they haven't.

So, now I'm seeing if we can work this out on this end. If Superfeedr didn't rewrite the guid, which to be honest I don't think it should be doing, the problem wouldn't have occurred in the first place. Do you think it's feasible to stop doing that? They also said you rewrite the dates to a different format, which I think is less paramount here.

The feed in question is http://www.apkmirror.com/feed.

Do you think you can help with a resolution and end the nightmares related to manual deleting of dupes here?

Thank you!

P.S. Is there any way to see the feed the way Superfeedr parses and outputs it, to debug the issue?

[julien51]

Indeed, you have the right diagnostic. However, as a matter of facts, we normalize the content we send to Dlvr.it (or to anyone else) to Atom... and Atom is XML. The XML spec is also quite clear:

The ampersand character (&) and the left angle bracket (<) may appear in their literal form only when used as markup delimiters, or within a comment, a processing instruction, or a CDATA section. If they are needed elsewhere, they must be escaped using either numeric character references or the strings "&" and "<" respectively. The right angle bracket (>) may be represented using the string ">", and must, for compatibility, be escaped using ">" or a character reference when it appears in the string "]]>" in content, when that string is not marking the end of a CDATA section.

Now, this probably does not help you very much. Is there any chance you change the guid in your feed so that they do not include ampersands? That would solve the problem?

[julien51]

Also, another solution would be to use the Atom feed from your WP install. For some reason, it's now empty... but that's probably a setting in your WP?

[archon810]

@julien51 Wait, so based on the XML spec, the & character should be escaped to become &, but it's already in the form of &. Plus, it doesn't say anywhere that you should be changing it to &, in fact.

Even if we didn't have a correctly encoded & already, wouldn't the right way according to spec be & anyway?

[archon810]

@julien51 Check it out yourself: view-source:http://www.apkmirror.com/feed.

<guid isPermaLink="false">http://www.apkmirror.com/?post_type=apps_post&#038;p=1661</guid>

[julien51]

Hum... the spec I linked above says & should be encoded as & ... which is what we do. not as &

[SeBsZ]

@julien51 the spec you listed says
either numeric character references or the strings &

So both & and & should be accepted

[archon810]

@julien51 @SeBsZ You're right that both should be accepted, but it would be ideal if you didn't change the content of our feed. If you see amp, leave it as amp, if you see 038, leave it as 038. That's the cause of the issue. I can't get any traction from dlvr.it to fix it, so I'm hoping you can make it happen.

[julien51]

@archon810 To be honest, at this point, even if we change it, it will take some time.
From there, you have 2 options:

• use an Atom feed or use another custom guid values
• Remove the PubSubHubbub plugin (this way won't ping Dlvr.it)

[archon810]

:(

Updated the GUID, but now I'm scared to see what happens and how many dupes we'll see pushed via Pushbullet and dlvr.it all over the place.

:S :S :S

[archon810]

Aaaand dlvr.it just barfed 10 of the latest posts all over the place. I can't believe they don't dedupe on the link too in addition to GUID.

[julien51]

Hum... make sure you unplug Dlvr.it before you change something in the feed first... wait a couple minutes/hours to make sure they 'see' the new feed version, and then, re-enable it.

[archon810]

Whoops, too late for that now :S