Backwards compatibility with PubSubHubbub 0.3

[evanp]

So, I'd like to call this out as a separate issue.

Compatibility between 0.3 and 0.4 clients is extremely important to me. We have tens of thousands of installations of StatusNet on the Web, and it's not easy for us to upgrade them all at once. I don't want our interconnection to end when sites supporting 0.4 go online.

I realize that the vast majority of feeds running PubSubHubbub are running on one of two hubs, but there are also clients that support 0.3 and lots of smaller hubs.

[apparentlymart]

I wonder if we can mitigate this problem during a transitional phase by adding some smarts to 0.4-compliant hubs so that they can recognize a requests from 0.3 clients and then behave in a 0.3-compatible manner when talking to those clients.

This compromise would free us to make more changes to the protocol to support the new features we want to add while putting the burden on hubs (which, as you say, there are relatively few of) to support the legacy clients for a certain period while we transition.

[evanp]

Well, we've got about 50,000 hubs on the status.net cloud, probably some high number on other StatusNet sites.

I guess my feeling is that backward compatibility is relatively easy to keep around, if we're willing to make it a priority.

[apparentlymart]

Okay; you use of "clients" in your initial description mislead me into thinking that StatusNet was only a client, not also a hub.

So I retract my earlier suggestion, but I think we also need to decide carefully whether we'll achieve compatibility by making the new protocol wire-compatible with the old, or whether it's reasonable to specify a "compatibility mode" that allows old clients to send/receive old-style messages while allowing us to break compatibility in the new protocol where both participants are 0.4-capable.

I worry about this because the assumption that everything would be Atom/RSS led to several design decisions that are troublesome in an arbitrary content world, a couple of which I've already opened separate issues for. I'd rather reach a simple design that completely addresses notifications of arbitrary resources in a natural way that jives well with HTTP's basic principles than try to shoe-horn arbitrary content support into a protocol that was designed around only supporting Atom/RSS deltas.

[evanp]

I just don't think that with literally hundreds of millions of feeds pubsubhubbub-enabled[1] we have the latitude to re-design this protocol from the ground up.

I think it's slightly harder, but not impossible, to support arbitrary data types without losing backwards compatibility. For example:

• Hub Discovery. Subscribers should do a HEAD to get the Link headers. If they're absent, but the content-type is Atom or RSS, the client should do a GET to get the full body, check for Link headers again (what the hell, why not?), and failing that parse the resulting Atom or RSS for atom:link headers. Publishers should provide the Link headers for HEAD and GET, and also for Atom or RSS payloads should include the atom:link as a fallback.
• Subscription Retain "sync" mode subscriptions, but disallow it for flows that require async (like authenticated subscription).
• Publishing Retain the description of diffs for Atom and RSS.

[1] Just count Blogger and Tumblr, not to mention WordPress.com and Feedburner!

[apparentlymart]

That special case you mentioned under "Hub Discovery" ("assume it's Atom/RSS if there's no link header") is exactly the sort of thing I meant by a "compatibility mode": language in the spec that says "if some certain condition is true, act like 0.3, otherwise, do this new thing".

[voxpelli]

Is there something in the 0.4 proposal that makes it impossible for a site to support both 0.3 and 0.4 at the same time? As long as 0.4 doesn't specifically prohibit 0.3 behavior then wouldn't a site be free to support both versions in parallel? And it's not something that 0.4 have to specifically allow for it to be possible? Or am I missing something?

Some of the interoperability challenges in OStatus could probably be handled in a new version of OStatus - the current 1.0 draft already indirectly says that it's PuSH 0.3 that should be used. An OStatus 1.1 could eg. define that OStatus sites needs to be able to talk both PuSH 0.3 and PuSH 0.4?

[evanp]

@apparentlymart Actually, I meant that you should check the Content-Type and continue with discovery if there's no Link header.

But I think my main point is that we should include compatibility issues in the spec, rather than using rules of thumb or folk wisdom.

[evanp]

@voxpelli I think it would be tricky, but it might be possible. Realistically, I don't think the improvements in 0.4 require incompatibility. There's no reason to buy the hassle of incompatibility.