Feature request: Durability

[estaylorco]

I've just come across a situation where durability has moved to the forefront.

In Durandal, often we compose views. The activate handler of a viewmodel is the best place to set up postal subscriptions. Many times, however, a parent view will publish a message to one or more of its children, but the child, or children, isn't ready yet.

It would be nice if messages could be marked "durable" so that when a subscriber subscribes, postal checks to see if there are any messages waiting for that subscriber (on that subscriber's channel and topic). Of course, we would need to be able to configure a timeout. One could even use postal.request-response and its promise to get back a fail after a timeout period. Another option could be to configure a timeout message to publish on some hard-coded channel and topic.

I solved the problem temporarily with a client-side queue. But this seems ideal for incorporation into postal--maybe not into core, but certainly as, say, postal.durable.

[ifandelse]

Yep - definitely sounds like good plugin material. I've had a few situations where I've manually rolled something similar (all ugly, unfortunately). I'll try and take a stab at postal.durable soon, then :-)

[estaylorco]

@ifandelse Awesome! Thank you!

[ifandelse]

@estaylorco - quick question: do you see any messages marked to be preserved as being kept around (indefinitely or within a timeout window, if applicable) for any new subscribers on that channel/topic - or was your use case one where the message wasn't kept around any longer once a subscriber came into the picture?

[estaylorco]

@ifandelse My use case referred to any new subscriber on that channel and topic. I suppose if I had to narrow it down, I could use the constraints feature of postal.

[ifandelse]

Played around with a very rough idea around this tonight: http://jsfiddle.net/ifandelse/KDaLB/

[estaylorco]

@ifandelse I just went over the code: Brilliant in its simplicity! What I see, too, is that between wiretapping and subscribing after, the feature is essentially available now. Are wiretapping and after artifacts of your test harness, or is that how you intend to fold the feature into a plugin?

[estaylorco]

@ifandelse It just occurred to me: I think it necessary to make it clear at both the point of publishing and at the point of subscribing that we're engaging durability. The subscriber should have to acknowledge that it is interested in durable (preserved) messages.

So, for example, at the time of publishing, we have e.preserve--it's clear there. On the subscriber side, however, we should have a boolean along the lines of e.enlistPreserved. The reason for it is twofold:

1. A subscriber may wish to reject preserved messages under certain circumstances; and
2. The readability and maintainability of the code

On the latter note, processing of preserved messages becomes a silent feature of the application if we don't have e.enlistPreserved, or its like, on the subscriber's end. I just had this vision of having to chase down phantom problems arising from lingering preserved messages. In a large code base, your publishers could be in California and your subscribers could be in New York, so to speak, and it may not be obvious this tacit connection between publisher and subscriber.

If a subscriber does not enlist in preserved messages, postal falls back to its default behavior, a behavior everyone should be familiar with if he's adopted postal as part of his application's backbone.

[EDIT]

I would perhaps modify e.enlistPreserved from a boolean to a string with the following options: 'enlist', 'reject', and 'standby'. The former would have the same meaning I described above. 'reject' would indicate that preserved messages are being rejected categorically, and that the message(s) should be deleted from the store. The latter would tell postal to stop the clock on the timeout because the subscriber has acknowledged future interest in preserved messages--it's just not ready for them at the moment.

var mySub = postal.subscribe({
    channel: 'myChannel',
    topic: 'myTopic',
    enlistPreserved: 'standby',
    callback: function (d, e) { }
})

Then, later:

mySub.enlistPreserved(); //at which point the callback is invoked

This could even be handled with promises, where standby causes the return of a promise that is later resolved on the subscription.

Just some thoughts.

[ifandelse]

@estaylorco definitely agree it should be an opt-in thing on the part of the subscriber as well. To answer your earlier question "Are wiretapping and after artifacts of your test harness, or is that how you intend to fold the feature into a plugin?"

Wiretapping is supported already via postal.addWireTap, and it seems like the ideal place to listen for messages marked with particular flags, since any add-ons could tap there.

The after method comes from ConduitJS - which, as of v0.9.0, is an internal dependency to postal. I refactored all the SubscriptionDefinition configuration methods (like defer, withThrottle, withConstraint, etc.) to use Conduit on the subscription callback - which gives you a hook to add your own custom "pipeline steps" to run before or after a subscription callback fires. So, for example, the withConstraint method on the SubscriptionDefinition prototype is effectively adding a step underneath sort of like this:

  var pred = function (data, env) {
     return boolCondition;
 }
 var subscription = channel.subscribe('blah', callback)
     .before(function (next, data, envelope) {
         if (pred.call(this, data, envelope)) {
             next.call(this, data, envelope);
         }
     });

(The SusbcriptionDefinition.prototype.withConstraint method saves you from the above boilerplate and gains readability, but using Conduit to drive it means you can add custom steps now if needed.) Anyway - reason why I explain that is that ConduitJS can be used to target any method and make it a sync or async pipeline (depending on what you need). Which means we can use it to add steps that fire before or after any publish or subscribe. Useful - of course, can be dangerous if abused (but that's usually the case with anything).

I thought long and hard on this, since AOP is often a super painful footgun. :-) So far it seems to be working out well.

All that being said - I think I'd roll this with ConduitJS behavior being applied to postal.publish (which means a minor, non-breaking update to v0.9.0, since I'll probably make postal.publish a conduit method by default. I might go ahead and promote ConduitJS to a real dependency (and remove it from being embedded in postal's lib), so that add-ons that utilize it don't have to source yet another copy of it, etc. I've avoided doing this before b/c some devs have seemed quite resistant to libs with deps, but with npm and bower pulling deps down automatically, this seems like a straw man argument, IMO.

Once I do that, I should be able to roll the add-on with the behavior we've discussed pretty easily... (famous last words?)

[estaylorco]

@ifandelse You know, when I read the code on your jsFiddle, I noticed Conduit.Sync and wondered what that was. But I haven't had time this morning to explore it.

AOP makes so much sense here. I grew up in web technologies using Spring on the Java side. We embrace AOP as a way of life there. I was actually looking recently for a client-side AOP framework for JavaScript. I was unaware of Conduit, or perhaps I was aware but just didn't associate it with AOP. Well-written and well understood AOP is incredibly powerful.

It should be fine with anybody to depend on low-level frameworks. It just means that, as a dev, you have to "own" those frameworks, in a manner of speaking, make sure you understand the codebase, contribute to that framework's community and ecosystem.

As for high-level frameworks, I don't know. Where's the fun? I use Durandal precisely because it is low-level and gets out of my way when I need it to. To each his own, I'd say.

[estaylorco]

@ifandelse Just some additional thoughts on the notion of 'standby'. Below is a use case that might help to understand how standby would influence a client of postal.durable (activate, attached, and compositionComplete are all lifecycle events on a Durandal viewmodel, which you can hook into):

var activate = function (activationData) {
    this.registerSubscriptions();

    //Prepare the viewmodel
}

var attached = function (view) {
    //Prepare the viewmodel further
}

var compositionComplete = function (child, parent, settings) {
    //Finish preparing the viewmodel

    //Now we can enlist the durables
    var enlistedSubscribers = getEnlistedSubscribersFor( {context: this} );
    _.each(enlistedSubscribers, function (sub) {
        sub.enlistPreserved();
    });
}

getEnlistedSubscribersFor, which is just sugar on top of postal's [new] getSubscribersFor, would return an array of all queried subscribers having enlistPreserved === 'standby'.

The ability to stand by allows proper separation of concerns. I'm free to register subscriptions that are both durable and non-durable, or in the case of the former, whose interest in preserved messages is immediate, rejected, or standby, in a single method: registerSubscriptions. This is what I typically do in my activate handlers.

Then, I can enlist when I'm ready (for me, it would be in the compositionComplete handlers). Without standby, I'm forced to break out the durables and move them around my code artificially to defend against the immediacy of their processing.

Finally, I see a potential problem with using promises in this case. We go back to having to hold on to subscriptions through specific variables, or through an array-push, which is what I was advocating getting away from with my request for additional utils, just so we can resolve or reject the promises.

[ifandelse]

Still digesting the last bit above - I took the fiddle and made it into a fledgling project: https://github.com/postaljs/postal.preserve. I'll think through the points you made about "standby" above and see if I come with anything that works well...

[estaylorco]

@ifandelse That looks great! Simple. I like the use of headers to convey durability. This takes us very close to the Durable Subscriber pattern.

It would seem at this point that you're asking the subscriber to always enlist if the subscriber is interested. That might actually be a better approach. Going back to my post on 'enlist', 'reject' and 'standy', through the single approach you illustrate in your fiddle, we have the following:

enlistPreserved	effect
Call on subscribe	enlist immediately
Never call	reject (preserved messages will just expire)
Call eventually	standby (enlist when called)

The second item above is self-explanatory: Never calling enlistPreserved is equivalent to rejecting preserved messages. As for the first and third items, the effect depends on when enlistPreserved is called: immediately upon subscribing, or later elsewhere in the code.

Finally, expired messages should be deleted from the store.