Design a Publish/Subscribe System (aka Futures)

[linas]

Various processing subsystems take certain atoms as inputs, and generate other atoms as outputs. Other subsystems need to monitor or get access to the newly created atoms. A generic infrastructure is needed for this.

Currently we have two systems for this, and both are inadequate, bordering on terrible. They are:

1. Atomspace signals
2. AnchorNode

There are many problems with 1.:

• One is notified of all atoms, insted of just the ones that are interesting to you.
• The notification is synchronous, which means that the entire atomspace, and all users of the atomspace, must wait (viz are blocked) as each subscriber does it's thing.
• An addAtomSignal() subscriber risks blocking itself (and hanging the entire system) it it adds an atom itself...
• The performance of the entire atomspace is degraded by these signals.

For these reasons, I really want to get rid to the atomspace signals. There are almost no users of them, so getting rid of them is not hard. (See issue #1745)

There are many problems with 2. The typical usage of 2. is that one system (the publisher) creates a link (Member (Anchor "foo") (SomeAtom)) while a second subsystem (the subscriber) monitors the incoming set for (Anchor "foo") then grabs (SomeAtom) then deletes the MemberLink. The problem with this is that:

• There can only ever be one subscriber per publisher.
• The subscriber must busy-poll the anchor-point; it has no way of being notified.
• There's probably (maybe???) not much point in adding the MemberLink to the atomspace, since its going to be deleted a while later.

The good things about 2. are:

• Its async; different threads can work at the same time.
• Its local: only the interested parties are involved, and the atomspace as a whole is not dragged down.

A well-designed publish-subscribe system should then allow a futures-and-promises system to be implemented, for users of the pattern matcher, after implemented #1502. That is, users of the pattern matcher can subscribe, wait for results, and mangle them when ready.

This is related to the idea of "promises" and "futures"

[linas]

Proposed solution (there may be several; here's one) -- Modify the AnchorNode idea to make it more general.

• Instead of creating MemberLinks on the anchor, place the published atoms into some value on that atom. This is MUCH MUCH faster than creating a MemberLink, and sticking it in the atomspace.
• Create a C++ class for the AnchorNode; the C++ class would deliver synchronous signals to all subscribers for that particular AnchorNode (or maybe Achor+the key for the value). Th signal handlers would trigger assorted scheme code to run.
• By default, the value is cleared, after the signals are delivered.

Here's an even better variant:

• Create a new QueueValue that is just like the LinkValue except that it is a queue: one can push Atoms onto one end, and pop them off the other end. That way, the publisher is not blocked by the subscribers.

The nice thing about this system is that the signals live with the AnchorNode, instead of living in the atomspace, and/or all-possible atoms. Subscribers are inherently local.

[linas]

FYI pull req #2316 proposes the creation of a QueueValue which would be a single-producer, single-consumer system for process results; in that case, the producer was the pattern matcher, and the consumer was a variant of the PutLink.