Improvements Brainstorm

How can Twine be more Dev Friendly?

Currently, it's quite a lousy experience. Some gripes:

• Coming up with service and command IDs, and keeping them in sync on both sides
• Knowing what to do with messages received (Do I need to reply with payload? Is OK/Error enough? Etc...
• General messiness of the whole setup. Lots of "handleMessage" with switch cases, etc... A more typical system would look more like a HTTP router.
• Not sure if this is the result of all the messiness above, but I get confused between the two "sides".
• The lack of visibility into Twine makes it harder to use.

Replace Command and Service IDs

I've thought that ids should be replaced with strings that completely identify the destination, like:

host/appspace-status/subscribe-id

The receiving side needs to register all their "routes" before they get called, but I don't think that's a problem.

twineClient.on("host/appspace-status/subscribe-id", function(){} )

(I wonder if having these "destination keys" means we can rethink how we generate ids? Like we could have a separate set of ids for each key? and reference the key + message_id?)

While the API uses strings, the protocol should turn these strings into ids using some clever technique. Like maybe on first use of the key, pass an id that tells the other side this is how this string will be referenced from now on.

Hint At Handling With Strong Types

When creating a message, each different method (Send, SendBlock, ...) should return a message (if applicable) that has all the affordances that are appropirate, and none that are not.

We can push this further, by encoding in the message the expectations. That way the message delivered on the other side would have the types expected, and the receiver would know what to do. (Although, the only way to do this is for teh receiver to listen in a manner similar to sending:

t.onSentBlocked...

Or something like that. Maybe with some thought I can come up with proper names.

The idea is both sides should agree so that things fail fast if there is a mismatch.

Pre-register all messages with details?

If sending and receiving messages is going to get more specific (as noted above), and if message receivers must register their listeners, maybe it makes sense to register sending keys? Or is that too burdensome?

One advantage would be runtime checks could pass/fail immediately. Each side could register their entire understanding of the messages they expect, their types, replies, etc... and if there is a mismatch anywhere then it's a fail. Teh protocol could support this "debug" mode by exchanging the expectations (registrations) of both sides and comparing.

However I worry that it would be burdensome, because it likely duplicates work (have to register, then have to actually listen / send).

Properties for Handling Messages

• Replies:
  • Expect a payload reply
  • Expect OK/Error reply
  • Expect no reply (twine protocol automatically replies OK so sender can close)
• Expect Stay open / close fast? timeout?
  • (No, timeouts will cause problems. Instead just give visibility so devs can see unclosed messages pile up)
• Use as ref:
  • Expect ref sends from sender
  • Expect ref send from receiver
• Expect single use of key
  • (yes, but probably at the message receiver side, and probably only for ref messages, like await m.on("done").once() and for await ( const refM of m.on("new-data").gen() ))

Edit: I don't think the above properties of comms should be sent back and forth in the protocol. Both sides have to be aware of expectations anyways, and the idea is both sides of a message are developed together. So these are things for the dev to think about, not for the protocol to enforce.

Give Visibility Into Twine

Basically it should be possible to connect to twine (on one or both sides simultaneously) and see the messages, their state, etc...

The easiest way to implement this right now:

• each implementation can generate a list of messages and states (including closed messages? see below) and package them into a common format, probably JSON.
• Each implementation can return this data using whatever transport makes sense:
  • Go implementation: HTTP server
  • Deno: HTTP?
  • web: postmessage? or..? Can we try to make a devtools extension?
• Have a frontend that can connect to any of these sources (how? CORS?)
  • frontend should be able to display sent/received data in chronological order, or related like replies together, and refs could be together (tree form) etc...
• May need a "record" mode that keeps all messages (even after closed), maybe with the ability to start/stop recording?
• Do this all without using Twine itself.

Improve Library Interface

For sending, we currently we have:

• client.send (and message.refSend)
• client.sendBlock (and message.refSendBlock)
• message.sendOK
• message.sendError
• message.reply

First, rename sendOK and sendError to replyOK and replyError.

Also I think send should return immediately. No await or anythign such thing. The comm gets sent after the call returns.

Would like to add a sendForget (and refSendForget). This would allow sending data and not concerning yourself with a response. Some thoughts:

• if a reply to a sendForget is received that is not OK or Error then it logs an error and library automatically sends a replyError
• if an error is received in reply to sendForget then it is logged by the twine client (logging facility tbd and library-dependent).
• sendForget should return immediately, like send.
• So what is the difference between send where caller ignores the sentMessage, and sendForget? It might be that with send, not paying attention to the response triggers an error? Not much on this yet, but I'm aiming to make errors of that type fail early, so if you want to ignore, you have to be explicit about it. Hence the "forget".

On the receiving side:

(Looking at twine-web) we currently have:

• client.registerService (expects an interface with handleMessage)
• sent.waitReply
• message.incomingMessages (async generator for all incoming ref messages)
• message.ok and messgae.error

Lots here. Would like to move to message.on format for receiving all the things. For example:

• client.on('destination key', ... ) ... for top level receives
• sent.waitOKError returns error or undefined if OK. Or maybe it's onOKError. It's for when you expect OK or Error only.
• sent.waitReply or onReply. Should be checked for error before use.
• message.on('sub key', ... ) ... for refs
• message.onSingle('sub key') :Promise<message> when expecting a single message. That way it can be waited for with await.

Not clear exactly what the functions that return multiple messages return? We could use an async generator again, or just use the callback mechanism. Just be consistent: client.on and message.on should have the same interface.

Trigger Errors When expected thing isn't happening

To prevent "sleeper" bugs, I'd like for Twine to throw or return errors when something amiss is happening. Some scenarios:

• Side A called client.send(), but reply is unhandled. When a reply arrives from side B, if no handlers are registered then have to put an error somewhere.
  • Side B returned a payload, but only OK/Error are being handled. Same as above really.
  • Unhandled OK/error when sent as third comm. Same as above.
• X sends non-OK/Error, recipient has properly registered handler, but does not reply.
  • is there any way to do this? Without timeouts? Any timeout may cause more problems (like imagine parsing giant doc upon receipt or whatever).
  • Devs should return OK/Error promptly, but to enforce that seems like a prickly path.
  • visibility into open messages is the answer here.