Add support for running external shell tasks

[Jermolene]

It can be difficult for experienced developers to become proficient in working with TiddlyWiki's code: besides requiring a good understanding of JavaScript, the internals are built around relatively unusual concepts and techniques that are likely to be unfamiliar to most.

However, many software developers are very familiar with using and creating text-based command line tools that adhere to the Unix philosophy of using stdin/stdout for input and output, allowing tools to be chained together in powerful ways.

The proposed external task mechanism allows Unix-style tasks to be registered and executed by TiddlyWiki under Node.js. Execution can be triggered via a server command or via a message propagated from the browser.

The initial functionality is to allow an external task to be invoked synchronously, piping tiddler text to it via stdin, and reading text from it via stdout.

@inmysocks I know you've got something similar in Bob, I'd welcome your thoughts.

---

Edit: 27th October 2021

This PR was originally created in collaboration with the late @joearms in preparation for a talk in November 2018:

• Notes on the talk: https://jermolene.com/intertwingling
• Video: https://www.youtube.com/watch?v=Uv1UfLPK7_Q

[inmysocks]

The details are different between the two approaches, but that seems to mainly be from the difference between the tiddlywiki commands and how I have the websockets set up. Also I didn't consider capturing the output in a tiddler. I think that the focus of our approaches is very different, mine came from wanting to be able to easily interact with physical systems for things like my robot or a home automation system where this seems to be for altering the wiki itself. I don't think that the two approaches are incompatible.

The only functional differences that I can see between the two approaches is that I have some queuing set up so that you can run scripts in sequence or run multiple scripts/queues asynchronously.
I think that would be a nice addition here but it is outside the scope of the current commits.

I don't know what the plan is for being able to trigger the execution from the browser. I have been trying to find a way to integrate the websockets I use with Bob into the normal server but I can't find a consistent way to allow both the single-wiki focused core server and the multi-wiki Bob server.

As an alternative to websockets I think that adding a REST-type api to run the commands would be relatively simple to implement and probably much more in line with what people expect from something like this.

[sukima]

Is this not the same as placing a node only plugin that opens a stream/pipeline?

[Jermolene]

The details are different between the two approaches, but that seems to mainly be from the difference between the tiddlywiki commands and how I have the websockets set up. Also I didn't consider capturing the output in a tiddler. I think that the focus of our approaches is very different, mine came from wanting to be able to easily interact with physical systems for things like my robot or a home automation system where this seems to be for altering the wiki itself. I don't think that the two approaches are incompatible.

The motivation for the work was to enable developers with Unix-y skills to be quickly productive with TiddlyWiki.

The only functional differences that I can see between the two approaches is that I have some queuing set up so that you can run scripts in sequence or run multiple scripts/queues asynchronously.
I think that would be a nice addition here but it is outside the scope of the current commits.

Yup, I'm interested in experimenting with more ways to trigger tasks: as a cron job, via a widget message etc. Queuing tasks could definitely useful too.

I don't know what the plan is for being able to trigger the execution from the browser. I have been trying to find a way to integrate the websockets I use with Bob into the normal server but I can't find a consistent way to allow both the single-wiki focused core server and the multi-wiki Bob server.

I'm thinking we'd have an HTTP POST endpoint for triggering a task, and eventually a means to check on the progress of tasks via HTTP.

As an alternative to websockets I think that adding a REST-type api to run the commands would be relatively simple to implement and probably much more in line with what people expect from something like this.

As I think we've discussed, I'm still interested in exploring using Server Sent Events for basic signalling between server and client.

[inmysocks]

As I am now very annoyed with my other projects I will try making a plugin to test out server sent events for adding tiddlers to a wiki.

[Jermolene]

Thanks @inmysocks!

[inmysocks]

Ok, I have a prototype for an extensible implementation of server sent events. I am happy with the browser side but I think that the server side needs some work. @Jermolene I am hoping that you have some insight about where to put the components.
I packaged a demo as a plugin and put how to use it and my concerns in the readme here:
https://github.com/OokTech/TW5-ServerSentEvents

[Jermolene]

Hi @inmysocks thank you, that's really helpful.

I think that the most valuable, generic application of SSEs is for real time propogation of tiddler changes from the server to connected browsers. I'd be inclined to wait until that point before we integrate with the core. It strikes me that doing this is going to require us to properly think through TW5's sync architecture. In particular, I think we need to switch from syncing the values of tiddlers to syncing their individual edit history. This (aka operational transforms) is a technique that distributed editing systems have been using for a decade, and I think it's time we bought it into TW5.

Do you think it's worth putting in some hooks in the meantime so that experiments like this can be done without needing to hack server.js?

[inmysocks]

@Jermolene Yes, I think that having a way to add server routes outside of server.js would be a huge help. I haven't taken a good look at the new architecture for the server in the upcoming version so I am not sure how much work it would take.

Between the SSE prototype and Bob I have almost all of the components needed to make a system for propagating changes from the server to browser without requiring external npm modules. I have been working on the file system monitoring part and I think that I am close to making it independent of the messaging channel between the server and browser. The biggest problem I have for converting Bobs functionality to work with the core is handling multiple wikis. I am in the process of making Bob use different $tw.Wiki objects for each wiki and I think that I can make it work with one or many wikis, but there may be some changes to how $tw.boot.files works and I have the feeling that is a problem.

I agree about syncing the edit history, luckily that changes very little about the communication model I have been using because the transport is independent of the actions on either end. I haven't done much research into operational transforms but my brother has done some and has some ideas about implementing simple versions for tiddlywiki. I don't know if he made any progress with it yet or not.

[BurningTreeC]

I love it, merge merge merge 👍 😁

[Arlen22]

Very curious about server sent events. It seems like this should be implemented as a simple route tiddler with a stream body type (which causes the router to immediately hand the request off to the specified route without waiting for the body to come in). I definitely had SSE in mind when I added that part.

exports.bodyFormat = "stream";

[Jermolene]

Hi @Arlen22 did you see @inmysocks prototype?
#3461 (comment)

[BurningTreeC]

ping

[inmysocks]

The updates to how the server components work may make this in a simple way. But I don't think it should be part of the next release because of how much testing is needed already.

[Jermolene]

Hi @inmysocks @BurningTreeC I agree let's merge this once v5.1.20 is out.

[Arlen22]

Hi @Arlen22 did you see @inmysocks prototype?
#3461 (comment)

I don't remember, but I think that the server-events should be setup using a route tiddler with the stream body type. This should answer the use-case perfectly. I also think that all server-sent events should have a known mount point, such as /events and then each module can pick off the connections they want to send to. In other words, there would be one server mount point which handles all requests starting with /events.

I think the best way is to add a serverEvents property to the $tw variable. It would contain an isServer property which would be set according to which side this is running on to make things simpler. This would be done with module-type: global. The route handler would call methods on this function.

https://github.com/OokTech/TW5-ServerSentEvents lists several concerns, which are good points. Here is my point-by-point response, which is mostly summarized above.

1. Put it in a route tiddler with body type stream.
2. The route tiddler should notify all listeners of a new connection in a way that would allow for seamless control and messaging. One possibility is to create a Connection class which would contain all the important information for each connection. Handlers would pick up the connections that apply to them, and since the messaging is only from server to client it would not matter if multiple modules are using the same connection as long as they are not using the same event names for a different purpose, which would create confusion. This should be easy to avoid.
3. You can address any individual connection you like using this method, but the only way to associate a client request with an SSE connection is through cookies. If done correctly this can make it easier to exclude the client that caused a server refresh, for instance, although this is usually not necessary. So I don't know if it matters.
4. This is a very interesting question, because we could go a lot of directions with this. On the server-side we could easily use this to create namespaces for events, such as the plugin author. So core messages would use tiddlywiki as the event type, plugins I make (if I ever do) would use arlen22, and this could be divided even further by using a slash or other delimiter (e.g. tiddlywiki/sync). But as mentioned this would prevent us from using the hooks mechanism on the client-side. However, when discussing websockets, Jeremy said that the hooks mechanism should be used for lower-level functions, such as startup hooks, and we should use a global variable for things such as messages, so that basically takes care of all the negatives.

Footnote 1: I think we should have a way of classifying plugins according to their compatibility with established protocols, to make it easy for users and plugin authors to ensure cross-compatibility.

[rykener]

I want to flag this PR as a desired feature. Would love to see it merged!

[sv158]

May I asked a question about the final URL in the video "Joe Armstrong & Jeremy Ruston - Intertwingling the Tiddlywiki with Erlang | Code Mesh LDN 18"?
in the video the URL is https://jermolene.com/intertwingled but it was 404, while https://jermolene.com/intertwingling is works. is that "intertwingled" a typo of "intertwingling"? @Jermolene

[Arlen22]

It would probably work better to post this on the TiddlyWiki Google Group.

[joshuafontany]

Bump. This would be incredibly useful. I'm going to try to implement calling "oEmbed" providers from the TW server and storing their "embed" response as a JSON tiddler in the wiki.

[Jermolene]

The discussion at #5402 reminded me that we should try to get this done. The new Server Sent Events stuff gives us some new possibilities too.

[Jermolene]

A few thoughts:

• The remote commands functionality is pretty separate from the pipes functionality, and so might be easier to handle in its own PR
• The pipes functionality is somewhat experimental, so I wonder if it might be better off packaged as a core plugin

[linonetwo]

I've been using https://github.com/tiddly-gittly/zx-script for 3 years (since tiddly-gittly/tiddlywiki-plugins@9e8080c), executing shell command on Tiddlywiki, especially JS powered shell command (zx script) is very satisifing.

I come here because I find this (the pipes part) might be the infra to build the "one-directional way to sync indexer from server to client" described in flibbles/tw5-relink#52 (comment)