Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for running external shell tasks #3461

Open
wants to merge 25 commits into
base: master
from
Open

Conversation

@Jermolene
Copy link
Owner

@Jermolene Jermolene commented Oct 1, 2018

It can be difficult for experienced developers to become proficient in working with TiddlyWiki's code: besides requiring a good understanding of JavaScript, the internals are built around relatively unusual concepts and techniques that are likely to be unfamiliar to most.

However, many software developers are very familiar with using and creating text-based command line tools that adhere to the Unix philosophy of using stdin/stdout for input and output, allowing tools to be chained together in powerful ways.

The proposed external task mechanism allows Unix-style tasks to be registered and executed by TiddlyWiki under Node.js. Execution can be triggered via a server command or via a message propagated from the browser.

The initial functionality is to allow an external task to be invoked synchronously, piping tiddler text to it via stdin, and reading text from it via stdout.

@inmysocks I know you've got something similar in Bob, I'd welcome your thoughts.

@inmysocks
Copy link
Contributor

@inmysocks inmysocks commented Oct 3, 2018

The details are different between the two approaches, but that seems to mainly be from the difference between the tiddlywiki commands and how I have the websockets set up. Also I didn't consider capturing the output in a tiddler. I think that the focus of our approaches is very different, mine came from wanting to be able to easily interact with physical systems for things like my robot or a home automation system where this seems to be for altering the wiki itself. I don't think that the two approaches are incompatible.

The only functional differences that I can see between the two approaches is that I have some queuing set up so that you can run scripts in sequence or run multiple scripts/queues asynchronously.
I think that would be a nice addition here but it is outside the scope of the current commits.

I don't know what the plan is for being able to trigger the execution from the browser. I have been trying to find a way to integrate the websockets I use with Bob into the normal server but I can't find a consistent way to allow both the single-wiki focused core server and the multi-wiki Bob server.

As an alternative to websockets I think that adding a REST-type api to run the commands would be relatively simple to implement and probably much more in line with what people expect from something like this.

@sukima
Copy link
Contributor

@sukima sukima commented Oct 3, 2018

Is this not the same as placing a node only plugin that opens a stream/pipeline?

@Jermolene
Copy link
Owner Author

@Jermolene Jermolene commented Oct 3, 2018

The details are different between the two approaches, but that seems to mainly be from the difference between the tiddlywiki commands and how I have the websockets set up. Also I didn't consider capturing the output in a tiddler. I think that the focus of our approaches is very different, mine came from wanting to be able to easily interact with physical systems for things like my robot or a home automation system where this seems to be for altering the wiki itself. I don't think that the two approaches are incompatible.

The motivation for the work was to enable developers with Unix-y skills to be quickly productive with TiddlyWiki.

The only functional differences that I can see between the two approaches is that I have some queuing set up so that you can run scripts in sequence or run multiple scripts/queues asynchronously.
I think that would be a nice addition here but it is outside the scope of the current commits.

Yup, I'm interested in experimenting with more ways to trigger tasks: as a cron job, via a widget message etc. Queuing tasks could definitely useful too.

I don't know what the plan is for being able to trigger the execution from the browser. I have been trying to find a way to integrate the websockets I use with Bob into the normal server but I can't find a consistent way to allow both the single-wiki focused core server and the multi-wiki Bob server.

I'm thinking we'd have an HTTP POST endpoint for triggering a task, and eventually a means to check on the progress of tasks via HTTP.

As an alternative to websockets I think that adding a REST-type api to run the commands would be relatively simple to implement and probably much more in line with what people expect from something like this.

As I think we've discussed, I'm still interested in exploring using Server Sent Events for basic signalling between server and client.

@inmysocks
Copy link
Contributor

@inmysocks inmysocks commented Oct 3, 2018

As I am now very annoyed with my other projects I will try making a plugin to test out server sent events for adding tiddlers to a wiki.

@Jermolene
Copy link
Owner Author

@Jermolene Jermolene commented Oct 3, 2018

Thanks @inmysocks!

@inmysocks
Copy link
Contributor

@inmysocks inmysocks commented Oct 3, 2018

Ok, I have a prototype for an extensible implementation of server sent events. I am happy with the browser side but I think that the server side needs some work. @Jermolene I am hoping that you have some insight about where to put the components.
I packaged a demo as a plugin and put how to use it and my concerns in the readme here:
https://github.com/OokTech/TW5-ServerSentEvents

@Jermolene
Copy link
Owner Author

@Jermolene Jermolene commented Oct 11, 2018

Hi @inmysocks thank you, that's really helpful.

I think that the most valuable, generic application of SSEs is for real time propogation of tiddler changes from the server to connected browsers. I'd be inclined to wait until that point before we integrate with the core. It strikes me that doing this is going to require us to properly think through TW5's sync architecture. In particular, I think we need to switch from syncing the values of tiddlers to syncing their individual edit history. This (aka operational transforms) is a technique that distributed editing systems have been using for a decade, and I think it's time we bought it into TW5.

Do you think it's worth putting in some hooks in the meantime so that experiments like this can be done without needing to hack server.js?

@inmysocks
Copy link
Contributor

@inmysocks inmysocks commented Oct 11, 2018

@Jermolene Yes, I think that having a way to add server routes outside of server.js would be a huge help. I haven't taken a good look at the new architecture for the server in the upcoming version so I am not sure how much work it would take.

Between the SSE prototype and Bob I have almost all of the components needed to make a system for propagating changes from the server to browser without requiring external npm modules. I have been working on the file system monitoring part and I think that I am close to making it independent of the messaging channel between the server and browser. The biggest problem I have for converting Bobs functionality to work with the core is handling multiple wikis. I am in the process of making Bob use different $tw.Wiki objects for each wiki and I think that I can make it work with one or many wikis, but there may be some changes to how $tw.boot.files works and I have the feeling that is a problem.

I agree about syncing the edit history, luckily that changes very little about the communication model I have been using because the transport is independent of the actions on either end. I haven't done much research into operational transforms but my brother has done some and has some ideas about implementing simple versions for tiddlywiki. I don't know if he made any progress with it yet or not.

Jermolene added 11 commits Oct 13, 2018
And add support for connecting via sockets to existing processers/tasks
Makes it possible to invoke any of TW5's Node.js commands from the browser. Useful for the new pipe command, but also handy for e.g. rendering tiddlers on the server, fetching remote files, or importing local files etc

Docs to come.
We were referencing it in the tw5.com edition, but that makes it harder to use the externalpipesdemo edition independently.
@BurningTreeC
Copy link
Contributor

@BurningTreeC BurningTreeC commented Oct 19, 2018

I love it, merge merge merge 👍 😁

@BurningTreeC BurningTreeC mentioned this pull request Nov 24, 2018
@Jermolene Jermolene added the v5.1.20 label Nov 29, 2018
@Arlen22
Copy link
Contributor

@Arlen22 Arlen22 commented Dec 19, 2018

Very curious about server sent events. It seems like this should be implemented as a simple route tiddler with a stream body type (which causes the router to immediately hand the request off to the specified route without waiting for the body to come in). I definitely had SSE in mind when I added that part.

exports.bodyFormat = "stream";

@Jermolene
Copy link
Owner Author

@Jermolene Jermolene commented Dec 19, 2018

Hi @Arlen22 did you see @inmysocks prototype?
#3461 (comment)

@BurningTreeC
Copy link
Contributor

@BurningTreeC BurningTreeC commented Jul 4, 2019

ping

@inmysocks
Copy link
Contributor

@inmysocks inmysocks commented Jul 4, 2019

The updates to how the server components work may make this in a simple way. But I don't think it should be part of the next release because of how much testing is needed already.

@Jermolene
Copy link
Owner Author

@Jermolene Jermolene commented Jul 4, 2019

Hi @inmysocks @BurningTreeC I agree let's merge this once v5.1.20 is out.

@Jermolene Jermolene added post-v5.1.20 and removed v5.1.20 labels Jul 4, 2019
@Arlen22
Copy link
Contributor

@Arlen22 Arlen22 commented Jul 19, 2019

Hi @Arlen22 did you see @inmysocks prototype?
#3461 (comment)

I don't remember, but I think that the server-events should be setup using a route tiddler with the stream body type. This should answer the use-case perfectly. I also think that all server-sent events should have a known mount point, such as /events and then each module can pick off the connections they want to send to. In other words, there would be one server mount point which handles all requests starting with /events.

I think the best way is to add a serverEvents property to the $tw variable. It would contain an isServer property which would be set according to which side this is running on to make things simpler. This would be done with module-type: global. The route handler would call methods on this function.

https://github.com/OokTech/TW5-ServerSentEvents lists several concerns, which are good points. Here is my point-by-point response, which is mostly summarized above.

  1. Put it in a route tiddler with body type stream.
  2. The route tiddler should notify all listeners of a new connection in a way that would allow for seamless control and messaging. One possibility is to create a Connection class which would contain all the important information for each connection. Handlers would pick up the connections that apply to them, and since the messaging is only from server to client it would not matter if multiple modules are using the same connection as long as they are not using the same event names for a different purpose, which would create confusion. This should be easy to avoid.
  3. You can address any individual connection you like using this method, but the only way to associate a client request with an SSE connection is through cookies. If done correctly this can make it easier to exclude the client that caused a server refresh, for instance, although this is usually not necessary. So I don't know if it matters.
  4. This is a very interesting question, because we could go a lot of directions with this. On the server-side we could easily use this to create namespaces for events, such as the plugin author. So core messages would use tiddlywiki as the event type, plugins I make (if I ever do) would use arlen22, and this could be divided even further by using a slash or other delimiter (e.g. tiddlywiki/sync). But as mentioned this would prevent us from using the hooks mechanism on the client-side. However, when discussing websockets, Jeremy said that the hooks mechanism should be used for lower-level functions, such as startup hooks, and we should use a global variable for things such as messages, so that basically takes care of all the negatives.

Footnote 1: I think we should have a way of classifying plugins according to their compatibility with established protocols, to make it easy for users and plugin authors to ensure cross-compatibility.

@shonin
Copy link

@shonin shonin commented Apr 23, 2020

I want to flag this PR as a desired feature. Would love to see it merged!

@v3470
Copy link

@v3470 v3470 commented May 3, 2020

May I asked a question about the final URL in the video "Joe Armstrong & Jeremy Ruston - Intertwingling the Tiddlywiki with Erlang | Code Mesh LDN 18"?
in the video the URL is https://jermolene.com/intertwingled but it was 404, while https://jermolene.com/intertwingling is works. is that "intertwingled" a typo of "intertwingling"? @Jermolene

@Arlen22
Copy link
Contributor

@Arlen22 Arlen22 commented May 10, 2020

It would probably work better to post this on the TiddlyWiki Google Group.

@joshuafontany
Copy link
Contributor

@joshuafontany joshuafontany commented Jun 20, 2020

Bump. This would be incredibly useful. I'm going to try to implement calling "oEmbed" providers from the TW server and storing their "embed" response as a JSON tiddler in the wiki.

@Jermolene
Copy link
Owner Author

@Jermolene Jermolene commented Jan 15, 2021

The discussion at #5402 reminded me that we should try to get this done. The new Server Sent Events stuff gives us some new possibilities too.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

8 participants