New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for running external shell tasks #3461

Open
wants to merge 21 commits into
base: master
from

Conversation

Projects
None yet
4 participants
@Jermolene
Owner

Jermolene commented Oct 1, 2018

It can be difficult for experienced developers to become proficient in working with TiddlyWiki's code: besides requiring a good understanding of JavaScript, the internals are built around relatively unusual concepts and techniques that are likely to be unfamiliar to most.

However, many software developers are very familiar with using and creating text-based command line tools that adhere to the Unix philosophy of using stdin/stdout for input and output, allowing tools to be chained together in powerful ways.

The proposed external task mechanism allows Unix-style tasks to be registered and executed by TiddlyWiki under Node.js. Execution can be triggered via a server command or via a message propagated from the browser.

The initial functionality is to allow an external task to be invoked synchronously, piping tiddler text to it via stdin, and reading text from it via stdout.

@inmysocks I know you've got something similar in Bob, I'd welcome your thoughts.

@inmysocks

This comment has been minimized.

Contributor

inmysocks commented Oct 3, 2018

The details are different between the two approaches, but that seems to mainly be from the difference between the tiddlywiki commands and how I have the websockets set up. Also I didn't consider capturing the output in a tiddler. I think that the focus of our approaches is very different, mine came from wanting to be able to easily interact with physical systems for things like my robot or a home automation system where this seems to be for altering the wiki itself. I don't think that the two approaches are incompatible.

The only functional differences that I can see between the two approaches is that I have some queuing set up so that you can run scripts in sequence or run multiple scripts/queues asynchronously.
I think that would be a nice addition here but it is outside the scope of the current commits.

I don't know what the plan is for being able to trigger the execution from the browser. I have been trying to find a way to integrate the websockets I use with Bob into the normal server but I can't find a consistent way to allow both the single-wiki focused core server and the multi-wiki Bob server.

As an alternative to websockets I think that adding a REST-type api to run the commands would be relatively simple to implement and probably much more in line with what people expect from something like this.

@sukima

This comment has been minimized.

Contributor

sukima commented Oct 3, 2018

Is this not the same as placing a node only plugin that opens a stream/pipeline?

@Jermolene

This comment has been minimized.

Owner

Jermolene commented Oct 3, 2018

The details are different between the two approaches, but that seems to mainly be from the difference between the tiddlywiki commands and how I have the websockets set up. Also I didn't consider capturing the output in a tiddler. I think that the focus of our approaches is very different, mine came from wanting to be able to easily interact with physical systems for things like my robot or a home automation system where this seems to be for altering the wiki itself. I don't think that the two approaches are incompatible.

The motivation for the work was to enable developers with Unix-y skills to be quickly productive with TiddlyWiki.

The only functional differences that I can see between the two approaches is that I have some queuing set up so that you can run scripts in sequence or run multiple scripts/queues asynchronously.
I think that would be a nice addition here but it is outside the scope of the current commits.

Yup, I'm interested in experimenting with more ways to trigger tasks: as a cron job, via a widget message etc. Queuing tasks could definitely useful too.

I don't know what the plan is for being able to trigger the execution from the browser. I have been trying to find a way to integrate the websockets I use with Bob into the normal server but I can't find a consistent way to allow both the single-wiki focused core server and the multi-wiki Bob server.

I'm thinking we'd have an HTTP POST endpoint for triggering a task, and eventually a means to check on the progress of tasks via HTTP.

As an alternative to websockets I think that adding a REST-type api to run the commands would be relatively simple to implement and probably much more in line with what people expect from something like this.

As I think we've discussed, I'm still interested in exploring using Server Sent Events for basic signalling between server and client.

@inmysocks

This comment has been minimized.

Contributor

inmysocks commented Oct 3, 2018

As I am now very annoyed with my other projects I will try making a plugin to test out server sent events for adding tiddlers to a wiki.

@Jermolene

This comment has been minimized.

Owner

Jermolene commented Oct 3, 2018

Thanks @inmysocks!

@inmysocks

This comment has been minimized.

Contributor

inmysocks commented Oct 3, 2018

Ok, I have a prototype for an extensible implementation of server sent events. I am happy with the browser side but I think that the server side needs some work. @Jermolene I am hoping that you have some insight about where to put the components.
I packaged a demo as a plugin and put how to use it and my concerns in the readme here:
https://github.com/OokTech/TW5-ServerSentEvents

@Jermolene

This comment has been minimized.

Owner

Jermolene commented Oct 11, 2018

Hi @inmysocks thank you, that's really helpful.

I think that the most valuable, generic application of SSEs is for real time propogation of tiddler changes from the server to connected browsers. I'd be inclined to wait until that point before we integrate with the core. It strikes me that doing this is going to require us to properly think through TW5's sync architecture. In particular, I think we need to switch from syncing the values of tiddlers to syncing their individual edit history. This (aka operational transforms) is a technique that distributed editing systems have been using for a decade, and I think it's time we bought it into TW5.

Do you think it's worth putting in some hooks in the meantime so that experiments like this can be done without needing to hack server.js?

@inmysocks

This comment has been minimized.

Contributor

inmysocks commented Oct 11, 2018

@Jermolene Yes, I think that having a way to add server routes outside of server.js would be a huge help. I haven't taken a good look at the new architecture for the server in the upcoming version so I am not sure how much work it would take.

Between the SSE prototype and Bob I have almost all of the components needed to make a system for propagating changes from the server to browser without requiring external npm modules. I have been working on the file system monitoring part and I think that I am close to making it independent of the messaging channel between the server and browser. The biggest problem I have for converting Bobs functionality to work with the core is handling multiple wikis. I am in the process of making Bob use different $tw.Wiki objects for each wiki and I think that I can make it work with one or many wikis, but there may be some changes to how $tw.boot.files works and I have the feeling that is a problem.

I agree about syncing the edit history, luckily that changes very little about the communication model I have been using because the transport is independent of the actions on either end. I haven't done much research into operational transforms but my brother has done some and has some ideas about implementing simple versions for tiddlywiki. I don't know if he made any progress with it yet or not.

Jermolene added some commits Oct 13, 2018

Rename external tasks to external pipes
And add support for connecting via sockets to existing processers/tasks
Add support for remote commands
Makes it possible to invoke any of TW5's Node.js commands from the browser. Useful for the new pipe command, but also handy for e.g. rendering tiddlers on the server, fetching remote files, or importing local files etc

Docs to come.
Bring "Alice in Wonderland" properly into the wiki
We were referencing it in the tw5.com edition, but that makes it harder to use the externalpipesdemo edition independently.
@BurningTreeC

This comment has been minimized.

Contributor

BurningTreeC commented Oct 19, 2018

I love it, merge merge merge 👍 😁

@BurningTreeC BurningTreeC referenced this pull request Nov 24, 2018

Closed

Releasing v5.1.18 #3573

@Jermolene Jermolene added the v5.1.20 label Nov 29, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment