aggregating remote tiddlers

[FND]

Greetings from the great beyond,

I've been playing with the idea of using TW5 as a front end to aggregate tiddlers from remote sources, some of which might also be writable. After doing some research, I've concluded that I might be Doing It Completely Wrong™, perhaps even reinventing the wheel - so despite being familiar with the TW universe from the olden days, I ended up with a bunch of (mostly basic) questions.

Here's the flow I imagine:

1. empty¹ TiddlyWiki is loaded and displayed

2. skinny tiddlers are loaded asynchronously, from multiple sources, populating the timeline

   some sources might require a little auth dance first, so we might have to prompt for user intervention (e.g. via a tiddler with login forms) before we can proceed

   typically a remote source would be a TiddlyWeb server, but it might be something else altogether

3. tiddler bodies are loaded asynchronously

   this might take a while, so the most relevant (e.g. recent) content should be prioritized

   some tiddlers are read-only, others might be editable (the respective server response will indicate this)

So much for the passive part. When a tiddler is edited, it should be saved back to the respective source, individually. Thus the TiddlyWiki document is only ever a snapshot with an in-memory store instead of being persisted as a whole.

Constructing the respective HTTP requests for remote sources is the easy part. What I'm unsure about is how to properly integrate that data with TW5. I'm aware of the TiddlyWeb adaptor, PUT saver as well as the discussion in #5269 - but it seems to me that the flow described above doesn't quite fit into existing patterns? (Happy to be proven wrong.)

Here's a set of tangible questions I've identified:

• basic plugin hygiene: I've successfully created an experimental plugin², but I'm not sure that's how it's supposed to work. Mostly I'm uncertain about tiddler fields (cargo-culted and/or guessed) and the require mechanism:

  plugin tiddlers (click to reveal)

  anticipating the need for modularization, I've extracted a simple library into a separate tiddler

  title: LazyLoaderPlugin
  type: application/javascript
  module-type: startup
  

  let { httpRequest } = require("./util.js");
  
  exports.name = "lazy-loader";
  exports.after = ["startup"];
  exports.platforms = ["browser"];
  exports.synchronous = false;
  exports.startup = async callback => {
      // we don't actually wanna block rendering here; instead we'll populate
      // the store in the background
      callback();
  
      let res = await httpRequest(…);
      …
      $tw.wiki.addTiddler({ … });
  };

  title: util.js
  type: application/javascript
  module-type: library
  library: yes
  

  module.exports = {
      // promisify TiddlyWiki's `httpRequest`
      httpRequest: (method, url, headers, body) {
          return new Promise((resolve, reject) => {
              $tw.utils.httpRequest(…);
          });
      }
  };

• saver precedence: How can I make sure that my custom saver is used and all others are disabled? (Assuming I don't have that minimal build¹ yet.)

• saver types: If I only ever want to save individual tiddlers, what's the proper interface/hook for this?

• skinny tiddlers: Should I use _is_skinny: true when first invoking addTiddler? What happens if a skinny tiddler is displayed; would that kick off automatic loading? If so, is there a hook for that (e.g. so I could factor it into my priority queue if tiddlers are still being loaded)?

• read-only tiddlers: Is there an exisiting flag to prevent editing? (Edit mode should still be accessible in the UI, e.g. to allow forking tiddlers.)

• dirty tiddlers: Assuming my saver is notified when a tiddler was edited, is there a "pending" state while that tiddler's asynchronous request is awaiting a response?

• dirty store:

  • Would the store's global dirty flag (if that concept still exists in TW5) be appropriate to signal conflicts, i.e. when a tiddler was edited but could not be saved?
  • Might that flag also be used to signal that all tiddlers have been fully ingested on startup?

Apologies for imposing a wall of text on y'all; the context seems relevant from my current perspective, but I'm happy to excise questions into separate discussions if that's preferred.

¹ ideally this would be a minimal build, e.g. keeping only those modules required for the main story, timeline and search - though that's an optimization we should probably discuss separately

² I currently use a custom script to extend emtpy.html's <div id="storeArea">, creating a <div ${fields}><pre>${content}</pre></div> for each tiddler file - merely because it was the Simplest Thing for starters

[twMat]

@inmysocks - maybe you have any thoughts on this? :-)

[inmysocks]

Bob has ways to do a lot of this, at least for the server part. A lot of it is only the pieces implemented without the full setup because I get distracted easily, but most of it is there.

And TWederation covers a lot of the components in the interface wiki. It is a generalisation of the mechanism Jeremy made for the plugin library that allows a wiki to fetch tiddlers from other wikis, and I have extended it a bit to allow it to also interact with a Bob server in a very similar way to what you are describing, but once again while the pieces are mostly there they are not put together completely.

As an alternative back-end I made a WikiDB which is a document database that uses tiddlywiki-style filters as the query language and the document storage model is based on tiddlers, but before I got enough time to polish all of this and release it my day job picked up and work has slowed down a lot.

So yes, I have lots of thoughts on this, unfortunately less time to work on it than I would like.

If you want to look at existing things to see if they are useful to you there is twederbob https://github.com/OokTech/TW5-TWederBob and the mechanism Jeremy is using for the links aggregator is something like twederation so that may help.

[FND]

Thanks for the pointers, that does sound interesting; I'll take a look and see what I can learn.

[Jermolene]

Greetings from the great beyond,

Hi! And apologies for the late response.

I've been playing with the idea of using TW5 as a front end to aggregate tiddlers from remote sources, some of which might also be writable. After doing some research, I've concluded that I might be Doing It Completely Wrong™, perhaps even reinventing the wheel - so despite being familiar with the TW universe from the olden days, I ended up with a bunch of (mostly basic) questions.

Here's the flow I imagine:

1. empty¹ TiddlyWiki is loaded and displayed
2. skinny tiddlers are loaded asynchronously, from multiple sources, populating the timeline
   some sources might require a little auth dance first, so we might have to prompt for user intervention (e.g. via a tiddler with login forms) before we can proceed
   typically a remote source would be a TiddlyWeb server, but it might be something else altogether
3. tiddler bodies are loaded asynchronously
   this might take a while, so the most relevant (e.g. recent) content should be prioritized
   some tiddlers are read-only, others might be editable (the respective server response will indicate this)

So much for the passive part. When a tiddler is edited, it should be saved back to the respective source, individually. Thus the TiddlyWiki document is only ever a snapshot with an in-memory store instead of being persisted as a whole.

That all makes sense.

Constructing the respective HTTP requests for remote sources is the easy part. What I'm unsure about is how to properly integrate that data with TW5. I'm aware of the TiddlyWeb adaptor, PUT saver as well as the discussion in #5269 - but it seems to me that the flow described above doesn't quite fit into existing patterns? (Happy to be proven wrong.)

The TiddlyWeb syncadapator works with the core syncer engine to synchronise individual tiddlers at startup and as they change on the server.

The PUT Saver saves the entire HTML file to a single location.

What you want could plausibly be done as a syncadaptor integrating with the core syncer engine, or you could bypass the syncer and make it work independently.

Here's a set of tangible questions I've identified:

• basic plugin hygiene: I've successfully created an experimental plugin², but I'm not sure that's how it's supposed to work. Mostly I'm uncertain about tiddler fields (cargo-culted and/or guessed) and the require mechanism:

TW5 uses it's own CommonJS compatible loader that loads modules from tiddlers, falling back to the Node "require()" method on the server. Cargo culting should serve you well.

let { httpRequest } = require("./util.js");

The core sticks to really old ES5 JavaScript for backwards compatibility. It's callbacks all the way. Of course you can write modules that do use fancy new stuff (and I do all the time), but you just need to watch your interfaces with the core.

• saver precedence: How can I make sure that my custom saver is used and all others are disabled? (Assuming I don't have that minimal build¹ yet.)

As I say, savers are for saving the entire wiki, and so are probably not relevant here.

• saver types: If I only ever want to save individual tiddlers, what's the proper interface/hook for this?

That would be a syncadaptor.

• skinny tiddlers: Should I use _is_skinny: true when first invoking addTiddler? What happens if a skinny tiddler is displayed; would that kick off automatic loading? If so, is there a hook for that (e.g. so I could factor it into my priority queue if tiddlers are still being loaded)?

Yes, displaying a tiddler should kick off the automatic loading.

• read-only tiddlers: Is there an exisiting flag to prevent editing? (Edit mode should still be accessible in the UI, e.g. to allow forking tiddlers.)

No, there's no direct way to restrict editing on particular tiddlers, although it's easy enough to hide the editing controls for tiddlers that don't meet criteria.

• dirty tiddlers: Assuming my saver is notified when a tiddler was edited, is there a "pending" state while that tiddler's asynchronous request is awaiting a response?

If you do write a syncadaptor, the syncer does indeed know that there is outstanding request and will wait for it to complete before initiating a new one.

• dirty store:

  • Would the store's global dirty flag (if that concept still exists in TW5) be appropriate to signal conflicts, i.e. when a tiddler was edited but could not be saved?

The saver handler and the syncer each handle the dirty flag slightly differently, essentially listening to change events on the wiki store and tracking changes themselves.

• Might that flag also be used to signal that all tiddlers have been fully ingested on startup?

If you want to hold up the UI until things have been loaded, you may want to load your tiddlers in a preload phase, before TiddlyWiki loads.

Apologies for imposing a wall of text on y'all; the context seems relevant from my current perspective, but I'm happy to excise questions into separate discussions if that's preferred.

¹ ideally this would be a minimal build, e.g. keeping only those modules required for the main story, timeline and search - though that's an optimization we should probably discuss separately

² I currently use a custom script to extend emtpy.html's <div id="storeArea">, creating a <div ${fields}><pre>${content}</pre></div> for each tiddler file - merely because it was the Simplest Thing for starters

A lot of the techniques you are describing can be seen in the work I've done on an AWS-based serverless TiddlyWiki. For example, this is the editing front end of my blog which is at https://jermolene.com/

https://xememex.com/jermolene/

If you poke around in devtools you'll see that:

• There is a preload script that loads updated tiddlers from the server at startup (there's a progress bar while that happens)
• There is a syncadaptor that tracks individual tiddlers via their source bag and etag stored in data tiddlers

I never quite got around to open sourcing all of that stuff, but happy to help you pull out anything useful.

[FND]

There's absolutely no need for apologies, particularly given my sweeping questions. Plus, as you know, asynchronous collaboration is kind of my inherited shtick...

Properly understanding the syncadaptor distinction gives me a whole new perspective; thanks for highlighting that. Contrary to my initial assumption, it sounds like syncadaptors allow for most of what I envision. I'll revisit the TiddlyWeb adaptor and will also dig into your custom one, that should give me enough to go on.

In the meantime, I've been sketching out this idea with a little stand-alone project; it helps me work out what it is I actually want/need, and I suspect much of the code (not that there's a lot) could be carried over into TW5. I've also used this as an excuse to play with Cloudflare Workers as a simplistic back end, plus I hope to look more closely into the File API (which made me rediscover this paper). If any of that results in useful artifacts, I'll share them here.

[saqimtiaz]

Hi @FND. Almost feels like its 2008 again. I was just reminded yesterday of some of our work on related matters back in the TW 2.x days. As it happens, I am currently contemplating a project that may share some of your described workflow.

I haven't had the time to flesh it out properly yet, let alone write it up. Skimming this thread may provide a general idea: https://groups.google.com/g/tiddlywiki/c/SJkmnGVQVjI

[FND]

Oh wow, it's like a proper reuinion here. Has anyone suggested using jQuery for TW5 yet?

Your ideas there sound intriguing. (I have no recollection of any such efforts, Senator.) I'll have to digest that, but wouldn't it be hilarious if this turned out to be an opportunity to get the band back together again? (I should mention though that I'm not sure how much energy I can put into my TW5 efforts in the coming weeks/months. I'm old now.)

[saqimtiaz]

@FND It was a pair programming session on one of my Osmosoft visits at the original office.. but it was the middle of a heatwave and the AC wasn't working so I am not surprised you do not remember!

The amount of time I can dedicate to this is extremely limited as well, and I am also in no rush. So if there proves to be sufficent mutual interest and overlap of needs, we could do a slow burn collaboration.

From my perspective, for the next few months the larger project proposal is more important to get fleshed out and approved with the relevant project partners, the technical details come later.

[FND]

I certainly do remember that AC situation now! I guess 200x wasn't as flawless as 202x might suggest...