a loader screen for large TiddlyWikis, supports deltas and progress.
Switch branches/tags
Nothing to show
Clone or download
kickscondor Fixed a bug in the 'archive' command that was preventing deltas from
being taken. Command does nothing now if no changes are found.
Latest commit d079b97 Oct 3, 2018


The TiddlyWiki Loader

This is a small collection of things to help with public TiddlyWikis. The idea is to prevent so much re-downloading of the whole wiki and to show progress to the reader---rather than just having the browser sit there while it loads.

Using the Loader (index.html)

So the first piece is the index.html---which shows progress of loading the wiki and then replaces itself with the wiki. There is no redirection, so the URL won't change.

To use this, put the index.html in a directory, next to your TiddyWiki---which must be named wiki.html.

┣━ index.html
┗━ wiki.html

NOTE: There might be a delay of several seconds while the wiki is loaded from memory. But it also seems that this happens when the file is loaded directly from the network---just be advised that there is probably still some work to do getting this to go faster.

Reduce Bandwidth with Deltas (kicksnap)

The kicksnap script copies your wiki and the index.html to a directory. If that directory exists already, then some patching files are generated in the same directory---these will be used by the loader above to patch the wiki file.

NOTE: You cannot just delete the directory generated by this script! It must be preserved so that subsequent runs of kicksnap will update the patch files.

To run it, supply the path to your wiki and the path to the output directory.

$ ./kicksnap ~/Downloads/MyWiki.html /var/www/html

Once the script starts generating patches, the directory will look like this:

┣━ index.html
┣━ wiki.html
┣━ wiki.html.diff
┣━ wiki.html.json
┗━ wiki.html.snap

The wiki.html is a copy of your full wiki, if you find that you need to access it.

Generating Archives

This delta strategy has the nice effect of making it very simple for us to generate archives of the wiki. So... that's built into the tool as well.

Just add the archive command at the end:

$ ./kicksnap ~/Downloads/MyWiki.html /var/www/html archive

This will generate an archives directory that will contain numbered archives. A wiki.html.lst file will also be created, containing a simple directory listing.

┣━ archives/
┣━ index.html
┣━ wiki.html
┗━ wiki.html.lst

The presence of these files will enable access to the archive list in the index.html. You can add "?archives" to the URL to the index.html and the archive list will appear.

Other Attempted Strategies

You might be wondering if there are other ways to do this---I have tried a few other ways and just want to mention them before you offer them up as suggestions for improving this setup.

Using Split Files

My first strategy was to split up the wiki file into anywhere from 20 to 200 separate files (split on line endings) and then using a diff to update only these 'chunks'. You can find the script kicksplit.py in the commit history of this repo.

The trouble with this strategy is that it was:

  • Not efficient: Daily diffs of a wiki are generally only 20k-100k. However, if twenty (of 200) chunks get touched (generally the case) then we are looking at 2 megs of download. By the end of a month, things would get up to 8 MB for a 20 MB wiki---so things got pointless really fast.
  • Overly complicated: Deleted lines would cause some chunks to disappear---I had to start rebalancing the data as time would pass, meaning that at least 10% of chunks were getting updated even with an increasingly complex algorithm.

Using Multiple Diffs

Another strategy was to add diffs for each time the file is changed. This wasn't a bad approach---but once the files began to build up, they became double the size of a single diff anyway. It wasn't worth the extra complexity.

Okay, Nothing Further

Except that there is a bit further.

Here is an article discussing my reasons behind this project:

And thank you to these two wikis for inspiring me to see the value of these wikis: