In-wiki core upgrading works. Good reasons to not feature this?

[twMat]

So @Telumire created a plugin for in-wiki core plugin upgrades. It works well AFAICT.

Could TW feature in-wiki upgrading natively (not necessarily Telumires implementatino), or are there good reasons, that outweigh the benefits, for why users should continue with the current quirky updating procedure?

[DesignThinkerer]

(@Telumire is my alias)

A few points to consider: to make my plugin work, I had to monkey-patch $tw.wiki.renderTiddler to intercept the native saving mechanism and replace the HTML output with the upgraded wiki, and monkey-patch $tw.notifier.display to intercept save operations and detect success or failure.

To make this part of the core, this would likely require:

• native event hooks or a dedicated upgrade API.
• standardizing the saver API so that all savers (local file system, GitHub, WebDAV, etc.) return clear, hookable promises upon completion.

Because this plugin relies on browser-native APIs (DOMParser, fetch, Blob, URL.createObjectURL), if integrated into the core, it would need to be strictly specified as a browser-only module and clearly disabled or hidden when running in a Node.js server environment.

I have a lot of free time right now so I'm open to the idea to work on a proposal, but only if the upgrade process itself is deemed safe enough by the maintainers. I consider myself as a junior level dev so I would need some guidance if this where to be integrated to the core.

I've made a repo with some tests for the plugin here: https://github.com/DesignThinkerer/inplace-upgrade

[twMat]

I have a lot of free time right now so I'm open to the idea to work on a proposal, but only if the upgrade process itself is deemed safe enough by the maintainers.

I'm not sure the gh discussions area alert maintainers but this is a generous offer not to be missed, so I'm taking the liberty of alerting @Jermolene and @pmario (Saq's nic says "busy", so skipping)

[pmario]

I highly appreciate the effort that was made to create the inplace-upgrade plugin. But I think it should stay a plugin.

• For a plugin it is OK to start compatibility with v5.3.2
• Because a plugin can point to the default upgrader as a fallback

For the core there are different priorities.

• Upgrading should be possible starting from v5.1.0.
• All core plugins need to be upgraded too
• If the wiki uses one or more languages, they need to be upgraded
• Is the code that is used to upgrade genuine

---

I do not really see, how the plugin makes the upgrade process simpler than:

• Open https://tiddlywiki.com/upgrade.html
• Import your wiki
• Unselect content that you do not want
• Click Upgrade
• Click Save button

And you are done.

[DesignThinkerer]

Upgrading should be possible starting from v5.1.0.

The reason why the plugin does not currently support wikis bellow 5.3.2 is simply because I'm using recent wikitext like <%if%> and the http range request message to signal the user that a new update is available, I think it should be relatively straightforward to enable compatibility with older wikis but I wanted to see if this plugin gained interest before investing more time in it.

All core plugins need to be upgraded too [..] If the wiki uses one or more languages, they need to be upgraded

Upgrading the core and languages should also be possible, it is just more work and they can manually be installed using the official in-wiki plugin library, but I could add this to the upgrade process if there is interest.

Is the code that is used to upgrade genuine

The code for the upgrade is fetched from https://tiddlywiki.com/empty.html, https://tiddlywiki.com/prerelease/empty.html, https://tiddlywiki.com/empty-external-core.html and https://tiddlywiki.com/prerelease/empty-external-core.html so it is as genuine as going manually to the official web page.

I do not really see, how the plugin makes the upgrade process simpler

With the plugin, the upgrade is one click on a button. This remove the navigating away steps.

For users hosting their wikis on tiddlyhost and github page, the usual upgrade process is more lengthy because they first need to find out how to download their wiki without just triggering a save, then download their wiki, upgrading it, then finding out how to upload the upgraded wiki and finally upload it (at least two more steps to the five you listed)

With the plugin, they only need to reload their tab after upgrading. The plugin also allow to upgrade to an external core plugin, which the current upgrader does not. There are ways to do it manually but it's not very intuitive for beginner and even more experienced users will need a bit of time to find the tiddlers that need to be changed to externalize their core.

Unselect content that you do not want

I could also show a modal to select what should or should not be included in the upgrade, like the official upgrader do.

---

Currently, this plugin serves primarily as a proof of concept to demonstrate that in-place TiddlyWiki upgrades without navigating away are technically feasible. I would appreciate feedback from core contributors on whether this approach introduces any unacceptable risks to user data that would outweigh its convenience

[pmario]

... why users should continue with the current quirky updating procedure?

What exactly is "quirky" with the current upgrade process?

[twMat]

@pmario

I now realize I phrased my OP poorly (and have rephrased it); My point is not for the standard distro to use Telumires specific plugin but simply that TW should feature in-wiki upgrading. The end result would/should of course be the same as via our current upgrading approach.

So my question is more clearly phrased as:

• Is there something problematic about in-wiki upgrading as a method, now that we know it works? For example, does the in-wiki approach somehow not allow for the same desired result as the current method does?

What exactly is "quirky" with the current upgrade process?

He he - we've become so accustomed to it that we have a hard time to even see it. But imagine what Steve Jobs would say about it. The current upgrade process is totally unique (not in the good sense). You must:

• go visit the separate upgrade site, locate your local file and upload it, have it updated, then download it again typically with a different title (I, annoyingly, even get a different file extension html vs htm) only then to restart/open the wiki
• ...not to mention if you're on Tiddlyhost - which many are, probably especially newcomers - then you need to go out to your /sites page, locate the download button and download it, go to upgrade page, loacate your copy to upload it, upgrade, download it, go back to TH and locate the upload button, upload and accept the warning, and finally you need to open the wiki which is now hopefully properly updated.

What other software would you accept this for?

Compare this to in-wiki upgrading.

[saqimtiaz]

It is really pleasing to see innovation in this area. I have only had time for a quick glance at the proposed implementation, but it appears there are two limitations with this approach:

1. Upgrader modules no longer work
2. The user does not get the opportunity to test the new upgraded wiki before saving it.

The alternative that has previously been proposed that would not have these shortcomings, is an upgrade button that opens upgrade.html in a fullscreen iframe or even an iframe in a new window, pass it the wiki content as a JSON payload triggering an import and the existing import UI, and then allow the user to save it over the existing wiki. See #9155.

[DesignThinkerer]

Good points. I hadn't considered that the upgrader modules actively reshape tiddler data during the transition, so bypassing the standard import pipeline via a raw JSON injection is definitely a flaw.

If I understand the alternative correctly, a safer and more robust architecture would look like this:

1. Bundle the current wiki's exported tiddlers into a single JSON payload tiddler (e.g., $:/temp/inplace-upgrade/Payload).

2. Generate a modal containing an iframe configured to allow necessary permissions:

<iframe src="..." sandbox="allow-scripts allow-same-origin allow-popups allow-downloads" allow="fullscreen"></iframe>

3. Load the target upgrade.html (stable or prerelease) into the iframe. Send the JSON payload into the iframe alongside a temporary $:/tags/StartupAction/PostRender tiddler designed to fire <$action-sendmessage $message="tm-import-tiddlers" $param="$:/temp/inplace-upgrade/Payload" />.

4. This triggers the new core's native import pipeline and runs the upgrader modules entirely within the iframe. The user can review the changes and test the upgraded wiki in a safe preview environment without touching their original file.

5. Once the user is satisfied and clicks save within the preview, the sandboxed wiki utilizes the postMessage saver (introduced in PR #5512) to broadcast the finalized HTML back up to the parent window.

6. The parent window listens for that broadcast, intercepts the updated HTML payload, and routes it to its own native saver to overwrite the original wiki.

Does this align with the approach you were suggesting?

[saqimtiaz]

@DesignThinkerer that is roughly the approach, yes. We will ideally need to add wikitext affordances for sending messages to an iframe and for receiving them, so that might be how we trigger tm-import-tiddlers.

If you have the opportunity to work on this, that would be very welcome.

[pmario]

@DesignThinkerer @twMat ... TiddlyHost and Git-* hosted wikis are a strong argument. I agree.

@saqimtiaz ...

• If it would be part of the core we could make upgrader modules work again.
• We will need to make sure that upgraders installed by 3rd party plugins also run. I need that for bundler plugin

---

@twMat About iDevices. Users trade convenience with "walled gardens, a moat with water and crocodiles" ;)

---

For TW we have no story about "module signing". If you visit https://tiddlywiki.com/upgrade.html you can be relatively confident, that it is genuine.

This already changes with the core "Get more plugins" button. While it loads the plugin library inside an iframe from tiddlywiki.com ... Users can not see where those plugins come from.

If we can solve this problem in reasonable way, I think an inline-upgrade button is doable. But I personally would consider this a pre-requisite.

Edit: There are elements in our build and publish process that are overdue to be improved. Especially the possibility for users to see, where their plugins came from. Ideally in a "signed" way. Which in this case will not increase security, but visibility. Which still is a win for me.

[twMat]

... be relatively confident, that it is genuine.

If native TW displays the url from where the upgrade is to be fetched - as, for example, is actually done in @DesignThinkerer 's plugin - then I think that'd be sufficient, as long as the user sticks to official TW stuff. (As you note, even the "Get more plugins" button is problematic otherwise.)

@twMat About iDevices. Users trade convenience with "walled gardens, a moat with water and crocodiles" ;)

True but pretty much all apps and popular software feature smooth upgrading these days. It is taken for granted, I must assume because it is important to keep the users.

[DesignThinkerer]

@pmario I agree that a provenance check is a very important topic but I think that considering it a pre-requisite for an in-place upgrading process is out of scope for this specific feature. If we follow the approach mentionned by saq, i.e a modal's <iframe> loading https://tiddlywiki.com/upgrade.html, then the security risk is identical to the user manually navigating to that url.

That said, your point about third-party upgrader modules is very relevant. If we pass the full wiki payload into upgrade.html within the iframe, the incoming third-party upgrader modules will be recognized by the import pipeline and can execute their migration routines just as they would in the standalone tool - it would just be more convenient.

[pmario]

I did update the post above, to remove the "blocking" pre-requisite. But I still think we should take the chance and improve our processes.

I did identify some elements that are relatively "cheap" to implement, without moving the core or it's build process into "gate keepers" for 3rd party content, which I did criticise in an other post.

3rd party libraries should also be able to increase there visibility of provenance, without the need for core staff involvement.

---

On the way I found a new bug, for which I will create a new issue ;)