Updates Part III: Smapp Update

[lrettig]

Requirements

• Primary: Maintain overall security, and integrity of user funds at all costs
• Secondary
  • Make things as easy as possible for non-technical users that are likely to be running smapp
  • Don't take agency away from the user/node operator or make minority forks unreasonably burdensome.
  • Minimize downtime and disruption
  • Maintain network stability
  • Keep the UI software up to date
  • Keep the underlying node software up to date
  • Make it easy to report problems and seek help if something goes wrong
• Tertiary: Educate the user that their actions (or lack thereof) are part of network governance and have important repercussions (like voting).

Design

• Add a screen to initial onboarding flow for configuring updates. Present four options: auto-update everything for me, auto-update UI only, auto-update node software only, I'll manage updates manually. By default, select the first, with a callout saying something like, "We recommend that you enable all auto-updates. This will ensure that you're always running the latest and greatest software released by the Spacemesh team, and will help keep the network secure. Note that this decision has implications on governance. Click here for more information."
• If the user clicks for more information, they're taken to a document that explains the update process and how it's like voting, and the implications of enabling auto-updates of frontend, backend, or both. The difference between the two types of updates, and the importance of each, is explained.
• If the user chooses to switch off some or all updates, a new callout appears saying something like (in addition to the above message), "...By disabling automatic updates, your node may fall out of sync with the rest of the network and you may be unable to send or receive transactions. Click here for more information."
• Save the user's choice and do not present them with the same screen again. Add a section to Settings that allows them to change this choice at any time in future, with the same callouts/links.
• If UI auto-update is enabled, fetch and install over-the-air (OTA) updates to the UI. Display an indicator (green/yellow/red) when an update has been fetched, indicating that the user must restart the app to install the update. On mouseover, the indicator should display a callout telling them that a new update is available and that they must restart to install it. They should be able to click here for more information on updates.
• (If explicit user action is not required to update the app, consider allowing auto-updates while the app is backgrounded, if possible. If the app is in the foreground, prompt the user before initiating the upgrade.)
• If node auto-update is enabled, pass the --auto-update flag to go-spacemesh (see Updates Part I: node auto-update #32). The app should track all node update events and surface them to the user: when a new update is detected, when it's downloaded, when the activation countdown has begun, when an update succeeds or fails to activate, etc. (This may be accomplished using the API or by monitoring logs.)
• Note: the user is likely to be confused by the difference between UI updates and node updates. However, we very likely need to distinguish between these as the former likely requires the user to restart the app, whereas the latter does not. Keep this in mind and be careful with how each of these is presented to the user.
• Ensure that the node software, and the API endpoints used by the smapp, remain backwards-compatible so that, even if the user chooses to upgrade the node software without upgrading the UI, they can continue to use the UI to control the node.
• If the user chooses to upgrade the UI but not the node software, ensure that the app fails gracefully if it's talking to an older version of the node software.
• If one or both auto-update modes are disabled:
  • If UI auto-update is disabled, check for updates anyway and display a warning indicator that the software is out of date and should be manually updated. Clicking on it should allow the user to manually update (either using OTA or with a link to a download), and to enable auto-updates.
  • If node auto-update is disabled, check for updates anyway (possibly using the same beacon that go-spacemesh uses, possibly via the API, possibly using a different endpoint) and display a warning callout if the node software is out of date. On mouseover it should read, "Node software is out of date, click here for more information." Clicking on it should allow the user to manually update the node software, and to enable auto-updates.

Tasks

• Consider whether it's possible to cleanly bundle node + UI updates so we can keep this distinction from the user
• Design, design, design. Finalize copy, mock and implement new screens
• Enable OTA updates of the app. Research whether these require explicit user action, such as restarting the app.
• Figure out how to detect new versions of both the node software and the smapp from within the UI
• Figure out how smapp can monitor update-related events in go-spacemesh. Consider adding one or more API endpoints for this, e.g., an updates stream.

[avive]

I'm not sure that letting the node take care of updating itself is the best option when it runs managed by Smapp. We are going to maintain the minimum latest node release version in the discovery web service so smapp can easily compare and know if there's a new node version and manage the node update process from outside the node.

I think this is much simpler solution because much less API interaction is needed between smapp and the go-sm node in this case. For example, on startup, smapp is going to get all data from the discovery service when it is configured to use a specific network - we need this anyhow. For example, for displaying the correct network dashboard in the ui and to configure explorer links. If it detects that a new node release is available it can easily handle the update process by gracefully shutting down the local node, downloading & replacing it with a new go-sm executable and starting it again with the proper cli flags.

The discovery web service is outlined here: https://product.spacemesh.io/#/public_webservices?id=discovery-web-service-phase-i

[lrettig]

I don't have a strong opinion about this. I'm fine with smapp managing updates for go-sm. And I don't think it changes much of this proposal. Just to fully explore the decision space here, another option would be to add a new API service to manage updates, that would let smapp tell go-sm to immediately initiate an update. But what you propose is simpler.

[avive]

All update related SMIPs needs to be productized. The go-sm full node config for updates need to be fully speced and the interaction between smapp and go-sm for updates needs to be defined. Thanks @lrettig for the good work on the related smips. We have a clear direction and consensus but there are many details to be figured out.

[avive]

I don't have a strong opinion about this. I'm fine with smapp managing updates for go-sm. And I don't think it changes much of this proposal. Just to fully explore the decision space here, another option would be to add a new API service to manage updates, that would let smapp tell go-sm to immediately initiate an update. But what you propose is simpler.

Yes, I'm inclined to head in this design direction: when go-sm is managed by smapp, smapp handles node updates. When go-sm is started from the command line w/o smapp it will use flags to determine update configuration which needs to include a discovery service url (where to get updates meta-data such as latest release for a network).

[avive]

There's more to discuss about this proposed smip. Currently a specific version of go-sm is bundled in smapp so there can't be a ui update only. It actually make sense to release a new version of smapp when a new node version is available and was tested to work with it, so updating the ui will also updated to the latest node release. I think it is a bit of an overkill to offer update-ui-only option as it requires testing all new ui changes with potentially several older node releases. These kind of complexities is also the reason why we decided to only support 1 network for updates and for web services and not several...

[avive]

Updated smapp update spec rough summary:

Preliminaries

1. A smapp release includes a go-sm release that it is designed to work with. We want to keep it this way to avoid users having to download go-sm on app first session as it requires to trust the discovery endpoint and for convince reasons.
2. We decided not to the support more than one sm network in smapp at any given time, as this requires maintaining a binary of go-sm for each network as not all networks may work with a specific go-sm release.
3. Smapp gets the 1 supported public network (e.g. a testnet release) meta-data such as config file, dash url and more via the discovery service. The only hard-coded network info in smapp is the discovery service canonical url https://discover.spacemesh.io/networks.json and a node release that was tested to work with this network and smapp assumes that the discovery service only returns data for 1 sm network. The discovery service include the latest smapp and go-sm binary release numbers for the network. For more info see https://product.spacemesh.io/#/public_webservices

Configuration

1. On smapp first run user has an option to opt-out from smapp checking from update (it is checked by default) and to disable auto update if check for updates is enabled:

• periodically check for app updates
  • auto-update app when a new update is available.

User should be able change these settings at any time from the settings screen. There should also be a manual check for updates button in the settings that will trigger a check for update when the user clicks it regardless of the other settings.

2. Smapp configures its managed node not to check for node update by-itself via the node cli flags. The node update checks are designed to work when the node is running w/o an app managing it. e.g. directly from the command line.
3. A new update of smapp may include a new node binary release. It should be packaged as an installer for each supported platform that can run in silent-mode by smapp or by an updater process. So we don't need to prepare a special updater binary, build ui-only update mechanism, and we just reuse the standard smapp installer for updates.
4. If check for app updates is on then smapp should periodically (say every 24 hours) check for available updates by using the discovery service endpoint.
5. If an update is available and auto-update is on and smapp is in the background (minimized window) then it should start the update process automatically. e.g. download and execute the update installer silently.
6. If smapp is in the foreground when a new update is available, then it should display a modal informing the user an update is available and prompt him to start the update regardless if auto-update is on or off. If user clicks on 'Later' because he's busy interacting with the app at the moment, then it should show the modal again in the next update check period.
7. After smapp installs an update it should automatically restart itself. e.g. it should run again after an update was installed.

Updating Smapp to join a new testnet

• A testnet that smapp is using may be discontinued from being supported by sm due to an unrecoverable error such as consensus failure. In this case sm is likely to release a new testnet with a new net id, and update the discovery service accordingly. e.g. with a new network id, supported smapp and go-sm releases and a config file for the new network.
• We want to find a process to migrate smapp users form an old testnet to a new one seamlessly but with keeping them informed and in the loop about the network change. Here's a proposed solution.

1. On startup, Smapp checks the discovery service to verify it is running on the supported network. It should compare the net id it has from its current config file with the advertised network id. If there's a mismatch then it should prompt the user to update to the new testnet in a modal box.
2. If user clicked to update then smapp should download the new config file and verify that its current included go-sm node release it has, is supported by the new network using the min node version data returned by the discovery service. In case it needs to update the managed node it should prompt the user to update smapp to join the new network and start the update process if user confirms. If its node release is supported by the new network then it can just join the new testnet by providing the new config file from the discovery service to its node.

Visual Design

App Startup Flow Changes

1. Add 'App Setup' step before the new wallet setup step in the first-run flow.
2. App Setup Screen content:

Title: App Updates
Content:

• periodically check for app updates

  • auto-update app when a new update is available.

• Learn more about updates link - links to docs in the testnet guide.

• If user un-checks the periodical updates checkbox then display an info area in the screen: We recommend that you enable all auto-updates. This will ensure that you're always running the latest and greatest software released by the Spacemesh team, and will help keep the network secure. Note that this decision has implications on governance..

App Update Section in Settings Screen

• periodically check for app updates

  • auto-update app when a new update is available.

• Check for Updates button to trigger a check for update.

New Update Available Modal Dialog Box

• Title: 'App Update Available'
• Content:
  • Installed app version: 0.x.x
  • New app version available: 0.y.y
  • 'Install Update' button - clicking on it should start an app update process.
  • 'Later' button - clicking on it should close the modal.

Migrate to a New Network Modal Dialog Box

• Title: 'Migrate to a new testnet'.
• Content:
  -- Testnet [net-id] is no longer supported and a new testnet [tn-id] is available. Would you like to migrate your app to work with the new testnet? -- Join new Testnetbutton - click will migrate smapp to work with the new net-id. --Later` button - click will close the dialog and smapp will continue to use the old testnet id.

Notes:

• Migrating to a new testnet should update the net-id in the wallet's file.
• If user was smeshing then we should highlight the smeshing tab with an icon such as ! and when user opens it we want to prompt him to start smeshing setup for the new network.

[lrettig]

A smapp release includes a go-sm release that it is designed to work with

Personally I would like to see less tight coupling between smapp and go-sm, e.g., so that a user could install go-sm via another trusted means (like a system package manager, homebrew or aptitude), that they could have multiple versions of go-sm installed (nvm/pyenv-style) and upgrade the two separately, etc., but I can see why it's simpler to do things this way for now.

'Later' button - clicking on it should close the modal.

I think a more common pattern these days is "remind me again in X hours" (e.g., 24 hrs), or "try again tonight" (i.e., specify a timeframe)

Migrate to a New Network Modal Dialog Box

Is this intended to work for mainnet hard fork network upgrades as well? We need to consider the mainnet use case.

Regarding the testnet, one issue here is that our testnet upgrades haven't been so smooth - there's typically been a period of a few days before we troubleshoot and relaunch the testnet, so there's isn't a new network available to switch to immediately. What should the UX look like in this case? In practice many smeshers will have closed the Smapp completely by the time the new testnet is online.

[avive]

1. Later is basically a polite way to say not now. I think that try again in x hours is an overkill. User is going to be repeatedly prompted to update every 24 hours until he updates.
2. Migration to a new network is not intended to work on mainnet unless in the event that we decide launch a new mainnet with a new network id.
3. The migrate to a new network will only be displayed when the discovery service endpoint had announced the availability of a new network. Until then, nothing will be displayed. We may add a link to the public status page from smapp network screen so users can see if there's an outage.

There's a fine line here between over-engineering for testnet and providing a decent testnet user experience.

[avive]

It is quite hard to write a spec in a comment. I moved the spec draft here https://product.spacemesh.io/#/smapp_update_spec and we can discuss it here and move back to a smip once it is ready.

[brusherru]

The spec that Aviv proposed sounds much easier to implement.
And I think we should definitely start with this solution because we can easily change the auto-update flow later.
However, I have some thoughts about the “ideal” auto-updating flow.

Bundling VS Downloading Go-Spacemesh

I don't really like the idea to download the whole Smapp to update go-sm.

I see the Pros:

1. No need to invent something — just update Smapp.

I see these Cons:

1. We'll need to release "meaningless" versions of Smapp (for Smapp itself). E.G. 0.3.0 -> 0.4.0 which bundles just a new version of go-spacemesh. I'd prefer to follow the semver standard and don't bump versions unnecessarily (and it also relates to another topic of versioning Smapp, now it is tied to roadmap milestones).
2. Smapp.dmg (~130MB) vs go-sm.zip (~40MB). Yep, it is not much in modern times, but still.
3. In case if we have a couple of running networks with different go-spacemesh versions, we can't just switch between them, we need to reinstall a corresponding version of Smapp.

So, I'd prefer to download and install go-spacemesh updates separately from Smapp.
Pros:

1. We don't need to bundle go-sm at all (so Smapp.dmg will be lighter)
2. Smapp will download the latest and actual node for the selected network, and use it.
3. Moreover, Smapp can contain a few versions of go-sm, and just switch between them on switching the network, instead of downloading it again. However, it may require some additional code for deleting the outdated node versions.
4. Smapp won't be tied up with go-spacemesh version directly, but...
   Cons:
5. It may require extra metadata served on CDN like a version match table. E.G. Smapp 0.8.3 supports >= 0.26.5, Smapp 0.9.0 supports >= 0.27.0, etc.
6. There might be a problems with running downloaded binaries without requiring any actions from User (aka com.apple.quarantine). Research needed.

But, as I said before, we can change it later, if we'll have any auto-update mechanism of Smapp :)

Who Updates the Node

(if not bundling it into Smapp)

We have two options:

• Node updates itself
• Smapp updates the Node

I think that since the Node can do this, it is much better to put it under her control.
Pros:

1. The Node can decide on its own should it download the update or not. I mean, if we will have some consensus mechanism on the network, it can rely on it, instead of a discovery service.
2. The Node can decide when is the best moment to update it (E.G. it finished producing a block, made votes, choose a proper "update window", etc).
3. And as a consequence, we should maintain only a single possible auto-update flow, managed by the Node itself.
4. In case if in the future will be some new clients for go-sm, it won't require any code related to updating the node.
   Cons:
5. However, the two last Pros require some extra API.
6. If this API will not only notify about available updates and updating process but also may wait for the "update now" command we probably need some authentication and this is a big disadvantage for this case. I describe a little bit more about it beneath.

API & Node Auto-Updating

First of all, API should have a stream to notify about availability of new updates, download process, and "ready-to-install" state.
Secondly, if we have turned off auto-update with the CLI flag (aka --no-auto-update), it will only notify about new updates and requires some extra interaction via API (aka "download&install"), which may require some kind of authentication to prevent availability for consumers of the remote node to update it. Or, probably, such interactions may be in the different layers, but I don't sure that it is easier.

So when the go-sm sends a message "ready-to-install" Smapp will check is it in the foreground and then prompt the User to install it or just install if auto-update is turned on and Smapp is in the background.

How to prompt about new update of go-sm

I think that is to prompt with modal dialog is not a good way, because it requires to make an immediate decision about it (to choose "Later" for example), and then it is a bit complicated to run update after interaction with Smapp (E.G. go to Settings, find "Update" section, click on "Update" button...). So I'd like more if we show such messages in the "Snackbar". I mean that this should not be a modal dialog, but a little notification, that would not block an application from interaction, and will be available on any screen. So the User can decide to click on "Update" any time when it is convenient.
But probably we have to implement some tricky thing to handle the case when the Smapp is in the foreground, but the User doesn't use it. A possible solution is to add a timer. E.G. Smapp shows up the message "A new version 0.3.1 of go-spacemesh is available" and two buttons "Update now (30 minutes)" and "Postpone for 24 hours". And this "30 minutes" is counting down.

How to prompt about new Smapp update

I think that updating Smapp might be the same as I proposed above for go-sm. So when the update is available, it will download it and:
a). Smapp is in the foreground: show prompt with timer
b). Smapp is in the background: update silently (if possible without requiring admin rights)

Avoid stopping the Node on Smapp update

If we decouple Smapp from go-sm, then we can tweak our App a little bit and run go-sm in the parallel process (Smapp would not be a parent anymore). It will make it possible to:

1. Update Smapp without stopping the Node
2. "Connect" Smapp to the running go-spacemesh process (it might be a bit tricky, but I think it is possible to find out such process, aka ps -a | grep go-spacemesh)
3. And even close Smapp (without running in the background), but keep go-spacemesh working. Such a solution will require switching the auto-updating mode of go-spacemesh (not just a cli flags).

Force auto-updating

Since auto-updating is the recommended option, I think that we can not bother Users with any questions about it on the initial startup and turn it on by default. But in case if someone wants to turn it off — he can open Settings screen and turn it off. It also will require to switch auto-updating mode of go-sm.

Switching auto-update on or off

In any case, if even we will ask the User on the initial startup to select do auto-updates or not, we will provide a possibility to User to change the mind and turn it on or off. And I think that is much desirable to avoid restarting Node. So probably this is another case that should be handled properly by API.

---

Updating Smapp to join a new testnet

Something, like described there, is done within spacemeshos/smapp#851
But it doesn't

• check for matching versions and won't download a new version of go-sm / smapp.
• prompt to switch the network, if the old one (and which is stored in the wallet file) still exist in the discovery service

---

To sum up, I think we can start with the simplest solution: make auto-updates for Smapp, which bundles go-sm.
This solution requires only a few things:

• upload JSON for auto-updates in GCS and point to it in electron-builder config
• run check for updates with some interval
• add a button to "check for updates" and "download&install" in the Settings screen
• add options to turn on/off auto-updates to the Settings screen
• add prompt "install now" or "postpone"

And then we can improve it at any moment, after discussing all the details.

---

See also comment about node updates: #32 (comment)

[lrettig]

So, I'd prefer to download and install go-spacemesh updates separately from Smapp.

Okay with me. I see the benefits.

5. It may require extra metadata served on CDN like a version match table

Why not just bundle this metadata with smapp?

API should have a stream to notify about availability of new updates, download process, and "ready-to-install" state... it will only notify about new updates and requires some extra interaction via API (aka "download&install"), which may require some kind of authentication to prevent availability for consumers of the remote node to update it

This is totally fine. The API is split up into multiple services, and some of them (e.g., the debug service) are not intended to be exposed remotely. We can create a new updates service and do the same.

2. "Connect" Smapp to the running go-spacemesh process (it might be a bit tricky, but I think it is possible to find out such process, aka ps -a | grep go-spacemesh)

There's no need for a PID or IPC, all communication can flow over the API. It just needs to try a default endpoint, e.g., localhost:1234.