Skip to content

addonupdaterinternals

Joseph Lee edited this page Jul 16, 2022 · 4 revisions

NVDA Add-on internals: Add-on Updater

Author: Joseph Lee

Introduction

An ancient quote states that, "... the only constant thing in life is change ...". People grow older, countries rise and fall, and computer programs change all the time. The last point is important as it highlights the ever-changing nature of technology, more so when trying to keep up with hardware and software changes to comply with laws, add new features, fix bugs, and improve computer security.

NVDA is no stranger to change. Every year, NV Access updates NVDA, sometimes to add features, sometimes to fix bugs, and about once a year, to change workings of NVDA ad-ons. Add-ons do change from time to time - adding features, updating localizations, and meet new maintainers.

Whenever add-ons change, users are asked to visit NVDA Community Add-ons website to browse and download updates. But what if users wanted to apply add-on updates without having to visit add-on repositories? Enter Add-on Updater, an add-on designed to update add-ons, including Add-on Updater itself. The purpose of Add-on Updater is to check for, download, and install add-on updates while notifying users about updates, as well as offering settings such as update channel and not updating certain add-ons. While there has been calls to let NVDA itself check for add-on updates, Add-on Updater serves as a proof of concept in response to such requests.

In NVDA Add-on Internals: Add-on Updater, we will tour how this add-on works, as well as how design decisions and keeping up with changes can have an impact on add-on maintenance. The add-on source code repository can be fond at https://gitub.com/josephsl/addonupdater.

IMPORTANT: Add-on Updater is a proof of concept for an NVDA issue on add-on updating (NVDA issue 3208). Therefore, when NVDA itself supports add-on update feature, this add-on will be discontinued.

History of Add-on Updater

Add-on Updater made its debut in 2018. Prior to this add-on, an add-on named NVDA Store was made available to browse and download add-ons hosted on a different website. Although several add-on repositories exist including the official community add-ons website hosted on addons.nvda-project.org, there was no add-on to download add-ons hosted on the official site. As a result, Add-on Updater was designed to obtain ad-on updates hosted on the official website while keeping its interface minimal.

In reality, the idea of updating add-ons from NVDA itself is not new. A few months after NVDA add-ons made their debut in 2012, several community members felt NVDA should add an "update add-ons" feature. This was officially recorded in early 2013 (NVDA issue 3208), and I (Joseph Lee) spent some time writing a client side implementation directly in NVDA source code while discussing details with several NVDA contributors and add-on authors.

Fast forward to 2016, and I was working on StationPlaylist add-on 7.0. One of its highlights was ability to check for add-on updates. Borrowing heavily from a combination of existing NVDA code for checking for screen reader updates and a pull request for issue 3208, I put together a rudimentary update check feature. In the beginning, this feature opened a web browser to download updates, but realized that this amounts to opening the add-ons website with a web browser. Then I revisited NVDA's own update check facility (updateCheck.py) and learned that a file transfer loop was used to download updates while providing a progress dialog. I then reworked StationPlaylist add-on to update its update check logic, and eventually this became the basis for Add-on Updater generation 1 (2018).

But it was Windows App Essentials add-on that brought improvements as it too gained add-on update feature in erly 2017. Because the development snapshots were released more often than StationPlaylist add-on, the update check feature was tested widely. The work done from Windows App Essentials implementation of add-on update feature included an early version of Add-on Updater settings interface, including an option to choose different update channels. In some way, much of Add-on Updater settings panel traces its origin to Windows App Essentials.

Then came 2018, and I was busying myself with early research into Python 3 transition for NVDA. In the midst of my research, I came across NVDA Store add-on from the French NVDA community, and I realized that NVDA issue 3208 was still open with no resolution. In July 2018, I spent several days working on what would eventually become a prototype of Add-on updater, and released the first test version in late July. For the next several days, the community provided bug reports, and after further refinements, Add-on Updater made its official debut in October. the initial version of Add-on Updater borrowed heavily from a combination of StationPlaylist and Windows App Essentials add-on, as well as digging the pull request I've been working on for years. After Add-on Updater was released, update check feature was removed from StationPlaylist and Windows App Essentials add-ons.

In its early days, Add-on Updater accessed community add-ons website (addons.nvda-project.org). The biggest limitation was that Add-on Updater did not know about new add-ons on the website for days after add-on releases as a static map was used to map add-on name (internal identifier) to add-on download key used on the community website. For this reason, new add-on releases were made whenever new add-ons were introduced. This was the days before add-on compatibility check, and NV Access would implement compatibility checks from 2019.3 (early 2020) with Python 3 transition. The latter led to a bug where Add-on Updater would download incompatible add-on releases just because versions were different.

To resolve compatibility check issues and to make add-on metadata more dynamic, in 2021 I and several add-on authors decided to create a dedicated add-on compatibility repository on GitHub. The idea was to improve the existing Add-on Updater update eligibility (version check code then) with metadata coming from a json file stored on this new repository that contained add-on name, summary, author, download key from the community ad-ons website, and compatibility information (minimum and last tested NVDA versions). The key requirement was that add-on metadata must be dynamic - that is, no more Add-on Updater releases simply to add new add-ons and to prevent download of incompatible add-ons. Luke Davis, one of the add-ons community members, provided valuable feedback in the overall metadata design, with community members providing additional feedback. The result is Add-on Updater generation 2 (21.08) and this became the basis for add-on update check protocol 2 (see below).

Even with the add-on update metadata becoming more dynamic, there was one major issue with Add-on Updater design from 2018: actual update routine was tied to the graphical user interface (GUI). Among other things, this prevented implementation of automatic add-on updating feature, similar to background app updates on smartphone operating systems such as iOS and Android. This led to Project Meteor in July 2022, an effort to separate actual update code from the progress dialog. This also led to update check facility refactor to update and reorganize the entire facility, resulting in creation of add-on update record and update check protocol classes (explained below), since before then a dictionary was passed around wen retrieving add-on update metadata.

As of time of this article, Project Meteor (Add-on Updater generation 3) is complete. The internals described below refer to Project Meteor unless noted otherwise. Even with these changes, the core goal of Add-on Updater remains the same: check for, download, and update add-ons in response to NVDA issue 3208. Eventually I hope the core code from this add-on can become part of NVDA screen reader, but until then, Add-on Updater is faithfully carrying out its duties. As a proof of concept (and an actual add-on), the day add-on updating comes to NVDA screen reader, Add-on Updater will become part of history.

Design and components

Add-on Updater is a collection of modules grouped under a global plugin. At the top of the hierarchy is init.py, the main module responsible for coordinating update checks at startup, defining the user interface, and providing settings interface. Other files in this global plugin are workers responsible for actual update checks, downloading and installing add-on packages, and from 2022, ways to locate update information from various sources and formats (called "update check protocol"). These worker modules are:

  • Extended add-on helper (addonHelperEx.py): borrows somewhat from NVDA's own add-on handler package and coordinates automated update checks, as well as downloading and installing ad-on updates in the background if told to do so. Prior to 2022, this module housed internals of add-on update check facility.
  • Extended add-on GUI (addonGuiEx.py): shows add-on updates dialog to let users choose which add-ons to update if updates are available, as well as coordinating add-on update download and installation.
  • Add-on Updater utilities (addonUtils.py): mostly responsible for housing add-on settings, as well as updating root security certificates if SSL error is seen when accessing add-on update sources.
  • Add-on URL's (urls.py, contributed by Luke Davis): houses an object returning various add-on update URL's.
  • Add-on update processes and procedures (addonUpdateProc.py): split from extended add-on handler in 2022, this module defines actual internals of add-on update check, download, and installation steps. For update check facility, the below module provides actual internals but update proc module is tasked with presenting the actual updates. This module also defines an add-on update record class (see below) for recording update metadata.
  • Add-on update protocols (addonUpdateProtocols.py): defines update check protocol, a family of classes allowing Add-on Updater to obtain update metadata from various sources in a variety of formats. See below for protocol internals.

Add-on update record

Introduced in 2022, an add-on update record (addonUpdateProc.AddonUpdateRecord) is an object that records update metadata. These include add-on identifier/name, summary, installed and update versions, update channel, and other attributes for future use. Add-on Updater uses update records to communicate update availability (an update is available if the installed and update versions are different).

Prior to 2022, a dictionary was used to record update metadata. For each installed add-on, it records its summary, current (installed) version, update version, and update channel. Because it was a dictionary, it did not allow addition of new attributes easily, therefore add-on update record class was defined in 2022 to clean up the add-on internals and to prepare for future possibilities such as hash value checks.

Update check protocols

An update check protocol (protocol for short) is a class defining routines to gather add-on update metadata from a variety of sources. While all sources (as of 2022) use JSON for data exchange, each source uses a different format to store update information. For example, community add-ons website uses a simple mapping between download key and URL, whereas an enhanced version of this protocol uses a different JSON file to store add-on name, summary, and compatibility information (minimum and last tested NVDA versions).

All protocol classes (housed in addonUpdateProtocols module) derive from a base protocol (protocol (0), a protocol that does nothing when asked to obtain update metadata. It's purpose is to give a blueprint and base services for other protocols, such as defining checkForAddonUpdates method, the heart of update check facility across protocols. Protocol implementations are responsible for defining the following methods for obtaining update information:

  • checkForAddonUpdate(current add-ons list): accesses a source URL for the protocol, taking in a pseudo update record - a list of add-on update records based on installed add-ons with update version set to installed version. See below for details on its mechanics.
  • fetchAddonInfo(add-on record, results, optional dictionary): locates actual update information based on different metadata formats. Among other things, it fetches update version if defined.

In addition to protocol 0, the following protocols are defined:

  • NVDA community add-ons website (protocol 1): accesses addons.nvda-project.org to obtain update information. Due to the format used, compatibility check is impossible. This was the behavior of Add-on Updater 21.05 and earlier.
  • Community add-ons website enhanced with compatibility information from the community (protocol 2): once a year, NV Access designates year.1 release as compatibility breaking release. NVDA will refuse to install add-ons if it is incompatible. To account for this, recent add-on packages include compatibility information (minimum and last tested NVDA versions) so NVDA can inform users about compatibility if the add-on about to be installed is deemed incompatible. In response, in 2021, I (Joseph Lee) and the add-ons community launched a GitHub repository hosting add-on compatibility statements so Add-on Updater will not download incompatible add-on updates. As this protocol also accesses comunity add-ons website, this protocol is an enhanced version of protocol 1 with compatibility information coming from a JSON file hosted on the new GitHub repository. This is the default behavior in Add-on Updater 21.07 and later and is the default protocol.
  • Spanish community add-ons catalog (protocol 3): not all add-ons created by the NVDA community can be found on community add-ons website. Therefore, a group of volunteers from Spain are maintaining a database of add-on metadata (accessed via nvda.es website) containing information on add-ons not hosted on community add-ons website. Unlike other protocols, a JSON-based list is returned, with each entry recording add-on information under an ID (integer) rather than add-on name (string). Supporting this catalog became the basis for introducing protocols.

Users can select between protocols 2 and 3 when checking for add-on update sources.

Core add-on mechanics

Startup and shutdown

In order for Add-on Updater to work, NVDA must be run in binary mode. In fact, this is one of the checks performed when starting the add-on (specifically, global plugin). Once this and security mode passes (Add-on Updater will not work in secure mode), the add-on will add varous user interface elements such as the update check entry in NVDA tools memu and Add-on Updater settings panel.

Once the user interface is ready, settings and other internal flags are loaded, then it will check if automatic update check timer must be run. If it has been more than 24 hours since last update check, the auto-check timer will run, eventually calling addonUpdateProc.checkForAddonUpdates function to coordinate update check processes.

At shutdown, Add-on Updater will remove the user interface, terminate auto-check timer if running, then save settings.

Add-on update check process

Add-on Updater can either check for add-on updates automatically or manually. Manual update check is initiated by the user by selecting "Check for add-on updates" item in NVDA tools menu. Once activated, Add-on Updater will turn off automatic update check timer if running, call update check function found in add-on update proc module, then present a dialog listing available add-on updates. While checking for updates, a progress dialog is displayed to inform the user about the overall progress.

In contrast, automatic update check uses a timer that calls update check routine. Once a day, a timer is awakened to call the update check routine, schedule the next automatic check, then goes to sleep. No progress dialog is shown to users, but update results dialog or a toast (Windows 10 and later) will appear if updates are available.

The core routine used for add-on update check is addonUpdateProc.checkForAddonUpdates. This function, formerly housed in extended add-on handler module, is responsible for gathering a list of installed add-ons, asking various update check protocols for update information, then presenting results if any. When called from either automatic or manual update check mode, it performs the following:

  1. Gathers a list of installed add-ons with specific conditions applied. While most add-ons can be updated, Add-on Updater will not check for add-ons if one or more of the following are true:
    • Updater add-ons: some add-ons include update check facility. As of time of this article, these include Braille Extender and Weather Plus.
    • Update check is disabled: if users inform Add-on Updater to not check for specific add-ons, these will be skipped.
  2. Sets update channels. This is done by checking for 'updateChannel' key in add-on manifests, and if defined, sets appropriate update channels.
  3. Builds preliminary add-on update records. Information from add-ons such as installed version and channel are used to define update record for each add-on. At this time, update version is set to installed version so update check protocols can specify the actual update version later.
  4. Calls checkForAddonUpdates method defined in the user configured update check protocol, passing in the just built update records, expecting to receive an updated add-on update records with update information filled in.
  5. The chosen update check protocol's chekcForAddonUpdates method calls chekcForAddonUpdate method, the actual core update check method for each protocol. Taking a list of update records, this method will retrieve add-ons metadata from an update source with help from urllib package, ask another method (fetchAddonInfo) to check update eligibility, then return an updated list of update records containing update version information. As part of this method, a json representation of the add-on update source database (metadata) accessed by each protocol is defined for use by fetchAddonInfo method.
  6. fetchAddonInfo checks update eligibility. checkForAddonUpdate function uses threads to call fetchAddonInfo, with each thread taking in the add-on to be examined and add-ons metadata obtained from the update source earlier.
  7. fetchAddonInfo threads are run. First, fetchAddonInfo checks if add-on metadata contains definitions for the add-on to be checked, and if yes, NVDA compatibility is checked. If either of these fail, Add-on Updater leaves fetchAddonInfo, keeping installed and update versions the same i.e. no updates.
  8. Retrieves version and update URL from different sources. Not all protocols and hence metadata include update version and URL as part of the metadata. If they are missing, fetchAddonInfo will retrieve actual update version and URL by accessing the pointer URL from add-ons metadata. If successful, version and URL are updated (for some, this involves a regular expression check to obtain actual version text).
  9. Update version and URL are updated. In older add-on releases, version equality check (string comparison) was performed, and update information was defined if installed and update versions differ. Now it is update record objects that perform actual update availability checks (string comparison).
  10. Back in checkForAddonUpdate method, list comprehension is used to build a list of add-on update records with available updates (AddonUpdateRecord.updateAvailable is True) and is passed back to checkForAddonUpdates method from the protocol, which in turn is retrieved by the overall checkForAddonUpdates function from add-on update proc module.

If add-on updates are available, manual update check route displays add-on updates dialog, and automatic update check route displays a toast message informing users of available updates.

Add-on update download and installation

After the user selects add-ons to be updated or a toast is displayed depending on background update setting, Add-on Updater will download and install selected add-on updates. If this is a manual update process (add-on updates dialog), a progress dialog is shown while downloading and installing add-ons. The process followed include:

  1. Add-ons are downloaded serially. In case of background updates, if the add-on to be downloaded is disabled by users, it is skipped, otherwise a temporary file path is generated to save add-on package contents.
  2. addonUpdateProc.downloadAddonUpdate is called wrapped inside a try block, taking in download URL, temp path, and hash value if any (hash is not checked as of time of this article). The try block is used to catch errors from the function such as runtime error and inability to access the given dowlnoad URL.
  3. If download was successful, temp path and add-on update record for the add-on is added to a list of downloaded add-ons. If this is a manual update check, download progress is updated.
  4. Successfully downloaded add-ons are installed serialy. For each add-on, addonUpdateProc.installAddonUpdate is called, passing in temp path used to house package contents and add-on summary. In return, the just called function returns a value from add-on install status enumeration, indicating install success or failure reasons (compatibility issues, generic error while installing add-on, etc.). A counter is used to keep track of successful install count.
  5. If successfully installed, install counter is incremented, otherwise error is recorded. If update progress is showing, an appropriate error message will be displayed depending on install error reason.
  6. If at least one add-on is updated, Add-on Updater will either prompt users to review updates or ask to restart NVDA. If reviewing add-on updates, closing the resulting add-on updates dialog will prompt users to restart NVDA.

Additional notes

Transforming Add-on Updater into an actual NVDA pull request

In some ways, Add-on Updater did the opposite: turn a pull request into an add-on. With some tweaking, it is possible to turn this add-on back into an NVDA pull request.

Why create a proof of concept/prototype

This was done so users can test an NVDA feature idea widely so problems can be identified early. As noted in add-on internals article for Windows App Essentials, parts of some add-ons are included in NVDA. In the case of Add-on Updater, the idea is to get the entire add-on to be included in NVDA. There is a precedent for it: screen curtain, introduced in NVDA 2019.3, was once an add-on before merged into NVDA. By letting users test ideas widely, it allows NV Access and contributors to see the value of an external feature.

Why are there communities hosting additional add-on metadata?

Add-ons registerd on community add-ons website are only a part of the entire add-ons ecosystem, numbering hundreds of add-ons. Additional add-ons include speech synthesizers, braille display support, add-ons for specific language communities, and others not registred on the community add-ons website for various reasons.

To host add-ons not listed on community add-ons website, several NVDA communities around the world host databases of add-ons metadata. One of the most extensive collection of add-ons can be found on Spanish NVDA community website (nvda.es, sometimes known as nvda-addons.org). Using a list of add-on Id's, this database hosts add-ons developed by Spanish-speaking NVDA users in addition to ones found in community add-ons website. In addition to the website, the Spanish NVDA community also developed Tienda (Store), an add-on designed to download add-ons from the Spanish community add-ons catalog.

The idea of letting Add-on Updater access sources other than community add-ons website wasn't in my mind until July 2022. While designing update check protocol class, I realized that there was a possibility of supporting different add-on update sources through subclassing. Therefore, as a way to test the protocols concept and to let users access updates for more add-ons, I implemented the client side of Spanish community add-ons catalog access from Add-on Updater as part of Project Meteor (Spanish community add-ons catalog is maintained by Spanish-spekaing NVDA community). With this work, it is now possible to support other add-on update sources in addition to community add-ons website which remains the primary add-on update source.

Conclusion

Add-ons brought many possibilities to the NVDA community: new features, synthesizers, and even bug fixes. But perhaps it brought an unexpected tool: an add-on to update add-ons, including itself. This is the primary goal of Add-on Updater, and despite feature additions and changes, this core goal remains the same.

In 2022, I announced my retirement from active contributions to NVDA community. But I realized that I needed something to keep me in touch with the community, so I decided to maintain add-ons. One of them is Add-on Updater, and this add-on is one of the last links to the community, specifically to NVDA screen reader source code. As I noted above, Add-on Updater's lifespan is tied to NVDA issue 3208 being addressed, and the add-on will be declared end of life the day a stable version of NVDA with add-on update feature is released. Until then, Add-on Updater will continue to change to meet the needs of the NVDA community, to keep up with NVDA source code, and to reflect changes to the add-ons ecosystem while checking for, downloading, and updating add-ons.

References