Handle incompatible add-ons when upgrading/downgrading NVDA API versi…

…ons (#15439)

Fixes #15437
Fixes #15412
Fixes #15414

Summary of the issue:
There are several scenarios which need to be supported when updating/downgrading NVDA with incompatible add-ons

Test Name	Upgrade from	Upgrade to	Test notes
Upgrade to different NVDA version in the same API breaking release cycle	X.1	X.1	Add-ons which remain incompatible are listed as incompatible on upgrading. Preserves state of enabled incompatible add-ons
Upgrade to a different but compatible API version	X.1	X.2	Add-ons which remain incompatible are listed as incompatible on upgrading. Preserves state of enabled incompatible add-ons
Downgrade to a different but compatible API version	X.2	X.1	Add-ons which remain incompatible are listed as incompatible on downgrading. Preserves state of enabled incompatible add-ons
Upgrade to an API breaking version	X.1	(X+1).1	All incompatible add-ons are listed as incompatible on upgrading, overridden compatibility is reset.
Downgrade to an API breaking version	(X+1).1	X.1	Add-ons which remain incompatible listed as incompatible on downgrading. Preserves state of enabled incompatible add-ons. Blocked add-ons which are now compatible are re-enabled.

Description of user facing changes
NVDA will reset compatibility overrides when updating to a different API breaking release, this means incompatible add-ons will be blocked again.
If an add-on is blocked due to compatibility and becomes compatible, e.g. via downgrading, it will be re-enabled.

Description of development approach
Store the BACK_COMPAT_TO version in the addon state pickle.
When updating the BACK_COMPAT_TO version, reset the incompatibility override state.

source/_addonStore/models/status.py

@@ -72,8 +72,8 @@ class AvailableAddonStatus(DisplayStringEnum):
 	INCOMPATIBLE_DISABLED = enum.auto()  # disabled due to being incompatible
 	PENDING_DISABLE = enum.auto()  # disabled after restart
 	DISABLED = enum.auto()
-	PENDING_INCOMPATIBLE_ENABLED = enum.auto()  # overriden incompatible, enabled after restart
-	INCOMPATIBLE_ENABLED = enum.auto()  # enabled, overriden incompatible
+	PENDING_INCOMPATIBLE_ENABLED = enum.auto()  # overridden incompatible, enabled after restart
+	INCOMPATIBLE_ENABLED = enum.auto()  # enabled, overridden incompatible
 	PENDING_ENABLE = enum.auto()  # enabled after restart
 	ENABLED = enum.auto()  # enabled but not running (e.g. all add-ons are disabled).
 	RUNNING = enum.auto()  # enabled and active.

source/_addonStore/models/version.py

@@ -61,10 +61,10 @@ def enableCompatibilityOverride(self):
 		and when this add-on is updated, disabled or removed.
 		"""
 		from addonHandler import AddonStateCategory, state
-		overiddenAddons = state[AddonStateCategory.OVERRIDE_COMPATIBILITY]
-		assert self.name not in overiddenAddons, f"{self.name}, {overiddenAddons}"
+		overriddenAddons = state[AddonStateCategory.OVERRIDE_COMPATIBILITY]
+		assert self.name not in overriddenAddons, f"{self.name}, {overriddenAddons}"
 		assert self.canOverrideCompatibility
-		overiddenAddons.add(self.name)
+		overriddenAddons.add(self.name)
 		state[AddonStateCategory.BLOCKED].discard(self.name)
 		state[AddonStateCategory.DISABLED].discard(self.name)
 

source/addonHandler/__init__.py

@@ -28,6 +28,7 @@
 	Set,
 	TYPE_CHECKING,
 	Tuple,
+	Union,
 )
 import zipfile
 from configobj import ConfigObj
@@ -43,7 +44,7 @@
 from types import ModuleType
 
 from _addonStore.models.status import AddonStateCategory, SupportsAddonState
-from _addonStore.models.version import SupportsVersionCheck
+from _addonStore.models.version import MajorMinorPatch, SupportsVersionCheck
 import extensionPoints
 from utils.caseInsensitiveCollections import CaseInsensitiveSet
 
@@ -96,6 +97,7 @@ def _generateDefaultStateContent() -> Dict[AddonStateCategory, CaseInsensitiveSe
 		}
 
 	data: Dict[AddonStateCategory, CaseInsensitiveSet[str]]
+	manualOverridesAPIVersion: MajorMinorPatch
 
 	@property
 	def statePath(self) -> os.PathLike:
@@ -106,14 +108,16 @@ def load(self) -> None:
 		"""Populates state with the default content and then loads values from the config."""
 		state = self._generateDefaultStateContent()
 		self.update(state)
+
+		# Set default value for manualOverridesAPIVersion.
+		# The ability to override add-ons only appeared in 2023.2,
+		# where the BACK_COMPAT_TO API version was 2023.1.0.
+		self.manualOverridesAPIVersion = MajorMinorPatch(2023, 1, 0)
+
 		try:
 			# #9038: Python 3 requires binary format when working with pickles.
 			with open(self.statePath, "rb") as f:
-				pickledState: Dict[str, Set[str]] = pickle.load(f)
-				for category in pickledState:
-					# Make pickles case insensitive
-					state[AddonStateCategory(category)] = CaseInsensitiveSet(pickledState[category])
-				self.update(state)
+				pickledState: Dict[str, Union[Set[str], addonAPIVersion.AddonApiVersionT]] = pickle.load(f)
 		except FileNotFoundError:
 			pass  # Clean config - no point logging in this case
 		except IOError:
@@ -122,8 +126,34 @@ def load(self) -> None:
 			log.debugWarning("Failed to unpickle state", exc_info=True)
 		except Exception:
 			log.exception()
+		else:
+			# Load from pickledState
+			if "backCompatToAPIVersion" in pickledState:
+				self.manualOverridesAPIVersion = MajorMinorPatch(*pickledState["backCompatToAPIVersion"])
+			for category in AddonStateCategory:
+				# Make pickles case insensitive
+				state[AddonStateCategory(category)] = CaseInsensitiveSet(pickledState.get(category, set()))
+
+		if self.manualOverridesAPIVersion != addonAPIVersion.BACK_COMPAT_TO:
+			log.debug(
+				"BACK_COMPAT_TO API version for manual compatibility overrides has changed. "
+				f"NVDA API has been upgraded: from {self.manualOverridesAPIVersion} to {addonAPIVersion.BACK_COMPAT_TO}"
+			)
+		if self.manualOverridesAPIVersion < addonAPIVersion.BACK_COMPAT_TO:
+			# Reset compatibility overrides as the API version has upgraded.
+			# For the installer, this is not written to disk.
+			# Portable/temporary copies will write this on the first run.
+			# Mark overridden compatible add-ons as blocked.
+			state[AddonStateCategory.BLOCKED].update(state[AddonStateCategory.OVERRIDE_COMPATIBILITY])
+			# Reset overridden compatibility for add-ons that were overridden by older versions of NVDA.
+			state[AddonStateCategory.OVERRIDE_COMPATIBILITY].clear()
+		self.manualOverridesAPIVersion = MajorMinorPatch(*addonAPIVersion.BACK_COMPAT_TO)
+		self.update(state)
 
 	def removeStateFile(self) -> None:
+		if not NVDAState.shouldWriteToDisk():
+			log.debugWarning("NVDA should not write to disk from secure mode or launcher", stack_info=True)
+			return
 		try:
 			os.remove(self.statePath)
 		except FileNotFoundError:
@@ -138,16 +168,17 @@ def save(self) -> None:
 			return
 
 		if any(self.values()):
+			# We cannot pickle instance of `AddonsState` directly
+			# since older versions of NVDA aren't aware about this class and they're expecting
+			# the state to be using inbuilt data types only.
+			picklableState: Dict[str, Union[Set[str], addonAPIVersion.AddonApiVersionT]] = dict()
+			for category in self.data:
+				picklableState[category.value] = set(self.data[category])
+			picklableState["backCompatToAPIVersion"] = self.manualOverridesAPIVersion
 			try:
 				# #9038: Python 3 requires binary format when working with pickles.
 				with open(self.statePath, "wb") as f:
-					# We cannot pickle instance of `AddonsState` directly
-					# since older versions of NVDA aren't aware about this class and they're expecting
-					# the state to be using inbuilt data types only.
-					pickleableState: Dict[str, Set[str]] = dict()
-					for category in self.data:
-						pickleableState[category.value] = set(self.data[category])
-					pickle.dump(pickleableState, f, protocol=0)
+					pickle.dump(picklableState, f, protocol=0)
 			except (IOError, pickle.PicklingError):
 				log.debugWarning("Error saving state", exc_info=True)
 		else:
@@ -166,6 +197,24 @@ def cleanupRemovedDisabledAddons(self) -> None:
 				log.debug(f"Discarding {disabledAddonName} from disabled add-ons as it has been uninstalled.")
 				self[AddonStateCategory.DISABLED].discard(disabledAddonName)
 
+	def _cleanupCompatibleAddonsFromDowngrade(self) -> None:
+		from _addonStore.dataManager import addonDataManager
+		installedAddons = addonDataManager._installedAddonsCache.installedAddons
+		for blockedAddon in CaseInsensitiveSet(
+			self[AddonStateCategory.BLOCKED].union(
+				self[AddonStateCategory.OVERRIDE_COMPATIBILITY]
+			)
+		):
+			# Iterate over copy of set to prevent updating the set while iterating over it.
+			if blockedAddon not in installedAddons and blockedAddon not in self[AddonStateCategory.PENDING_INSTALL]:
+				log.debug(f"Discarding {blockedAddon} from blocked add-ons as it has been uninstalled.")
+				self[AddonStateCategory.BLOCKED].discard(blockedAddon)
+				self[AddonStateCategory.OVERRIDE_COMPATIBILITY].discard(blockedAddon)
+			elif installedAddons[blockedAddon].isCompatible:
+				log.debug(f"Discarding {blockedAddon} from blocked add-ons as it has become compatible.")
+				self[AddonStateCategory.BLOCKED].discard(blockedAddon)
+				self[AddonStateCategory.OVERRIDE_COMPATIBILITY].discard(blockedAddon)
+
 
 state: AddonsState[AddonStateCategory, CaseInsensitiveSet[str]] = AddonsState()
 
@@ -188,8 +237,16 @@ def getIncompatibleAddons(
 				addon,
 				currentAPIVersion=currentAPIVersion,
 				backwardsCompatToVersion=backCompatToAPIVersion
+			)
+			and (
+				# Add-ons that override incompatibility are not considered incompatible.
+				not addon.overrideIncompatibility
+				# If we are upgrading NVDA API versions,
+				# then the add-on compatibility override will be reset
+				or backCompatToAPIVersion > addonAPIVersion.BACK_COMPAT_TO
+			)
 		)
-	))
+	)
 
 
 def removeFailedDeletion(path: os.PathLike):
@@ -206,7 +263,7 @@ def disableAddonsIfAny():
 	# Pull in and enable add-ons that should be disabled and enabled, respectively.
 	state[AddonStateCategory.DISABLED] |= state[AddonStateCategory.PENDING_DISABLE]
 	state[AddonStateCategory.DISABLED] -= state[AddonStateCategory.PENDING_ENABLE]
-	# Remove disabled add-ons from having overriden compatibility
+	# Remove disabled add-ons from having overridden compatibility
 	state[AddonStateCategory.OVERRIDE_COMPATIBILITY] -= state[AddonStateCategory.DISABLED]
 	# Clear pending disables and enables
 	state[AddonStateCategory.PENDING_DISABLE].clear()
@@ -222,8 +279,9 @@ def initialize():
 	# #3090: Are there add-ons that are supposed to not run for this session?
 	disableAddonsIfAny()
 	getAvailableAddons(refresh=True, isFirstLoad=True)
+	state.cleanupRemovedDisabledAddons()
+	state._cleanupCompatibleAddonsFromDowngrade()
 	if NVDAState.shouldWriteToDisk():
-		state.cleanupRemovedDisabledAddons()
 		state.save()
 	initializeModulePackagePaths()
 

tests/manual/addonStore.md

@@ -203,3 +203,23 @@ For "Action" button and "Other details" text field controls:
 1. Tab to another control and check that `alt+<letter>` allows to move the focus back to the control.
 1. From another control tab to the control and check that `alt+<letter>` is reported.
 1. In the control, check that `shift+numpad2` reports the shortcut key.
+
+## Updating NVDA
+
+### Updating NVDA with incompatible add-ons
+
+There are several scenarios which need to be tested for updating NVDA with incompatible add-ons.
+This is an advanced test scenario which requires 3 versions of NVDA to test with.
+Typically, this requires a contributor creating 3 different versions of the same patch of NVDA, with different versions of `addonAPIVersion.CURRENT` and `addonAPIVersion.BACK_COMPAT_TO`
+- X.1 e.g `CURRENT=2023.1`, `BACK_COMPAT_TO=2023.1`
+- X.2 e.g `CURRENT=2023.2`, `BACK_COMPAT_TO=2023.1`
+- (X+1).1 e.g `CURRENT=2024.1`, `BACK_COMPAT_TO=2024.1`
+
+
+| Test Name | Upgrade from | Upgrade to | Test notes |
+|---|---|---|---|
+| Upgrade to different NVDA version in the same API breaking release cycle | X.1 | X.1 | Add-ons which remain incompatible are listed as incompatible on upgrading. Preserves state of enabled incompatible add-ons |
+| Upgrade to a different but compatible API version | X.1 | X.2 | Add-ons which remain incompatible are listed as incompatible on upgrading. Preserves state of enabled incompatible add-ons |
+| Downgrade to a different but compatible API version | X.2 | X.1 | Add-ons which remain incompatible are listed as incompatible on upgrading. Preserves state of enabled incompatible add-ons |
+| Upgrade to an API breaking version | X.1 | (X+1).1 | All incompatible add-ons are listed as incompatible on upgrading, overridden compatibility is reset. |
+| Downgrade to an API breaking version | (X+1).1 | X.1 | Add-ons which remain incompatible listed as incompatible on upgrading. Preserves state of enabled incompatible add-ons. Add-ons which are now compatible are re-enabled. |

user_docs/en/changes.t2t

@@ -80,6 +80,7 @@ There's also been bug fixes for the Add-on Store, Microsoft Office, Microsoft Ed
   - Fixed bug preventing overridden enabled incompatible add-ons being upgraded or replaced using the external install tool. (#15417)
   - Fixed bug where NVDA would not speak until restarted after add-on installation. (#14525)
   - Fixed bug where add-ons cannot be installed if a previous download failed or was cancelled. (#15469)
+  - Fixed issues with handling incompatible add-ons when upgrading NVDA. (#15414, #15412, #15437)
   -
 - Fixed support for System List view (``SysListView32``) controls in Windows Forms applications. (#15283)
 - NVDA once again announces calculation results in the Windows 32bit calculator on Server, LTSC and LTSB versions of Windows. (#15230)