Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve and refactor documentation #109

Merged
merged 11 commits into from
Mar 7, 2023
Merged

Improve and refactor documentation #109

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
b3c33b9
Improve and refactor documentation
seanbudd Mar 2, 2023
ff041f5
fix md
seanbudd Mar 2, 2023
6e0a318
fix md
seanbudd Mar 2, 2023
dd2ac8a
fix up links
seanbudd Mar 2, 2023
81eed31
remove apostrophes
seanbudd Mar 2, 2023
32c8f2b
Apply suggestions from code review
seanbudd Mar 6, 2023
a6c146d
Apply suggestions from code review
seanbudd Mar 6, 2023
9905a3b
improve docs on channels
seanbudd Mar 6, 2023
799d090
Merge remote-tracking branch 'origin/master' into improveDocumentation
seanbudd Mar 7, 2023
f4bd5ed
Use issue form for submission
seanbudd Mar 7, 2023
c5ccd92
fix up instructions
seanbudd Mar 7, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
220 changes: 16 additions & 204 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,220 +1,32 @@
# Add on Store
# Add-on Store

The intention the add-on store is to improve the end-to-end process and supporting infrastructure for hosting add-ons.
This is the current plan from NV Access, work is ongoing.
Inspiration has been taken from [Windows Package Manager Community repo](https://github.com/microsoft/winget-pkgs)
A highly automated process would be ideal, and we can keep the ideas in mind to work in that direction.
The addon-datastore repository is a data pipeline of submitting, validating and transforming add-on data to views.
These views are hosted on the NV Access server and will become available in the NVDA add-on store.

Goal: enable an "NVDA add-on store" accessible from within NVDA itself.
In this proposal the word "store" is used to refer to the system used to store metadata about add-on releases and APIs to access this data.
Aims:
- Enable any necessary, API, process, or infrastructure to support users to browse, search, install and update add-ons for NVDA.
- A secure and robust provision of addon-metadata.
- No intention of supporting paid add-ons at this stage.
- Transparent process, to make it simple (for developers) to understand the current state of the
Add-on Store, or the state of a submission of a new / updated add-on.
- Faster release process for add-ons, by-passing human review.
- Non-subjective review process for add-ons.
The add-on store is a planned NVDA feature, which allows browsing, installing and updating add-ons from within NVDA.
The current work in progress can be tracked in this pull request: https://github.com/nvaccess/nvda/pull/13985

### Guide for submitters
Add-on authors who wish to have their add-on distributed through the add-on store should refer to [the submission guide](./docs/submitters/submissionGuide.md).

### Design overview
For an overview of the whole add-on store, read [the design overview](./docs/submitters/designOverview.md).

### About security
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not move this paragraph into designOverview.md?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this affects more than developers. I believe submitters, developers and other community contributors would find this information helpful.

Ensuring that an add-on is safe to run is a difficult challenge that isn't addressed here.
However, the metadata for a new submission (add-on release) can be confirmed to match its manifest
description.
Additionally, add-on file integrity can be enforced via a Sha256 checksum.
Additionally, add-on file integrity can be enforced via a SHA256 checksum.
The checksum allows NVDA to ensure that add-on releases are immutable.

### Human review process / code audit
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would move this paragraph into submissionGuide.md, unless the last point that would fit in designOverview.md.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this information is more relevant to other parties than submitters: including developers, add-on reviewers, and other community members.

- NV Access doesn't require a manual review of the add-on (code or user experience) itself
before the add-on submission.
- Source code reviews or audits could exist outside the store.
The Sha256 checksum of the `nvda-addon` prevents undetected changes.
- NV Access doesn't require a manual review of the add-on (code or user experience) itself before the add-on submission.
- You are welcome to review code / UX of add-ons and provide that feedback directly to add-on authors.
- The SHA256 checksum of the `.nvda-addon` prevents undetected changes.
- User reviews/rating of add-ons are currently out of scope.

### Non-exclusivity
seanbudd marked this conversation as resolved.
Show resolved Hide resolved
This proposal does not intend to restrict add-on authors from developing, publishing, and distributing an add-on outside this store.
This system does not restrict add-on authors from developing, publishing, and distributing an add-on outside this store.
NVDA will still allow local installation from a `*.nvda-addon` file.
The data hosted here is distributed under the [ODC-PDDL](https://opendatacommons.org/licenses/pddl/1-0/) license.
A plain language summary can be found [here](https://opendatacommons.org/licenses/pddl/summary/).

## Too Long; Didn't Read for add-on authors
With this proposal if an add-on author wishes to submit their add-on to be visible in this add-on store they will need to:
- Copy and fill out the metadata template
- Create a pull request to merge the branch to master of this repository.

## Too Long; Didn't Read for add-on reviewers
Reviews of metadata will be automated.
You are welcome to review code / UX of add-ons and provide that feedback directly to add-on authors.

## Considerations

- Submissions and automated checks should be easy to find and get the status of.
- Make it possible to automate many steps in the process.
- The `.nvda-addon` file accessible via the download URL must continue to match the SHA.
NVDA will verify the file has not changed by comparing the checksum (SHA256).
This gives users certainty when installing a "known version" of an add-on.
To update or make changes to an add-on, a new unique URL should be used, and a new add-on
submission made to the Add-on Store.
- Allow add-on authors to easily revoke a version if it is buggy / no longer supported.
Removed releases are no longer presented in the store, halting new installations.
- Enable support in the store for multiple versions of an add-on, based on NVDA API version.
- EG add-on version 1.2.5 for NVDA 2019.3 and add-on version 1.3.2 for NVDA 2020.1
- Enable support in the store for 'beta' and 'dev' add-ons, for instance:
- 'dev' add-ons are being developed against alpha / beta / rc NVDA, these would only be offered to alpha / beta / rc NVDA users.
- 'beta' add-ons are from authors who want early feedback from end users, signaling that not all edge cases are handled.
- End users of NVDA can select "show pre-release add-ons"

## Overview

[Addon store system design diagram](./docs/design/designOverview.svg) ([PlantUML markup](./docs/design/designOverview.puml))

- Use GitHub pull requests for store submissions.
- Use GitHub for storage of meta-data for add-ons available from the store.
- Use GitHub actions (or other integrations) to automate construction of the data store, and as many of the metadata checks as possible.
This will all be open source and extensible by the community.

### Why GitHub Pull Requests?
- GitHub is where much of the NVDA development ecosystem is already based.
- Handles all the of the CRUD ([create, read, update, delete](https://en.wikipedia.org/wiki/CRUD))
for users, authentication is handled, and we can determine how a user relates to an add-on
repository and what their permissions are for that repository.
- GitHub PR's keep a record of the outcomes of automated checks.
They also facilitate discussion should there be any confusion or disagreement with the outcome.
- GitHub PR's provided an atomic view of a store submission.
- The status (open / merged / closed) of the PR is clear.

## Infrastructure

- `addon-datastore` GitHub Repository
- Authors submit new add-on versions.
- The "source of truth" for add-on releases.
- This repository acts as a back-end database, it is open and easy to inspect.
- Since our needs are simple, preconfigured "views" of the data will suffice.
- `nvaccess/addon-datastore-validation` GitHub Repository
- Metadata / submission schema.
- Tools used to validate the submission.
- NV Access server - To provide the endpoint for "available add-ons" meta-data
- While this is technically not necessary, it provides a good separation from implementation.
If we wished to change our storage mechanism, we would not be breaking old versions of NVDA.

## `addon-datastore` GitHub Repository

Essentially this repository holds metadata about all the accepted versions of add-ons which are included in the store.
Metadata about old versions of an add-on remains until it is explicitly removed or becomes invalid.
This allows delivery to older versions of NVDA.
If a newer add-on release is removed (in response to a critical bug being found) NVDA can fall back
on a prior add-on release.
Add-ons versions are submitted by submitting a pull request, adding a file for that version of the add-on.

### Layout

Root directory of repository:
- `readme.md` - A guide for submission
- `addons/addon1/majorVersion.minorVersion.patch.json`
- `addons/addon2/majorVersion.minorVersion.patch.json`
- `addons/addon3/majorVersion.minorVersion.patch.json`

Note: `addonName` is the add-on ID, and must be unique and match the add-on ID from the add-on manifest.

Example for the NV Access add-on, 'NVDA - OCR':
- Filename: `addons/nvda-ocr/1.6.0.json`
- add-on ID `nvda-ocr`

### Metadata format
For a full description of the schema see the
[_validate/addonVersion_schema.json file](https://github.com/nvaccess/addon-datastore-validation/blob/main/_validate/addonVersion_schema.json).
It includes an example of the file contents.

### Submitting an add-on version

#### Pre-requisites:
- Familiarity with GitHub
- Familiarity with Git, including working with branches.

Process to add a new NVDA add-on version:
1. Fork the `addon-datastore` repository
1. On a new branch, copy the `_template_addon_release.json` file.
- Rename / move the file to `addons/<addonID>/<version>.json`
- `<addonID>` is the ID of the add-on. This should match the `name` field in the add-on manifest, E.G. "nv-speech-player"
- `<version>` is the add-on version in the form: `Major.Minor.Patch` E.G. "2.4.1"
1. Fill out the template.
1. Create a PR to merge your branch into master on the `addon-datastore` repository
1. Automated checks for common issues will complete. Either giving feedback or merging the PR.
1. When the PR is merged the add-on becomes available in the store.

### Converting an add-on manifest to a valid submission:
1. Fork the `addon-datastore` repository
1. Install requirements to python environment:
`pip install requests configobj`
1. Convert an add-on hosted at `addonUrl` to a json file:
`python convertAddonManifest.py [addonId] [stable|beta|dev] [addonUrl]`
1. Edit the auto-generated json file as required.
1. Create a PR to merge your branch into master on the `addon-datastore` repository
1. Automated checks for common issues will complete. Either giving feedback or merging the PR.
1. When the PR is merged the add-on becomes available in the store.


### Automated checks
See https://github.com/nvaccess/addon-datastore-validation

### Other notes
- By using a git repository and PR process, `git blame` and `git log` can be used to get more
context about add-ons listed in the store.
For instance:
- When was the add-on accepted?
- What did the review look like?
- How often is the add-on updated?
- GitHub allows assigning reviews to reviewers

## API data generation details

Triggered by a new commit to the `master` branch, [a GitHub workflow](./.github/workflows/transformDataToViews.yml), [addon-datastore-transform](https://github.com/nvaccess/addon-datastore-transform), transforms the data into the required views.

For each NVDA API version and channel, the add-on metadata with the highest version number is written.

These views are then committed by the GitHub Action to the [views branch](https://github.com/nvaccess/addon-datastore/tree/views).

### Data views
The following views will only be available on the [views branch](https://github.com/nvaccess/addon-datastore/tree/views) and located in a `views` folder.
Required transformations of the data:
- `/NVDA API Version/addon-1-ID/stable.json`
- `/NVDA API Version/addon-1-ID/beta.json`
- `/NVDA API Version/addon-2-ID/stable.json`

Notes:
- 'NVDA API Version' will be something like '2019.3', there will be one folder for each NVDA API Version.
- The `beta.json` and `stable.json` contain the information necessary for a store entry.
- The contents for each add-on will include all the technical details required for NVDA to download, verify file integrity, and install.
- The file will include translations (if available) for the displayable metadata.

This simplifies the processing on the hosting (E.G NV Access) server.

## NV Access Add-on Store endpoints

Using the NV Access server as the endpoint for this is important in case the implementation has to change or be migrated away from GitHub for some reason.

### `GET` All
Get a complete list of the latest add-on releases.
Includes both stable and beta versions

- Format: `https://www.nvaccess.org/addonStore/<language>/all/<NVDA API Version>.json`
- Example: <https://www.nvaccess.org/addonStore/en/all/2021.2.0.json>

### `GET` All stable
Get a complete list of the latest stable add-on releases.
Includes only stable versions

- Format: `https://www.nvaccess.org/addonStore/<language>/stable/<NVDA API Version>.json`
- Example: <https://www.nvaccess.org/addonStore/en/stable/2021.2.0.json>

### `GET` All beta
Get a complete list of the latest beta add-on releases.
Includes only beta versions

- Format: `https://www.nvaccess.org/addonStore/<language>/beta/<NVDA API Version>.json`
- Example: <https://www.nvaccess.org/addonStore/en/beta/2021.2.0.json>

## Suffix

### Terminology: add-on version vs add-on release

Since this proposal supports beta addons, I have tried to avoid using the term "add-on release", instead favouring "add-on version".
Loading