Marketplace Design

Bolt Marketplace - Technical Design

Introduction

For the Bolt 2.0 release, we want to introduce a "Market Place" for Bolt extensions and themes (further referred to as "extensions" unless indicated otherwise), comparable to the Android/Apple/... app stores, PHP's Packagist, and the plugin repositories for platforms like Drupal or WordPress.

Inspiration

• http://www.sache.in/
• http://flysystem.thephpleague.com/
• Newscoop PluginsController
• http://www.edit4u.nl/nieuws/20-joomla-3-2-infographic

Requirements

One centralized index for extensions

We have explicitly decided not to offer extension hosting; we merely provide a centralized index that references extensions hosted elsewhere, as well as the required infrastructure to install and manage these. The term "Marketplace" is meant to refer to this centralized index service.

A web-based public extension browser

We want people to be able to browse all the indexed extensions independently from Bolt itself; this means we need a public-facing web-based front-end to the extension repository.

Extension developers must be able to add and update extensions

We need to provide a straightforward way of adding extensions and updates to developers.

Package format must follow KISS

• Metadata must be provided in a human-readable format (preferred: YAML).
• The toolset required to build a package must be kept minimal (preferred: just a text editor and a zip archiver).
• Boilerplate is to be avoided / abstracted away.

Install and manage extensions in the Bolt backend

Many users are restricted to FTP and HTTP access; Bolt needs to be able to manage marketplace extensions through the web backend alone.

An extension browser that integrates with the Bolt backend

For the best possible UX, we want an extension browser inside the Bolt backend, which allows users to install extensions into their Bolt setup directly from the marketplace browser.

Install and manage extensions with Nut

For users that do have shell access and prefer using Nut to manage their Bolt install, it must be possible to install and manage extensions directly from the marketplace using Nut.

Install extensions manually

Installing extensions manually and/or from sources other than the marketplace must remain possible.

Trust Management

Cryptographically sound methods must be used to manage trust. This includes the following features:

1. Verify the identity of the Marketplace server
2. Verify the integrity of the Marketplace index and metadata
3. Verify the integrity of any downloaded extension packages
4. Verify the identity of the package's author
5. Verify the integrity of any downloaded extension packages independently from the Marketplace
6. Verify the identity of the package's author independently from the Marketplace
7. Avoid "throwaway" developer accounts

These features are all necessary to allow for meaningful social control and enforce it technically. Points 5 and 6 require a source of trust outside the both the Marketplace and the packages themselves.

Automatic update checks

Bolt should be able to automatically check for updates, and provide an easy way of installing them. Available updates should be listed in the Bolt backend.

Integrate with popular source control hosting services

The vast majority of (free) extensions is going to be hosted on github, bitbucket, or a similar service. Some basic integration with these services would be a useful feature.

Functional Use Cases

1.1. Create a developer account

1. Developer creates a Marketplace account.
2. Developer creates a private/public key pair for signing.
3. Developer links public key to Marketplace account.
4. Marketplace verifies developer identity and signs public key with Bolt key.

1.2. Create and register a package

1. Developer creates a package and metadata in the correct package format.
2. Developer signs metadata file.
3. Developer exposes metadata file through a publicly-accessible URL.
4. Developer signs into Marketplace and registers the package through a web interface, providing the URL of the metadata file.
5. Marketplace downloads metadata file, verifies signature, and updates index.

1.3. Push a package update

1. Developer creates new versions of package and metadata.
2. Developer signs metadata file.
3. Marketplace crawls known packages on a regular basis, verifies signatures, and updates index as required.
4. For each index entry that it has modified, Marketplace sends an e-mail message to the maintainer.

1.4. Delete a package

1. Developer signs into Marketplace and deletes the package using a web interface.

2.1. Browse available packages

1. User visits public Marketplace web interface, enters search terms, browses categories, etc.

2.2. Manually install package

1. User finds package in public Marketplace web interface.
2. User clicks 'download' link and receives a zip file.
3. User checks zip file integrity by checking the included signature (optional)
4. User unpacks zip file in the appropriate location.

2.3 Semi-automatic install using Nut

1. User finds package in public Marketplace browser.
2. User copies metadata URL from the web interface.
3. User pastes metadata URL into a Nut command
4. Nut downloads package, verifies
5. User confirms install.
6. Nut unpacks and installs.

2.4 Semi-automatic install using Bolt

1. User finds package in public Marketplace browser.
2. User copies metadata URL from the web interface.
3. User pastes metadata URL into Bolt's backend.
4. Bolt downloads and verifies package.
5. User confirms install.
6. Bolt unpacks and installs.

2.5 Automatic install using Bolt

1. User opens Marketplace browser from within Bolt backend.
2. User finds package in Marketplace browser.
3. User clicks 'Install Package'
4. Bolt downloads and verifies package.
5. User confirms install.
6. Bolt unpacks and installs.

3.1 Show available updates in Bolt

1. User opens Bolt backend
2. Bolt shows a list of available package updates

3.2 Update packages using Bolt

1. User opens Bolt backend.
2. Bolt shows a list of available package updates.
3. User selects packages to update.
4. Bolt downloads updates, verifies, installs.

Technical Design

System Components

1. Package Format Specification

A package consists of three parts:

1. The package itself, a .zip file containing all the required files. Paths are relative to Bolt's extensions or themes directory, depending on the type of the extension.
2. A metadata file in YAML format. This file contains at least the following information:
   • A human-readable package name (e.g. "Joe's Super Extension").
   • The unique Marketplace name for the extension (e.g. "joe-super").
   • A version number, following a somewhat semantic numbering scheme.
   • The package author (Marketplace username, e.g. "joe").
   • Author/maintainer e-mail address (e.g. "joe@superduper.org").
   • The package type ("theme" or "extension")
   • A download URL for a .zip file containing the actual package.
   • A secure hash of the .zip package.
3. A signature file. This file contains a signature for the metadata file, created with the specified developer's public key.

2. Marketplace Index Server

The index server is a database-backed web service with a web crawler.

3. Package Browser Front-End

A web-based read-only client for the Index Server that exposes the following functionality:

• Search packages
• Display package meta-information
• Display package verification status
• Provide download links to the package itself

4. Package Browser Front-End In Bolt Backend

The same as 3., but additionally, this version provides integration with the Bolt backend. This could work as follows:

• When opening the package browser from within the backend, Bolt adds some extra information to the redirect: at least the base URL of the current Bolt install, possibly some information about currently installed packages.
• The package browser displays 'install now' links where appropriate.
• Any 'install now' link redirects to a confirmation page in the Bolt backend, attaching a package identifier to the request.

5. Developer Web Front-End

A web-based read/write client for the Index Server, providing the same functionality as the Package Browser, plus the following:

For developers:

• Developer login/logout
• Create and manage developer accounts
• Manage developer keys:
  • Upload own pubkey
  • Request pubkey signing
  • Blacklist own pubkey
  • Delete own pubkey
• Manage extensions per developer
  • Register extension
  • View extension verification status
  • Unregister extension
  • Modify extension data (metadata URL)
  • Request revisiting by spider

For administrators:

• Administrator login/logout
• Manage developer accounts (create, modify, delete, block/unblock)
• Manage developer keys:
  • Upload pubkey
  • Sign pubkey
  • Blacklist pubkey
  • Delete pubkey
• Manage extensions
  • Register extension
  • View extension verification status
  • Unregister extension
  • Modify extension data (metadata URL)
  • Request revisiting by spider

5. Nut

Package installation features

• app/nut extensions:install http://example.org/bolt-extensions/foobar/extension.yml (fetch the extension metainformation from the specified URL, verify it, download the extension, verify and install)
• app/nut extensions:install marketplace://tdammers/foobar or shorthand: app/nut extensions:install tdammers/foobar (fetch extension metainformation for 'foobar' by developer 'tdammers' from the marketplace, verify it, download the extension, verify and install)
• app/nut extensions:delete tdammers/foobar (delete the 'foobar' extension by developer 'tdammers'.
• app/nut extensions:update
• app/nut extensions:list (display a list of installed packages, including author, name, version, description, validity, download URL, and available updates)
• app/nut extensions:search 'search query' (send a search query to the marketplace index and list the results)

6. Bolt Backend

Integration with the package browser:

• A menu option that opens the package browser with the required metadata.
• A return page that reads a package identifier from the request, fetches the package from the marketplace, verifies it, and if the user confirms, installs it.

List of currently installed packages

• Sorted by up-to-date and updates-available
• Updatable packages come with one-click update

Updating packages from within Bolt

Because many shared-hosting installs run the web interface under a different user than the owner of the code files, we may have to resort to some tricks to successfully update. Strategies are:

1. Copy files locally. This works if web server and shell account run under the same user.
2. Configure FTP credentials, and use an FTP connection to localhost to perform the copying.
3. Configure SSH credentials, and use an SSH connection to localhost to perform the copying.
4. (to be researched) Use available PHP functionality to impersonate the local shell user from within PHP, using configured credentials.
5. chmod 777

Package updates have to be atomic; this is generally the case, but even more important in the web-based installation scenario, due to PHP execution constraints. Most importantly, Bolt must not break in any way when a package installation crashes on RAM limit or execution timeout.

At the same time, we need to make sure that existing configuration files remain intact; because of this, we cannot simply swap out the old extension for the new one.

To solve this, we'll use the atomicity of the mv operation; the procedure goes like this:

1. Download foobar.zip, verify.
2. Unpack foobar.zip to a temporary directory, e.g. /tmp/foobar/
3. Check if app/extensions/foobar exists (if it doesn't, a previous install failed and we need to roll back or finish this pending install)
4. Delete app/extensions/foobar.old if it exists
5. Copy existing extension directory app/extensions/foobar to app/extensions/foobar.new
6. Recursively copy /tmp/foobar over app/extensions/foobar; this will keep the existing configuration file.
7. Atomically move app/extensions/foobar to app/extensions/foobar.old and app/extensions/foobar.new to app/extensions/foobar.

This isn't 100% watertight; if the installation fails between the moves in step 7, we are left with app/extensions/foobar.old and app/extensions/foobar.new, but no app/extensions/foobar. Fortunately, Bolt will still work (albeit without the extension), and we detecte the situation. We can then ask the user to decide whether the installation should be rolled back (mv app/extensions/foobar.old app/extensions/foobar && rm app/extensions/foobar.new) or finished (mv app/extensions/foobar.new app/extensions/foobar).

Security considerations

• Communications between the Index Server and any privileged client (i.e., anything that can write to the index, and anything that downloads or verifies code, metadata, or keys) must use SSL. No exceptions.
• The package signing mechanism must be freely available, transparent, proven technology.
• For package and developer verification, code must be available in pure PHP; we cannot rely on anything but the PHP defaults to be installed on the systems running Bolt. Shelling out to GPG, for example, is not an option.
• The Bolt key is the root of our trust chain; it must be provided out-of-band. The most feasible way for this is to include it in the Bolt distribution.

Open Questions / To Be Researched

• Which signing algorithm? Requirements:
  • State-of-the-art security (cryptographically sound, strong keys)
  • Proven solutions - no reinventing square wheels, no experiments.
  • Open-source implementations. No black boxes.
  • A pure PHP implementation must be available for the signature verification process. Platform-specific solutions are not an option, because the verification must work on all target platforms.
• Web-based installs: how to get around problematic permission situations.
  • SSH from PHP - starting point: http://php.net/manual/en/book.ssh2.php. SSH to localhost to impersonate the shell user from within the web server process.
  • FTP from PHP - http://php.net/manual/en/book.ftp.php. This should do, but we need to decide whether we need a fallback pure-PHP implementation for systems that don't have the FTP extension installed.
  • Credentials: decide whether to store SSH/FTP credentials in the Bolt config or prompt the user as needed.