Skip to content

Developing Offen

Frederik Ring edited this page Dec 11, 2019 · 2 revisions

Required tools

Developing Offen currently requires Docker and docker-compose to be installed and working correctly. Both tools are free and available for Linux, Windows and Mac.

If your system knows how to run make it will help a lot with common tasks like installing dependencies and building, yet you will still be able to work on Offen without it (this will likely be the case if you are developing on Windows).

If you are on an older version of Windows and want to use make, consider installing Git Bash, if you are using Windows 10, you could use the Windows Subsystem for Linux. Alternatively, you can look up the commands in the Makefile and resort to manually running them in sequence.

Programming languages

The server side application is written in Go while the client is written in plain JavaScript

Go

Go is currently at version 1.13 and all code can expect all features present in that version. Formatting happens using go fmt and go vet is enforced in CI. In case you would like to contribute to Offen, but haven't used Go before don't be scared. It is easy to pick up and has great learning resources, head over to the Go wiki if you're interested.

JavaScript

offen tries to keep its build setup simple, which is why transpilers like babel are not part of the build pipeline. In turn this means, Offen uses ES5 syntax only (with the exception of nanohtml template strings, see section below) so older browsers do not choke on the bundle and can fall back gracefully. Another reason for not using ES6+ syntax is keeping the codebase readable for non-technical people who are interested in what Offen is doing under the hood.

Code is following the Standard JS styleguide which is enforced in CI. Unit tests are being run using mochify.js which means you can access a native and up-to-date DOM API in your tests.

Tagged template literals and nanohmtml

An exception to the ES5-only rule is the use of tagged template literals when interacting with nanohtml. This code, however, will be compiled away into ES5 by its browserify transform, so the client will never receive these template literals in any way.

Setup

First run

After cloning the repository, you will need to do two things:

  1. Pull the required docker images and install each container's dependencies:
    make setup
    
  2. Bootstrap the development database:
    make bootstrap
    

You should now be able to start your local version of offen using:

make up

To see if the application is working as intended visit http://localhost:8080 in your browser or use the application's health check:

➜  ~ curl -X GET http://localhost:8080/healthz
{"ok":true}

You can stop the application using Ctrl+C, in case you want to tear down the current environment and delete the data stored in the local database you can use:

make down

Updating dependencies

In case the dependencies consumed by one of the containers have changed, you can run the following command to get an up-to-date version:

make update

Rebuilding Docker images

In case the Dockerfiles defining the development images have changes you will need to rebuild the local images using:

make dev-build

Running database migrations

If your local database schema is out of date and you need to apply the latest migrations, run

make migrate

Generating usage data for development

As all usage data is encrypted and bound to user keys it is not possible to script the generation of seed data used for development. In the default development setup, the index page at http://localhost:8080 deploys the Offen script using the "develop" account that is available using the development login. This means you can manually create usage data by simply visiting this index page.

Understanding the development setup

When built for production use, Offen builds into a single binary file that includes the server application, as well as all static assets like Stylesheets and JavaScript files that it serves. This is great for distribution, but it's a tedious process getting in the way of a rapid feedback cycle in development. This is why when running the development environment the following setup will be launched instead:

  • the server application starts and is routed as-is through an nginx reverse proxy. Package refresh is used to live reload the application on code changes
  • on top of the server application, the routes for the client side sub-applications (the auditorium, the vault and the script) are overridden by the nginx setup and routed to a live-reloading development version of these subapps

All routes served are identical to the compiled version so the development environment will behave just like a production one, only with automated rebuilding and faster feedback.

Running tests

To run the test of a single container, the easiest way is simply using docker-compose to execute the command inside the container, e.g.:

docker-compose run auditorium npm t

for client containers and

docker-compose run server make test

for the server application.

Running all tests for all containers can be done using:

make test

Handling localization

All of the subapplications use a "gettext"-style approach for handling localization of strings. This means, any user facing string should appear in the source code in english (which is the default locale), and wrapped using the __ function.

In JavaScript this would look like:

return html`
  <h1>${__('Welcome to Offen')}</h1>
`

and like this in a Go template:

<h1>{{ __ "Welcome to Offen" }}</h1>

Extracting strings

The current set of strings can be extracted for all supported locales using the top level make extract-strings command. Each subapp contains a locales folder containing the generated .po files.


Understanding the application architecture

Offen is composed of four major parts.

The script

In order to use Offen to collect usage data on a site, the script needs to embedded on each page that is supposed to be contained in the statistics. It is a simple script that will collect and transmit basic pageview data on pageload as well as on pushstate events.

The vault

Offen aims to protect all of the data it is handling from 3rd party scripts that might be running on the same host page as the script. This is why the vault acts as intermediate layer between client side applications like the script and the auditorium, leveraging the protection offered by the Same-Origin-Policy.

The vault is a basic HTML document that is supposed to be loaded in an invisible iframe by any application that wants to use it to interact with the server. It runs a script that is listening for postMessage messages emitted by its host document. It is also able to query for data - either in the local IndexedDB or against the server - and respond to messages if needed. Message origins are checked and responses are limited to messages coming from the same domain, ensuring no data is leaked to 3rd parties.

The vault should be the only interface for both the script and the auditorium to read and update data on the server as it ensures all data exchange is protected. The vault also has two other important responsibilities: it handles all client-side cryptography, ensuring event data is encrypted before leaving the browser, as well as managing the data that is persisted in the local IndexedDB. Handling IndexedDB operations from the vault also ensures it is only accessible to offen and protected from 3rd party access.

The auditorium

A pure client side application, the auditorium is responsible for displaying usage data to both users and operators. It is also used to manage data and accounts.

This application is only concerned with display logic and performing requests against the vault, all business logic will be a concern of the vault instead. The auditorium should never read or write data from and to sources other than the embedded vault.

The server

The server is a server side application that has two responsibilities: it serves the static assets generated by the script, the vault and the auditorium (in local development these assets are being proxied by nginx though). Also, it exposes a HTTP JSON API these client side scripts can use to create and query event data and manage accounts.


Understanding the cryptographic entities in use

Event data in Offen is encrypted before leaving the user's browser and will also be stored like this at rest. Decryption only happens on the client side. In order to share data between users and operators the following cryptographic entities will be used:

Account keypair

Each account owns a unique RSA keypair. The public key can be accessed by anyone that knows about the account's AccountId. This way users can encrypt secrets and make them available to certain accounts only.

User secrets

Before a user sends data to an instance of Offen for the first time, the following procedure which roughly resembles a PGP exchange will happen:

  • in the client, a random symmetric UserSecret will be created and persisted locally
  • the account's public key will be used to encrypt the UserSecret
  • the EncryptedUserSecret is sent to the server and associated with a hashed version of the client's user id, thus enabling the operator to generate a decrypted UserSecret when required
  • event data is encrypted using the UserSecret and sent to the server

Decrypting event data

Users

Users can decrypt any event they sent using their local version of the respective UserSecret

Operators

Operators can decrypt all events belonging to their account by decrypting the encrypted UserSecrets and then decrypting the event payloads using these secrets.

Clone this wiki locally
You can’t perform that action at this time.