Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Milestone 1 #192

Merged
merged 164 commits into from Nov 28, 2019
Merged

Milestone 1 #192

merged 164 commits into from Nov 28, 2019

Conversation

@m90
Copy link
Member

m90 commented Nov 28, 2019

Milestone 1 has been titled "laying the foundation". This means we focused on doing essential work on the underlying application setup, build an architecture that will help us getting the features defined in the upcoming milestones added in a robust and efficient manner.

Achievements included in this release

Things we achieved in the last 6 weeks are (this list obviously is non-exhaustive):

Application architecture

We further fleshed out the existing application architecture to be more extensible, robust and accessible for outside contributors. A pattern we discovered as a good fit for all parts of the application is extension by middleware. This means additional behavior can transparently be added by any consumer by adding or removing middleware from the stack. PR #130 is a good example for what this looks like.

In addition to that, we refactored our persistence layer towards two goals:

  • The application in its default build now supports SQLite, MySQL and Postgres databases. This can simply be configured by specifying the dialect and the database location. See PR #170 for the implementation.
  • Database read/write access has been factored out into a separate package that is not concerned with any kind of business logic. This means third parties could easily extend the application to support any other (relational as well as non-relational) datastore. This has been implemented in PR #179.

Build

We want offen to be easy to deploy, and also make it easy for people interested to give it a test drive. This is why we revamped our build process so that offen now is compiled into a single binary file that contains both the server-side as well as the client-side application and all of the required assets. This means you can now download a offen binary and execute it on your local machine to instantly have a (ephemeral) instance up and running. See PR #154 for the implementation. Builds are also being run in Continuous Integration, making the artifacts available to the public (the build after merging this PR is available for download here).

Right now, this setup only supports Linux, but we are aiming to support Windows and MacOS just as well soon.

Dev Setup

We further improved the development setup to be accessible to contributors without any complex setup required. We opted for using a Docker and docker-compose based solution that allows us to have these two tools as the only hard requirements for developing offen on any operating system. We successfully tested this on Linux, MacOS and Windows. Our wiki has a section on how to get the setup up and running.

Frontend setup

We did further research and experiments and decided upon an approach for building the frontend for offen that allows us to build lightweight, accessible and resilient interfaces. We chose choo as the application framework and tachyons as the CSS framework.

In PR #180 a working prototype for the "Auditorium" (the part of offen that is used to access and manage user data) has been implemented. We are running automated Accessibility Audits as part of our build.

Localization

We implemented a basic localization setup for the application. We chose a gettext-style approach because of its ubiquity and the fact that it still provides context in code. All user-facing content in both server-side and client-side applications is now localizable and commands for extracting changed strings from all parts of the application are available. See PR #177 for the implementation.

Right now, English is the only locale supported, but we'd be ready to add more locales at any time. We'll proactively look for contributors of such locales once we have fleshed out the exact content structure after Milestone 3.

Documentation

We started collecting and structuring documentation in our wiki, both for developers as well as people that want to deploy and run offen.

Production grade server

To further iterate on the distribution via single binary topic, we invested time into further hardening and improving the HTTP server that offen exposes. It's now possible to expose this server directly to the internet without the need for a reverse proxy running in front of it. Security related headers like Content-Security-Policy are enabled by default, just like we default to privacy friendly settings like Referer-Policy (PR #166). Caching is leveraged as fitting for static assets (PR #191). In case a deployment scenario prefers a reverse proxy, these features can be disabled through configuration.

As offen expects to be served via HTTPS to ensure maximum protection of the data being transferred, an AutoTLS feature has been added that lets the application automatically issue LetsEncrypt certifcates (PR #188) in case the operator isn't able to supply a certificate (PR #187) themselves.

Application configuration

All configuration of the application has been consolidated into environment variables. These variables are now also properly namespaced (i.e. OFFEN_OPTION_XYZ) in order to avoid clashes with other applications running on the same host.

.env-style configuration files are also supported, following the default Linux cascade (#175) of /etc/offen, user home and working directory. Bootstrapping an application database will also generate and persist application secrets using this approach.


Coming up next

Milestone 2 will be about "collecting data securely". In the course of the coming weeks this means we will work on topics like these:

User Opt-In

A major piece of getting the software ready for experimental usage (both by ourselves as well as by others), we will implement an Opt-In-First approach to data collection. No data is ever collected in case the user has not explicitly allowed the offen instance to do so. In case the user decides not to contribute data, the decision is persisted and the user can use the host site without any restrictions. In case the user opts in, they can revoke this at any time, just like they can permanently delete all of their data at any time. #155 on this topic is already in progress. This topic might not be too interesting from a technical perspective, yet getting the user experience right here is probably a crucial part that can make or break offen.

Userland cryptography

All usage data in offen is encrypted on the client before being transferred over the network or persisted in any database. This makes sure possible database leaks do not expose any user data and it's also pointless for 3rd party scripts to intercept and inspect network traffic on the host page.

Right now offen is using the cryptography primitives built into modern day browsers known as window.crypto.subtle. To use these, a host page using a https connection is required. While this is the recommended way of deploying your website, we also want to enable people that are not able to deploy their website using https to use offen. This is why we will work on a fallback cryptography module that is used in these cases. We plan to base this on forge which is well hardened and actively maintained.

Collecting and deriving meaningful statistics

A huge part of this milestone will be about defining which usage data can be collected in a privacy-friendly manner and how we can process it so that operators as well as users receive statistics that are useful to them. Do we need to add additional metrics to supply these metrics, or can we even remove some of what the current version is collecting? In a conflict of usefulness vs privacy, we will always opt for privacy.


We hope to finish milestone 2 by the end of January and provide another project update about what happened in the meantime then.


Bonus: Interested in getting your hands dirty?

If you want to give the current state of offen after this PR has been merged a test drive (this section is referring to commit 1787d43 from the 28th of November), this section is for you. All of the following assumes you are on a Unix-like system where you can use make and have Docker and docker-compose installed.

Setting up the development environment.

Clone the github.com/offen/offen repository and check out the master branch (or this commit above in case you are reading this at a later point in time).

Now you need to do two things:

  • Build the development images and install the dependencies (this might take a while if you are running this for the first time, it will also print some npm warnings, which is ok):
$ make setup
  • Once your images are built, you need to seed the development database with account and user data:
$ make bootstrap

In case both of these commands exited zero, you are now ready to use your development environment:

$ make up

This will start the offen server listening on port 8080, meaning you can access the offen instance on http://localhost:8080

The index page is collecting usage data already, so next you can head to http://localhost:8080/auditorium to access and manage this data.

To access the data available to site operators, head to http://localhost:8080/auditorium/login and login using develop@offen.dev with password develop

If you want to hack on any portion of the application, your changes will automatically be recompiled, so all you need to see them is hitting refresh in your browser.

You can shut down the server again using Ctrl+C. Data is persisted in Docker volumes, so one can stop and start the dev environment at any time.

The wiki has further info on the development setup.

Building the binary and running it

IMPORTANT: The following currently only works on Linux. The built application is also still missing features that make it ready for production usage. Do not deploy this binary into any user facing scenario.

Assuming you have a working development setup, you are ready to build the application:

$ make build

Once this command has finished, your repository will contain a binary called offen

You can check if it is working on your system by printing its version:

$ ./offen version

This will print your current git revision.

Next, you can bootstrap the application using a local ephemeral database:

$ ./offen bootstrap -email you@example.net -name test -password yourpassword

This creates an account, keys and runtime secrets and persists them in the database and a config file in ~/.config/offen.env

Now you are able to have the application being served:

$ ./offen

This will bring up your instance on port 8080 (you can change the port by setting OFFEN_SERVER_PORT)

Extending the application

In case you are interested in gaining understanding of how the entire application works, let's implement a rudimentary version of a feature that is still missing: user opt-in. Before offen collects data, we want the user to express consent by confirming a prompt.

The topmost piece of code that runs when collecting data is placed in the script application where the following is run on pageview (see index.js):

app.on('PAGEVIEW', supportMiddleware, function (context, send, next) {
  var message = {
    type: 'EVENT',
    payload: {
      accountId: accountId,
      event: events.pageview()
    }
  }
  send(message)
})

app works like a router that passes the event through the following middleware functions. A middleware has the signature of function (context, send, next). context is the event context, send actually sends the given message to the server and next calls the next middleware in the stack.

This means that if we want to write a simple consent middleware this would look like:

function consentMiddleware (context, send, next) {
  if (window.confirm('Are you ok with having basic usage data about your visit being collected?')) {
    next()
  }
  // in case the above branch did not run, having this function end without
  // calling next will mean no following handler will be invoked, thus no
  // event being sent to the server
}

Now, the middleware can be mounted like this:

app.on('PAGEVIEW', supportMiddleware, consentMiddleware, function (context, send, next) {
  var message = {
    type: 'EVENT',
    payload: {
      accountId: accountId,
      event: events.pageview()
    }
  }
  send(message)
})

This is a very crude approach from a UX perspective, but it's probably a good example on how the applications offen is composed of are being architected.


Feedback? Found a bug?

If you have any feedback, comment or bug report on this milestone release, we'd love to hear from you. Open an issue, leave a comment on this PR or send us an email at hioffen@posteo.de.

m90 and others added 30 commits Oct 10, 2019
Update docs, add global test command to makefile
Make banner text more generic to fit any deployment
clean up nginx conf, enforce trailing slashes in spa urls
Relicense as Apache 2.0
Extract binary from docker build and store as build artifact
Add subcommand for printing binary revision
Remove babel, add migration command for dev
Set http headers for static assets on app level
* mention nlnet

* remove junk folder
Use node 12 to build and develop client side apps
Update vulnerable dependency
Enable calling through to other gorm dialects, use sqlite in dev
hendr-ik and others added 28 commits Nov 24, 2019
Add created at date to accounts
Enable users to pass local SSL certificate
Add autotls option
Remove dead code from keys package
Auditorium update, tachyons css, wordings and layout
Derive secure cooking setting from development env and request URL
Improve HTTP caching behavior
@m90 m90 merged commit 1787d43 into master Nov 28, 2019
6 checks passed
6 checks passed
ci/circleci: auditorium Your tests passed on CircleCI!
Details
ci/circleci: build Your tests passed on CircleCI!
Details
ci/circleci: packages Your tests passed on CircleCI!
Details
ci/circleci: script Your tests passed on CircleCI!
Details
ci/circleci: server Your tests passed on CircleCI!
Details
ci/circleci: vault Your tests passed on CircleCI!
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
2 participants
You can’t perform that action at this time.