Feel free to skip ahead to Development, if you're more interested in technical implementation.
Points are used by ToS;DR to analyse privacy policies and terms of service agreements across the web, then assign grades to online services based on their respect for users' privacy and personal data. By services, we mean websites and web applications. Think YouTube, or Amazon.
Once a service has been comprehensively analysed in Phoenix, as determined by our curators, the grades appear on tosdr.org.
An A grade indicates that the service treats users with a high level of respect for their privacy and personal data. DuckDuckGo is an example of an A-graded service. Grades D and E indicate that the services treat users with a low level of respect for their privacy and personal data. Examples include CNN and Facebook.
The grades are calculated based on an algorithm, which, in our case, is really just a simple list of conditions that factor in the balance of any given service's points, the relationship of the points to each other, and certain thresholds. This algorithm is subject to change. You can read more about it on our forum. We discuss how it has changed here: Updating the algorithm for service grades – thoughts?
Phoenix is a Ruby on Rails application, with some Vue.js mixed in. It uses a Postgres database.
This is a new implementation, merged for use in production on May 17, 2023. That said, it is still experimental.
Phoenix, when used, also connects to two other services: Atlassian, which runs the widget that helps us to monitor the application's health, and jrbit.de, which is where our databases live as well as our system for reporting errors.
- Docker and docker-compose
- We rely on Docker as an attempt to be OS-agnostic
- If you are not able to use Docker, install the following:
- Ruby 3.0.6
- Rails (Phoenix uses version 184.108.40.206)
- Bundler 2.4.14
- Postgres 11.5
- Node.js (to access Yarn)
- Please refer to the QUICKSTART.md guide for more information on manual set-up. If anything is missing, please let us know.
There are two ways to run Phoenix in development: with Hypothesis, or without Hypothesis. Plenty of hacking around can be done without running Hypothesis, particularly if you just want to get your feet wet or see what the codebase is all about. If you are not interested in running Hypothesis, please skip ahead to Running Phoenix.
Disclaimer: Use of Hypothesis within Phoenix is not supported without Docker
First, clone Phoenix :)
The following steps should be completed from the working directory of
If you have installed Docker compose, getting the application running involves five configuration commands, after which it can be started with a single command in the future.
To prepare the application, run the following commands inside the repository folder to build the application and initialise the database:
$ docker-compose build $ docker network create elasticsearch $ docker network create dbs $ docker-compose up $ docker exec -it edittosdrorg_web_1 rails db:seed
A note on Docker networks: we use them to facilitate development with Hypothesis. Hypothesis and Phoenix share a database, as well as an Elasticsearch instance. The
dbs network is the shared database, and the
elasticsearch network is the shared Elasticsearch instance.
From then on, you can start the application by running:
$ docker-compose up
--build argument if you add or remove dependencies, such as gems)
This will launch the following services: 1) the database (
postgres:11.5-alpine), 2) the Phoenix web application, 3) Elasticsearch (needed to run Hypothesis), 4) adminer (for inspecting the database).
If you wish simply to run the web application and the database, you can launch these services on their own by specifying them in the docker-compose command:
docker-compose up web db
The Phoenix web application runs on port 9090 (http://localhost:9090).
For debugging: To debug application code with
byebug, open a new terminal tab and attach to the running web application container,
docker attach edittosdrorg_web_1.
For running tests: We are in the process of developing our test coverage. Our test environment also runs in Docker and relies on rspec. You can launch tests with the following command:
docker-compose -f docker-compose.yml -f docker-compose.test.yml run --rm web bundle exec rspec
To access the rails console:
docker-compose run web rails c
Create your user via the sign-up page, or use one of the seeded users. There are three seeded users: tosdr_admin, tosdr_curator, tosdr_user. You can log in with any one of these email addresses: email@example.com, firstname.lastname@example.org, email@example.com, with the password
Justforseed1, which will work for all three.
If you created your own user, confirm your user manually
docker-compose run web rails c
- Find your user:
user = User.find_by_username('your_username')
- Confirm user:
To annotate a service, navigate to the services page from the top-right menu, choose a service, and click
View Documents. Begin by highlighting a piece of text from this page. H and the Hypothesis client must be running.
For a demonstration of how annotations work, feel free to inspect the video attached to this PR.
Hypothesis installation - part 1
H is the Hypothesis web service and api.
To use it with Phoenix, clone our fork of H into the same directory as the Phoenix clone, and
cd h/. The correct branch to work from is the phoenix-integration branch.
ATTENTION: The official documentation for installing H is here. Please also consult these docs as needed.
Note the prerequisites:
Before installing your development environment you’ll need to install each of these prerequisites:
- Node and npm. On Linux you should follow nodejs.org’s instructions for installing node because the version of node in the standard Ubuntu package repositories is too old. On macOS you should use Homebrew to install node.
- Docker. Follow the instructions on the Docker website to install “Docker Engine - Community”.
- pyenv. Follow the instructions in the pyenv README to install it. The Homebrew method works best on macOS.
Your Node version in the shell in which you are developing must be more recent than 12.20.0. To manage Node versions, we suggest nvm.
With pyenv, you will need to install python version 3.8.12. From the
pyenv install 3.8.12 pyenv init
Relaunch your shell.
pyenv shell 3.8.12
If pyenv has trouble finding the python binary, you may need to add configuration to
.zshrc, as documented here.
make services, which launches the docker services needed to run H.
If this is your first time,
make devwill install the dependencies. To do so, it requires both node and yarn.
make devwill start the server on port 5000 (http://localhost:5000)
make devcommand will not start H in debug mode, i.e., you will not be able to run
To run H in debug mode, exit the
make devprocess, and run
tox -e dev --run-command 'gunicorn --paste conf/development-app.ini 'h:app:create_app' -b 0.0.0.0:5000'.
You will have to exit and restart
toxwhenever changes are made to the code. Additionally, in debug mode, certain functionalities may be restricted. You will not be able to create and persist annotations from the client if H is running in debug mode, for example.
To launch the shell and poke around in the database, run
Create an admin user from the shell
You will need an admin user to set up OAuth between H and the Hypothesis client.
Follow these instructions.
Log in as admin, and configure OAuth
Ensure that you export the environment variables
h/directory, i.e. the same shell in which you launch the h dev server.
Hypothesis installation - part 2
Clone our fork of the Hypothesis client into the same directory as Phoenix and H. The correct branch to work from is the phoenix-integration branch.
You will need a Node version that is more recent than 12.20.0. H will also have to be running.
Instructions are here, if needed.
This wiki has been deprecated. We are in the process of updating it.
All the details on the database schema can be found on the wiki.
All the details on the API can be found here
- https://edit.tosdr.org (production)
AGPL-3.0+ (GNU Affero General Public License, version 3 or later)