Skip to content
Permalink
Browse files

Mozilla global sprint (#31)

* Setting up docker compose environment

Based on template: https://github.com/dternyak/Flask-Postgres-Docker

* Basic db setup

* cleaning up frame script

* Basic ansible setup

* Add messages table and collector

* - Added endpoint to serve the script
- Added endpoint to create ids

* Fixing docker not rebuilding on requirements.txt changes

* Fixed: added missing files

* adding site if

* Fixing logging issue in gunicorn, adding site id in template

* Making first version of dashboard, template files etc

* Use string site id in message table

* Fix database migrations

* Changing db name and migration scripts

* fixing migation deployment

* Probe script build system

* keep probe tests folder

* Fetch dashboard stats from db

* Live reloading of probe script

* Simpler deploys

* Get list of domains for a site

* Basic dashboard

* Adding theme to landing page (#28)

* Trying a new bootstrap template

* addind theme

* addind theme

* Mozilla global sprint (#29)

* Trying a new bootstrap template

* addind theme

* addind theme

* adding tokens on landing page

* fixing render template

* adding tokens on landing page

* moving iframe out of script, fixing typos (#30)

* Trying a new bootstrap template

* addind theme

* addind theme

* adding tokens on landing page

* fixing render template

* adding tokens on landing page

* renaming DB file, adding script to landing page

* moving iframe out of script, fixing typos

* moving iframe out of script, fixing typos

* moving iframe out of script, fixing typos

* using host instead of origin. (#32)

* Trying a new bootstrap template

* addind theme

* addind theme

* adding tokens on landing page

* fixing render template

* adding tokens on landing page

* renaming DB file, adding script to landing page

* moving iframe out of script, fixing typos

* moving iframe out of script, fixing typos

* moving iframe out of script, fixing typos

* change origin to hostname

* fix (#33)

* Fixing dashboard (#39)

* The data time was not getting updated while querying, hence serving stale dashboard.
* Added page-views to the dashboard.
* Changed to site-id for homepage.

Already deployed, hence merging.
  • Loading branch information...
chrmod authored and konark-cliqz committed Jun 6, 2017
1 parent 561c15f commit 4dc44f31643aff50de05acc35727500ce0dcfcb1
Showing with 38,715 additions and 648 deletions.
  1. +13 −0 .env
  2. +1 −0 .gitignore
  3. +40 −24 README.md
  4. +25 −0 ansible.yml
  5. +0 −1 conf/nginx.conf
  6. +44 −0 docker-compose.yml
  7. +1 −0 init.sql
  8. +3 −0 nginx/Dockerfile
  9. +21 −0 nginx/sites-enabled/flask_project
  10. +2 −0 probe/.gitignore
  11. +7 −0 probe/Dockerfile
  12. +13 −0 probe/package.json
  13. +685 −0 probe/src/iframe.js
  14. +6 −0 probe/src/index.js
  15. 0 probe/tests/.gitkeep
  16. +3,322 −0 probe/yarn.lock
  17. +0 −158 src/server.py
  18. +0 −291 src/static/dashboard.css
  19. +0 −168 src/templates/dashboard.html
  20. +11 −0 web/Dockerfile
  21. +45 −0 web/app.py
  22. +14 −0 web/config.py
  23. 0 web/controllers/__init__.py
  24. +20 −0 web/controllers/analyze.py
  25. +11 −0 web/controllers/green_analytics_script.py
  26. +30 −0 web/controllers/landing.py
  27. +13 −0 web/controllers/messages.py
  28. +123 −0 web/controllers/metric_detail.py
  29. 0 {src → web/controllers}/report.py
  30. +21 −0 web/controllers/sites.py
  31. +8 −0 web/index.py
  32. 0 {src → web}/logs/collect.jl
  33. 0 {src → web}/logs/collect.jl.bkp1
  34. +11 −0 web/manage.py
  35. +1 −0 web/migrations/README
  36. +45 −0 web/migrations/alembic.ini
  37. +73 −0 web/migrations/env.py
  38. +22 −0 web/migrations/script.py.mako
  39. +51 −0 web/migrations/versions/2a922c2d4cd_.py
  40. 0 web/models/__init__.py
  41. +24 −0 web/models/message.py
  42. +13 −0 web/models/site.py
  43. +14 −0 web/requirements.txt
  44. +5,677 −0 web/static/dashboard.css
  45. +9 −6 {src → web}/static/dashboard.js
  46. 0 {src → web}/static/extern/Chart.min.js
  47. 0 {src → web}/static/extern/bootstrap.min.css
  48. 0 {src → web}/static/extern/bootstrap.min.js
  49. 0 {src → web}/static/extern/bootstrap_with_easter_egg.min.js
  50. +319 −0 web/static/extern/css/grayscale.css
  51. +5 −0 web/static/extern/css/grayscale.min.css
  52. 0 {src → web}/static/extern/hmac-sha1.js
  53. BIN web/static/extern/img/downloads-bg.jpg
  54. BIN web/static/extern/img/intro-bg.jpg
  55. BIN web/static/extern/img/map-marker.png
  56. 0 {src → web}/static/extern/jquery.flot.min.js
  57. 0 {src → web}/static/extern/jquery.min.js
  58. +38 −0 web/static/extern/js/grayscale.js
  59. +6 −0 web/static/extern/js/grayscale.min.js
  60. +8 −0 web/static/extern/ua-parser.min.js
  61. +6,757 −0 web/static/extern/vendor/bootstrap/css/bootstrap.css
  62. +6 −0 web/static/extern/vendor/bootstrap/css/bootstrap.min.css
  63. BIN web/static/extern/vendor/bootstrap/fonts/glyphicons-halflings-regular.eot
  64. +288 −0 web/static/extern/vendor/bootstrap/fonts/glyphicons-halflings-regular.svg
  65. BIN web/static/extern/vendor/bootstrap/fonts/glyphicons-halflings-regular.ttf
  66. BIN web/static/extern/vendor/bootstrap/fonts/glyphicons-halflings-regular.woff
  67. BIN web/static/extern/vendor/bootstrap/fonts/glyphicons-halflings-regular.woff2
  68. +2,377 −0 web/static/extern/vendor/bootstrap/js/bootstrap.js
  69. +7 −0 web/static/extern/vendor/bootstrap/js/bootstrap.min.js
  70. +2,199 −0 web/static/extern/vendor/font-awesome/css/font-awesome.css
  71. +4 −0 web/static/extern/vendor/font-awesome/css/font-awesome.min.css
  72. BIN web/static/extern/vendor/font-awesome/fonts/FontAwesome.otf
  73. BIN web/static/extern/vendor/font-awesome/fonts/fontawesome-webfont.eot
  74. +685 −0 web/static/extern/vendor/font-awesome/fonts/fontawesome-webfont.svg
  75. BIN web/static/extern/vendor/font-awesome/fonts/fontawesome-webfont.ttf
  76. BIN web/static/extern/vendor/font-awesome/fonts/fontawesome-webfont.woff
  77. BIN web/static/extern/vendor/font-awesome/fonts/fontawesome-webfont.woff2
  78. +34 −0 web/static/extern/vendor/font-awesome/less/animated.less
  79. +25 −0 web/static/extern/vendor/font-awesome/less/bordered-pulled.less
  80. +12 −0 web/static/extern/vendor/font-awesome/less/core.less
  81. +6 −0 web/static/extern/vendor/font-awesome/less/fixed-width.less
  82. +18 −0 web/static/extern/vendor/font-awesome/less/font-awesome.less
  83. +733 −0 web/static/extern/vendor/font-awesome/less/icons.less
  84. +13 −0 web/static/extern/vendor/font-awesome/less/larger.less
  85. +19 −0 web/static/extern/vendor/font-awesome/less/list.less
  86. +60 −0 web/static/extern/vendor/font-awesome/less/mixins.less
  87. +15 −0 web/static/extern/vendor/font-awesome/less/path.less
  88. +20 −0 web/static/extern/vendor/font-awesome/less/rotated-flipped.less
  89. +5 −0 web/static/extern/vendor/font-awesome/less/screen-reader.less
  90. +20 −0 web/static/extern/vendor/font-awesome/less/stacked.less
  91. +744 −0 web/static/extern/vendor/font-awesome/less/variables.less
  92. +34 −0 web/static/extern/vendor/font-awesome/scss/_animated.scss
  93. +25 −0 web/static/extern/vendor/font-awesome/scss/_bordered-pulled.scss
  94. +12 −0 web/static/extern/vendor/font-awesome/scss/_core.scss
  95. +6 −0 web/static/extern/vendor/font-awesome/scss/_fixed-width.scss
  96. +733 −0 web/static/extern/vendor/font-awesome/scss/_icons.scss
  97. +13 −0 web/static/extern/vendor/font-awesome/scss/_larger.scss
  98. +19 −0 web/static/extern/vendor/font-awesome/scss/_list.scss
  99. +60 −0 web/static/extern/vendor/font-awesome/scss/_mixins.scss
  100. +15 −0 web/static/extern/vendor/font-awesome/scss/_path.scss
  101. +20 −0 web/static/extern/vendor/font-awesome/scss/_rotated-flipped.scss
  102. +5 −0 web/static/extern/vendor/font-awesome/scss/_screen-reader.scss
  103. +20 −0 web/static/extern/vendor/font-awesome/scss/_stacked.scss
  104. +744 −0 web/static/extern/vendor/font-awesome/scss/_variables.scss
  105. +18 −0 web/static/extern/vendor/font-awesome/scss/font-awesome.scss
  106. +11,008 −0 web/static/extern/vendor/jquery/jquery.js
  107. +5 −0 web/static/extern/vendor/jquery/jquery.min.js
  108. +160 −0 web/static/family-lora.css
  109. +48 −0 web/static/family-montserrat.css
  110. BIN {src → web}/static/favicon.png
  111. BIN web/static/fonts/FiraMono-Regular.0b6138c5b386.ttf
  112. BIN web/static/fonts/FiraMono-Regular.690950e8d89c.woff
  113. BIN web/static/fonts/IQHow_FEYlDC4Gzy_m8fcmaVI6zN22yiurzcBKxPjFE.woff2
  114. BIN web/static/fonts/fira-mono/FiraMono-Regular.0b6138c5b386.ttf
  115. BIN web/static/fonts/fira-mono/FiraMono-Regular.690950e8d89c.woff
  116. +563 −0 web/static/lanyon.css
  117. 0 {src → web}/static/rogue.html
  118. 0 {src → web}/static/style-log.css
  119. 0 {src → web}/static/style.css
  120. +80 −0 web/templates/dashboard.html
  121. 0 {src → web}/templates/template_frame.html
  122. +4 −0 web/templates/template_iframe.html
  123. 0 {src → web}/templates/template_introspect_localstorage.html
  124. 0 {src → web}/templates/template_introspect_messages.html
  125. +210 −0 web/templates/template_landing.html
  126. +20 −0 web/templates/template_layout.html
  127. 0 {src → web}/templates/template_log.html
  128. +51 −0 web/templates/template_signup.html
  129. 0 {src → web}/templates/template_sites.html
13 .env
@@ -0,0 +1,13 @@
# Add Environment Variables

DEBUG=True
SECRET_KEY=5(15ds+i2+%ik6z&!yer+ga9m=e%jcqiz_5wszg)r-z!2--b2d
DB_NAME=green_analytics
DB_USER=postgres
DB_PASS=postgres
DB_SERVICE=postgres
DB_PORT=5432
PROTOCOL=http
DOMAIN_NAME=green-analytics.test.cliqz.com
COLLECTOR_PATH=/collect
DASHBOARD_PATH=/dashboard
@@ -1,3 +1,4 @@
*.pyc
deploy/extern
ansible.retry

@@ -1,43 +1,59 @@
# Green anaytics
As per the study Tracking the Trackers[1], ~78% of the page loads in Germany have at least one third-party tracker, almost ~42% of Page loads of Google analytics and ~22% have a Facebook tracker.

Green analytics is meant to provide solutions :
* A. Privacy first analytics.
* B. Privacy first social sharing features.

## A. Privacy first analytics :

The aim is to build an analytics service, which the users can use on their websites, blogs etc.
The existing solutions like Google Analytics, Piwik etc all rely on server-side aggregation which means they cookies, IP or other parameters to link the message on the backend.
Privacy preserving analytics service is meant to cover a wide range of use-cases: Unique visits and page loads. Returning customers.
* Goal conversion to track campaigns.
* Cross site correlations.
* In-site click-throughs.
Green analytics is meant to provide solutions :
* A. Privacy first analytics.
* B. Privacy first social sharing features.

## A. Privacy first analytics :

The aim is to build an analytics service, which the users can use on their websites, blogs etc.
The existing solutions like Google Analytics, Piwik etc all rely on server-side aggregation which means they cookies, IP or other parameters to link the message on the backend.
Privacy preserving analytics service is meant to cover a wide range of use-cases: Unique visits and page loads. Returning customers.
* Goal conversion to track campaigns.
* Cross site correlations.
* In-site click-throughs.
* Visits and time in page per user (without beacons).

Without having to rely on the cookies or any such thing. This approach unlike the industry de-facto standard, relies on client-side aggregation. Which means the user is the owner of the data, any only emits an aggregated event, without any identifier.
Without having to rely on the cookies or any such thing. This approach unlike the industry de-facto standard, relies on client-side aggregation. Which means the user is the owner of the data, any only emits an aggregated event, without any identifier.

Based on the implementation of Green-tracker by Josep from Cliqz[2] , Demo[3], objective at the Mozilla global sprint is to :
* 1 Make people aware of doing analytics in a privacy preserving manner.
* 2 Provide a generic script, which should be ready-to use for people who want to do analytics on their projects.
Based on the implementation of Green-tracker by Josep from Cliqz[2] , Demo[3], objective at the Mozilla global sprint is to :
* 1 Make people aware of doing analytics in a privacy preserving manner.
* 2 Provide a generic script, which should be ready-to use for people who want to do analytics on their projects.

Apart from just providing an analytics scripts, the task would be to provide :
* 1 Green analytics for websites & open-source market leading CMS like Wordpress / Drupal.
* 2 Basic backend which collects data and generates analytics report and dashboards. If the time permits, we would also like to build a small SDK for Mobile app analytics based on the same approach.
Apart from just providing an analytics scripts, the task would be to provide :
* 1 Green analytics for websites & open-source market leading CMS like Wordpress / Drupal.
* 2 Basic backend which collects data and generates analytics report and dashboards. If the time permits, we would also like to build a small SDK for Mobile app analytics based on the same approach.

## B. Privacy first social sharing features :
## B. Privacy first social sharing features :

The other very common way of tracking users across websites is adding social sharing buttons, another component of Green analytics project will be identify plugins which still allow the users to share over social media but needs explicit consent. There are already open-source solutions available for static websites / CMSes , we should not try and re-invent the wheel, instead we should create a comprehensive list of the available solutions and create a in-depth how-to guide.

## Footnotes:
* 1 Tracking the Trackers: http://www2016.net/proceedings/proceedings/p121.pdf
* 2 Data Collection without Privacy Side-Effects : http://josepmpujol.net/public/papers/big_green_tracker.pdf
## Footnotes:
* 1 Tracking the Trackers: http://www2016.net/proceedings/proceedings/p121.pdf
* 2 Data Collection without Privacy Side-Effects : http://josepmpujol.net/public/papers/big_green_tracker.pdf
* 3 Demo Green-tracker : http://site1.test.cliqz.com/


## Getting started

**TODO**
### Run locally

Everything is dockerized, just run:

```bash
docker-compose up --build
```

### Deploy to own server

Deploy green analytics with ansible >= 2.3

```bash
ansible-playbook ansible.yml -i <hostname>`,
```
(notice trailing comma `,`, it is important!)
## Contributing
@@ -0,0 +1,25 @@
- hosts: all
become: true
gather_facts: no
tasks:
- name: Install docker
shell: which docker || curl -sSL https://get.docker.com/ | sh

- name: Install pip
apt:
name: python-pip

- name: Install docker-compose
pip:
name: docker-compose

- name: Deploy new version
synchronize:
src: .
dest: .
delete: yes
recursive: yes

- name: Docker compose
docker_service:
project_src: green-analytics

This file was deleted.

@@ -0,0 +1,44 @@
web:
restart: always
build: ./web
expose:
- "8000"
links:
- postgres:postgres
volumes:
- ./web:/usr/src/app
env_file: .env

nginx:
restart: always
build: ./nginx/
ports:
- "80:80"
volumes_from:
- web
- probe
links:
- web:web

probe:
build: ./probe/
volumes:
- ./probe:/usr/src/probe
- ./probe:/workspace

data:
restart: always
image: postgres:latest
volumes:
- /var/lib/postgresql
command: "true"

postgres:
restart: always
image: postgres:latest
volumes_from:
- data
volumes:
- ./init.sql:/docker-entrypoint-initdb.d/init.sql
ports:
- "5432:5432"
@@ -0,0 +1 @@
CREATE DATABASE green_analytics;
@@ -0,0 +1,3 @@
FROM tutum/nginx
RUN rm /etc/nginx/sites-enabled/default
ADD sites-enabled/ /etc/nginx/sites-enabled
@@ -0,0 +1,21 @@
server {

listen 80;
# server_name example.org;
charset utf-8;

location /static {
alias /usr/src/app/static;
}

location /probe {
alias /usr/src/probe/dist;
}

location / {
proxy_pass http://web:8000;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}
@@ -0,0 +1,2 @@
node_modules
dist
@@ -0,0 +1,7 @@
FROM kkarczmarczyk/node-yarn

RUN yarn global add raureif

WORKDIR /workspace

CMD yarn && raureif serve
@@ -0,0 +1,13 @@
{
"name": "green-analytics-probe",
"version": "1.0.0",
"main": "dist/index.js",
"license": "AGPLv3",
"devDependencies": {
"raureif": "1.3.0"
},
"dependencies": {
"crypto-js": "^3.1.9-1",
"ua-parser-js": "^0.7.12"
}
}

0 comments on commit 4dc44f3

Please sign in to comment.
You can’t perform that action at this time.