Skip to content

Map building and bin *.js

Jump to bottom
Claudio Agosti edited this page Nov 8, 2018 · 4 revisions

bin/matrix-creation.js

 ~/Dev/invi.sible.link/campaigns/krgotrex DEBUG=* node bin/matrix-creation.js 
[
  {
    "distance": 0,
    "square": [
      0,
      0
    ],
    "left": 13.428,
    "bottom": 52.493,
    "right": 13.43,
    "top": 52.495,
    "day": 1
  },

output: a JSON file usable by the next script in the pipeline. The output produced represent the X,Y coordinates of the area around the starting point (for us, is the Google campus, for you: change the variable rootOfEvil. This produces a map of 7x7 blocks, each of them, would be called in this page as area-block

You have to: save the output in config/fetchmap.json

bin/osmfetch.js

This command fetch OSM API using one of the object inside of config/fetchmap.json, in the example below, --iteration 6 means we are using the object 6. Please note: if you query an area without any website attribute, you'll not get anything. if you have area too big, you'll have a limit of XML nodes from OSM api.

~/Dev/invi.sible.link/campaigns/krgotrex DEBUG=* bin/osmfetch.js --iteration 6
  osmfetch Building destination path, osm and 6 +0ms
  osmfetch checking if destination osm/6 exists +2ms
  osmfetch creating osm/6 +4ms
  osmfetch Reading locationmap config/fetchmap.json +2ms
  lib:various Retrieved 49 objects from config/fetchmap.json +0ms
  osmfetch Getting URL: https://api.openstreetmap.org/api/0.6/map?bbox=13.428,52.489,13.43,52.491 +3ms
  osmfetch Saving 258396 XML from OSM in osm/6/osm-output.xml +1s
  osmfetch Retireved 1016 nodes +63ms
  osmfetch looking in 130 notes with tag +2ms
  osmfetch -> Adding http://www.stadtentwicklung.berlin.de/bauen/brunnen/de/neu/01.shtml | Fritz-Reuter-Brunnen | null | fountain +1ms
  osmfetch -> Adding http://damensalon-berlin.blogspot.com/ | Damen-Salon | Reuterstraße | pub +0ms
  osmfetch -> Adding http://www.kauzundkiebitz.de | Kauz & Kiebitz | Reuterstraße | pub +1ms
  osmfetch -> Adding http://www.goldberg-bar.de/ | Goldberg | Reuterstraße | restaurant +0ms
  osmfetch -> Adding http://www.crop-cosecha.de/ | Crop Cosecha | null | clothes +0ms
  osmfetch -> Adding http://www.standard-saubere-sachen.de/ | standard saubere sachen | Reuterstraße | clothes +0ms
  osmfetch Saving 6 sites on osm/6/selected-shops.json +1ms
  lib:csv register in `sites` of http://www.stadtentwicklung.berlin.de/bauen/brunnen/de/neu/01.shtml +1s
  lib:mongo:read read in sites by {"id":"5bc41e4c3aecb5f3a874e47534b4a84561aaf1db"} sort by {} got 0 results +22ms
  lib:csv register in `sites` of http://www.standard-saubere-sachen.de/ +10ms
  lib:mongo:read read in sites by {"id":"9ecf36804edf15fb63d362cbb54c047d69dd1f45"} sort by {} got 0 results +2ms
  lib:csv register in `sites` of http://www.crop-cosecha.de/ +3ms
  lib:mongo:read read in sites by {"id":"f7044ca9ee9064ee122c5628bfc3926f746ff9cd"} sort by {} got 0 results +2ms
  lib:csv register in `sites` of http://www.goldberg-bar.de/ +2ms
  lib:mongo:read read in sites by {"id":"ee13036cdd420a859a1f1f9929ed5ee7fc5cb6be"} sort by {} got 1 results +2ms
  lib:csv register in `sites` of http://www.kauzundkiebitz.de +0ms
  lib:mongo:read read in sites by {"id":"61289bb788afa071f3ffb9e39bf79195aabde10a"} sort by {} got 0 results +2ms
  lib:csv register in `sites` of http://damensalon-berlin.blogspot.com/ +2ms
  lib:mongo:read read in sites by {"id":"a715bd165491fee170be607f6c74c7843deeed3f"} sort by {} got 1 results +1ms
  osmfetch Saved in database 4 new sites, checked with frequency of 5 +51ms

Output:

~ /invi.sible.link/campaigns/krgotrex ls -l osm/6/
total 260
-rw-rw-r-- 1 oo oo 258494 Out 30 10:16 osm-output.xml
-rw-rw-r-- 1 oo oo   1159 Out 30 10:16 selected-shops.json

And the .json file matter for us:

~/Dev/invi.sible.link/campaigns/krgotrex more osm/6/selected-shops.json 
[
  {
    "site": "http://www.stadtentwicklung.berlin.de/bauen/brunnen/de/neu/01.shtml",
    "address": null,
    "kind": "fountain",
    "name": "Fritz-Reuter-Brunnen",
    "lat": "52.4892715",
    "lon": "13.4298862"
  },
  {
    "site": "http://damensalon-berlin.blogspot.com/",
    "address": "Reuterstraße",
    "kind": "pub",
    "name": "Damen-Salon",
    "lat": "52.4909171",
    "lon": "13.4292557"
  },

The site is the URL which would be analyzed and the lat lon represent the marked in the map.

Side effect

The last command writes in mongodb of the invi.sible.link framework, and schedule if the proper software are running.

Opportuninstic push

This is the command which I'm keeping in crontab, so every time a new website get registered among the 'sites', it is legit analyze it

$ DEBUG=* bin/queueOPush.js --campaign krgotrex --kind basic    
  bin:queueOPush Welcome to the queue opportunistic pusher +0ms
  bin:queueOPush - campaign is set as krgotrex +4ms
  bin:queueOPush First step: looking in `sites` table +1ms
  bin:queueOPush SORRY! in this moment option [basic] is hardcoded, fix the code below +0ms
  lib:mongo:read read in sites by {"campaign":"krgotrex"} sort by {} got 74 results +77ms
  bin:queueOPush 74 sites appears to need be analyzed +16ms
  bin:queueOPush Statistics: {
  "krgotrex": 74
} +1ms
  lib:mongo:read read in promises2 by {"id":"a6f1d54ac9e1bdc663a30c3786104afc367fffbb"} sort by {} got 0 results +43ms
  [...]
  lib:mongo:read read in promises2 by {"id":"44c8f2de5abd4ab2e7c920fc2eec4e2518f0fdff"} sort by {} got 0 results +6ms
  lib:mongo writeMany done: in promises2 74 objects +24ms
  lib:queue added [krgotrex] 74 promises  +0ms
  lib:queue 74 object, a random obj is {
  "kind": "basic",
  "start": "2018-11-08T00:00:00.000Z",
  "id": "5a277bb8b2b4771e5663713c3318967ed3e858f6",
  "href": "http://www.kauzundkiebitz.de",
  "campaign": "krgotrex",
  "when": "2018-11-08T13:53:00.000Z",
  "description": "Reuterstraße pub Kauz & Kiebitz",
  "rank": 25,
  "siteId": "61289bb788afa071f3ffb9e39bf79195aabde10a",
  "subjectId": "d706f5425e1064dcb2ce3dbea3905bb3733dce76",
  "testId": "4069979c6c871c97ada3865a420a348223a05b1b",
  "_id": "5be43fd90eb24515d93031ff"
} +1ms

and after the analysis is completed

DEBUG=* bin/computeResults.js --campaign krgotrex

And this build:

A system which query periodically OpenStreetMap, and fetch website from an area. the center of this area is the Berlin Google Campus. Then, analyzes the trackers of said websites.

The website kreuzberg.google.tracking.exposed is the representation of this git repository.

From OpenStreetMap the websites are retrieved, and then analyzed with the invi.sible.link framework, it computes an analysis of third-party trackers. We use this results to do a campaign explaining third-party tracking.

Clone this wiki locally