map-trends

Show trends on maps as part of a term paper.

• Formulate project proposal
• Define scope of project

Install

Requirements

• OSX or Linux
• Python 2.7, Python 3.4 or PyPy
• GNU parallel

Install Python requirements.

pip install -r requirements.txt

Prepare Data

You have several possiblities to prepare the data:

• Follow the instructions
• Use ./prepare.sh
• Download prepared tiles directly

Download, Extract and Transform

Download all logs from the server.

wget -nH -A xz -m http://planet.openstreetmap.org/tile_logs

Extract logs and remove compressed versions.

unxz *.xz
rm *.xz

Expand tile coordinates in CSV to separate columns (from 1/2/3 123 to 1 2 3 123).

sed -i 's/\// /g' *.txt

Rename .txt files to .csv.

rename 's/\.txt$/\.csv/' *.txt

Calculate Coordinates

Convert tile indizes from the logs with slippy tile names to Spherical Mercator coordinates add CSV header.

mkdir -p tile_coords

for filename in $(ls tile_logs/*.csv); do
  echo $(basename $filename)
  echo "z x y requests latitude longitude" > tile_coords/$(basename "$filename")
  cat $filename | ./calc_coords.py >> tiles_coords/$(basename "$filename")
done

Convert from space delimited to comma delimited CSV (requirements for use in Mapbox).

mkdir -p tile_mapbox

for filename in $(ls tile_coords/*.csv); do
  cat $filename | tr ' ' ',' > tile_mapbox/$(basename "$filename").mapbox.csv
done

Analyze

Sum Tile Requests

aggregate_requests.py takes in a list of absolute filepaths and sums up the requests of the tiles. The csv files must already be prepared with the prepare.sh script.

ls -d -1 tile_logs/tiles-2015*.csv | ./aggregate_requests.py >> tiles-2015.csv

Filter tiles by Level

cat tiles-2015-01-01.csv | awk '$1 == "18"' > cat tiles_18-2015-01-01.csv

Extract timeframe for tiles at level 18

echo "date requests latitude longitude" > tiles_over_time_switzerland.csv
for filename in $(ls tile_coords/*.csv); do
    base_filename=$(basename "$filename")
    date=$(echo "$base_filename" | sed 's/tiles-//' | sed 's/\.csv//')
    echo $date
    cat $filename | \
        awk '$1 == "18"' | \
        cut -d ' ' -f4-6 | \
        python filter_switzerland.py | \
        sed -e "s/^/$date /" >> tiles_over_time_switzerland.csv
done