A script to pull Phabricator task history out of MySQL and denormalize it for reporting
Python PLpgSQL R HTML CSS Shell
Pull request Compare This branch is 778 commits ahead of jaufrec:master.
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
html
.gitignore
README.md
Rplots.pdf
ana_recategorization.csv
ana_scope.py
and_recategorization.csv
and_scope.py
batch_phlog.bash
col_recategorization.csv
col_scope.py
cot_recategorization.csv
cot_scope.py
dis_recategorization.csv
dis_scope.py
discir_recategorization.csv
discir_scope.py
dismap_recategorization.csv
dismap_scope.py
diswik_recategorization.csv
diswik_scope.py
fr_recategorization.csv
fr_scope.py
get_palette.R
ios_recategorization.csv
ios_scope.py
ja_recategorization.csv
ja_scope.py
lan_recategorization.csv
lan_scope.py
loading_functions.sql
loading_tables.sql
make_charts.R
make_report_csvs.sql
make_tranche_chart.R
phl_recategorization.csv
phl_scope.py
phlogiston.py
reconstruction_functions.sql
reconstruction_tables.sql
red_recategorization.csv
red_scope.py
rel_recategorization.csv
rel_scope.py
reporting_functions.sql
reporting_tables.sql
requirements.txt
sc3_recategorization.csv
sc3_scope.py
site-phlogiston
tpg_recategorization.csv
tpg_scope.py
ve_recategorization.csv
ve_scope.py

README.md

Phlogiston

Purpose

Generate burnup, cycle time, and other charts from Phabricator. Intended for experimental prototyping and proof of concept for similar functionality to built into Phabricator. Also a platform to do complex scripted data handling for Phab projects prior to reporting. works from data at at http://dumps.wikimedia.org/other/misc/phabricator_public.dump, generated by https://gerrit.wikimedia.org/r/#/c/214398/2/wmfphablib/phabdb.py

Function

  1. --load
    • reads a JSON file produced by phabricator dump.
    • loads a postgresql database containing the data
  2. --reconstruct
    • Reconstructs the historical state of all tasks in the specified projects project day by day
  3. --report
    • Generates CSV files for all reports
    • Uses R to graph the data as PNG files

TODOs

Typical usage:

wget http://dumps.wikimedia.org/other/misc/phabricator_public.dump
cd phlogiston
python3 phlogiston.py --load --reconstruct --report --scope ve_source.py

Environment:

~phlogiston/                       <- dump goes here
~phlogiston/phlogiston/            <- program goes here
~phlogiston/html                   <- html index and reports go here
~/tmp/                             <- PNG output goes here
Postgresql database named "phab"   <- data goes here

Installation Notes:

  1. Procure an Linux server. Instructions are tested on a WMF labs virtual Ubuntu 14.04 host, and should work verbatim for any standard Ubuntu 14.04 host; they may need modificaton otherwise.
  2. Install prerequisites on the system. As root:
    1. Create a Phlogiston user. adduser phlogiston
    2. Download Phlogiston:
    3. su - phlogiston
    4. git clone https://github.com/wikimedia/phlogiston
    5. exit
    6. (Probably obsolete: As of 2016-02-24, Labs 14.04 image seems to include Postgresql 9.5, which is better. If using stock 14.04, follow instructions to add Postgresql backport to get 9.4. 9.3 and earlier are missing features that Phlogiston uses): http://www.postgresql.org/download/linux/ubuntu/)
    7. Get access to newer R
      • echo deb http://cran.es.r-project.org/bin/linux/ubuntu trusty/ > /etc/apt/sources.list.d/r.list
      • gpg --keyserver keyserver.ubuntu.com --recv-key E084DAB9
      • gpg -a --export E084DAB9 | sudo apt-key add -
      • apt-get update
    8. Install ubuntu packages
      • apt-get install nginx postgresql python3-pip python3-psycopg2 python3-dev postgresql-contrib r-base r-base-dev python3-dateutil
    9. Install R packages.
      • R
      • install.packages(c("ggplot2", "ggthemes", "argparse", "reshape"))
      • update.packages()
      • quit()
      • Note: there seems to be a fatal bug in ggthemes 3.0.1; get 2.2.1 or 3.0.2.
    10. Set up Nginx website
      • cp ~phlogiston/phlogiston/site-phlogiston /etc/nginx/sites-available
      • rm /etc/nginx/sites-enabled/default
      • ln -s /etc/nginx/sites-available/site-phlogiston /etc/nginx/sites-enabled
      • service nginx restart
    11. Change permissions so that the postgres user can share files with phlogiston
      • usermod -a -G phlogiston postgres
      • restart postgres so that this takes effect
    12. Add Python packages
      • pip3 install pyscopg2 pytz jinja2
  3. Set up database. As user postgres,
    • createuser -s phlogiston
    • createdb -O phlogiston phab
  4. Set up the website and cron job. As phlogiston,
    • mkdir ~/html
    • cp ~/phlogiston/html/*.html ~/html
    • cp ~/phlogiston/html/*css ~/html
    • Set up the script to run via cron
      • crontab -e
      • put 15 4 * * * bash ~/phlogiston/batch_phlog.bash -m incremental -s phl -s tpg -s ve -s and -s ios -s col -s cot -s discir -s dismap -s dis -s diswik -s fr -s ja -s red -s rel -s ana >>~/phlog.log 2>&1 at the end of the crontab and save and exit
  5. Do the initial run of Phlogiston. As phlogiston:
    • cd ~/phlogiston
    • ~/phlogiston/phlogiston.py --initialize
    • bash ~/phlogiston/batch_phlog.bash -m incremental -s phl -s tpg -s ve -s and -s ios -s col -s cot -s discir -s dismap -s dis -s diswik -s fr -s ja -s red -s rel -s ana >>~/phlog.log 2>&1