Skip to content
This repository has been archived by the owner on Oct 3, 2018. It is now read-only.

Handle process names in the json creation script #78

Closed
EricRahm opened this issue Feb 13, 2015 · 4 comments
Closed

Handle process names in the json creation script #78

EricRahm opened this issue Feb 13, 2015 · 4 comments
Assignees

Comments

@EricRahm
Copy link
Contributor

Once process names are added to the DB (#77) we should update our scripts to actually use them.

@Nephyrin
Copy link
Contributor

create_graph_json.py handles existing metadata by adding it to the datapoint, e.g.

Meta Datapoint
Iteration 1:CheckpointFoo blah/explicit/js

becomes Iteration 1/CheckpointFoo/blah/explicit/js

The reason these fields aren't just baked into the datapoint by the test to begin with is the silly string de-duplication stuff. We don't want to store each reporter name iterations * checkpoints times per run in sqlite. A database with proper transparent compression would solve this, so we could just prefix whatever onto the reporter name and it would work.

Also, the website is hardcoded to mark certain 'depths' of the tree as special in a few spots, but these are easy to increment if we add another level for process:
https://github.com/mozilla/areweslimyet/blob/master/html/slimyet.js#L820
https://github.com/mozilla/areweslimyet/blob/master/html/slimyet.js#L821
https://github.com/mozilla/areweslimyet/blob/master/html/slimyet.js#L928

@Nephyrin
Copy link
Contributor

For the record, if you change the JSON format, the procedure to re-export the old databases is roughly:

# Run this over a weekend, it wont be quick, these DBs are huge and sqlite slow
for compressedMonthDB in db/areweslimyet-*.sqlite.xz; do
  monthDB="${compressedMonthDB%.xz}"
  seriesName="$(basename "${monthDB%.sqlite}")"
  monthJSON="html/data/$seriesName.json.gz"
  util/unarchive_db.sh "$compressedMonthDB"
  util/update_db.py "$monthDB"
  mv -v "$monthJSON" "$monthJSON.oldFormat"
  ./create_graph_json.py "$monthDB" "$seriesName" html/data/
  util/archive_db.sh "$monthDB"
done

(do this one at a time with a script because having them all uncompressed at once will fill the disk)

@EricRahm
Copy link
Contributor Author

As of ae88a34 create_graph_json.py handles the new DB format but doesn't actually use anything but the main process. The next step is to update the script to use the process names (and annotate kinds).

@EricRahm EricRahm self-assigned this Mar 26, 2015
@EricRahm
Copy link
Contributor Author

This landed in a11f3ec on the e10s branch.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

2 participants