A work item that came out of the Socorro Postgres work week is to dump the dump tables and store cooked dumps as gzipped files. Drop dumps table
convert each dumps table row to a compressed file on disk
'done' as of 2009-05-07 in socorro.lib.dmpStorage (Coding, testing is done; integration testing is done, 'go live' is today) Socorro UI
Will stop using the dumps table.
- Will start using gzipped files
Will use the report uuid to locate the dump on a file system
Will use apache mod-rewrite to serve the actual file. The rewrite rule is based on the uuid, and is 'simple': AABBCCDDEEFFGGHHIIJJKKLLM2090308.jsonz => AA/BB/AABBCCDDEEFFGGHHIIJJKKLLM2090308.jsonz
report/index will include a link to JSON dump
link rel='alternate' type='application/json' href='/reporter/dumps/cdaa07ae-475b-11dd-8dfa-001cc45a2ce4.jsonz'
- Will be gzip compressed JSON encoded cooked dump files
- Partial JSON file
- Full JSONZ file
application.conf dumpPath Example for kahn $config'dumpPath'? = '/mnt/socorro_dumps/named';
In the dumps directory we will have an .htaccess file:
AddType "application/json; charset=UTF-8" jsonz AddEncoding gzip jsonz
Webhead will serve these files as:
Content-Type: application/json; charset=utf-8 Content-Encoding: gzip
**Note:* You'd expect the dump files to be named json.gz, but this is broken in Safari. By setting HTTP headers and naming the file jsonz, an unknown file extension, this works across browsers.
- Existing URL won't change.
- Second JSON request back to server will load jsonz file
mod rewrite rules will match /dump/.jsonz and change them to access a file share.
A future enhancement if we find webheads are high CPU would be to move populating the report/index page to client side.
http://people.mozilla.org/~aking/Socorro/dumpingDump/json-test.html - Uses browser to decompress a gzip compressed JSON file during an AJAX request, pulls it apart and appends to the page.
Test file made with gzip dump.json