How to store user data: discuss

[pigworker]

Sometime soon, we'll need to make suitable arrangements for storing various forms of persistent data. Most of it is form data, per user per page. Some of it is configuration data, per page. Of course, there are also administrative data (information about cohorts, group assignments, etc).

I'll be the first to confess that the way I handled this in Marx was fairly ghastly. The per user per page data just lived in log files recording the time of each visit and which key-value pairs got created/mutated by that visit. The only things logged were form fields: generated content could be served to the browser but not logged. Log files were written by creating a temporary file, then cp-ing over the old log file. There was no serious attempt to deal with data races. For students, with everybody editing their own data, races aren't a big deal. Data races are more of an issue for config data being edited by staff, or people sharing the marking jointly editing the macros for common comments.

Also, it's important to be able to query these data sources with friendlier tools than grep.