improve I/O tools for working with data

[hawkrobe]

Importing data from a file

We want a read() function for the command-line, which reads data from the given file and transforms it into an easily processed data structure for use in webppl functions. We expect this to be used primarily for csv files, but it would be nice to support other input types as well.

The call would be read(filename, [opts]), where opts is an object containing any relevant parameters.

Here are some file types we might want to consider:

• CSV: Needs to properly get the types of numeric values, strings, and booleans. opts could specify the delimiter (default to comma), whether there's a header (default to true), and so on. We could implement the internals using nodeCSV or babyparse.

  We thought the best output would be a list of objects, indexed by header names, with each element in the list corresponding to a row of the input csv. This would support post-processing steps like filtering, plucking a single column, and omitting irrelevant values.

    [{headerLabel1 : value1,
      headerLabel2 : value2,
      headerLabel3 : value3,
      ...},
     {...}]

• JSON: Could be implemented using a combination of fs and JSON.parse:

    var data = JSON.parse(fs.readFileSync('file', 'utf8'));

• text: For NLP applications, sometimes you just want to pull in a bunch of text. opt could take the encoding to use.
• images: For vision applications, it might be cool to be able to read in a .png or .jpg and parse it into a matrix of pixels. Could be implemented using node-opencv.

We can determine the file type by using the file extension. If it is not .csv, .tsv, .json, or in the set of image encoding we support, it could default to text.

Writing ERPs to file

We want a write() function that will write out one or more ERPs to a file for post-processing and analysis in other languages like R or python. We want to write to a .csv in long-form. There are two major issues to consider here:

1. The results of many model-fitting exercises are ERPs with lists of lists or lists of objects in the support. To put this in long-form, we need to write one line for each of these internal lists or objects, which share the same probability. We'd also like to be able to support multiple sets of object keys: one element in the support may be {type : 'a', alphaVal : .5} and another may be {type: 'b', betaVal : 1.5}. We'd like to write a csv with 'type', 'alphaVal' and 'betaVal' as column headers and NAs filled for rows where they aren't specified.
2. For model comparison applications, we often want to write multiple ERPs to the same file with one or more labels identifying which ERP is which. To put this in long-form, we need to prepend these labels to each row. We also need to be able to specify whether we're going to append to the given file or write a new one.

It seems like the right call for this is to first create a writer object:

var myWriter = csvWriter(filename, mode)

where mode can either be w (write) or a (append). This is kind of copying the python way of doing things. Then pass this writer object into the write function with an object setting various options:

write(myWriter, opts)

The two options we had in mind are (1) parameterHeaders, which specifies which headers will be found in the list of lists or list of objects, and (2) additionalLabels, the list of labels to prepend to each line of the csv.

[hawkrobe]

@mhtess @feste Did I forget anything?

[ngoodman]

this seems good, thanks!

do you think it's worth a more explicitly r-style dataframe object as one target of file reading and writing? or would we get most of the benefit from the embedded object idiom you describe above? are there idiomatic dataframe manipulation (ala hadley wickham) that we don't get simply by using HoFs (map/fold) on the objects you describe? we could ask @mcfrank for opinions.

[mhtess]

it's not obvious to me at this point that we need the dataframe object. i've started documenting various data munging tricks on this wikipage. the underscore library gets us a lot with the embedded object idiom. populating the wikipage with our various idioms will give us a better idea.

one half-baked idea for incorporating the hadley-verse could be something like

var model = function(data){
    var theta = beta(1,1)
    factor(data)
    return theta
}

df.tidy %>% 
     group_by(condition, participant) %>%
     summarise(theta = Infer(model))

though I'm not really sure this syntax is going to work.

Also, though this might be useful for some cases, there are many cases where we will want cross-cutting hierarchical structure: Some parameters shared by participant, others by condition (the image in my head is of a graphical model with non-overlapping plates). My feeling is that the hadleyverse is not suited for those cases, though I could be mistaken.

[mhtess]

It may be useful to have some basic loadERP() and saveERP() functions that behave similar to R's save() and load() commands. In R, these commands, respectively, save a data frame as a .RData file, and load an .RData file as a ready-to-go data frame. We could have a webppl erp filetype: .erp. And you could do something like

var priorERP = loadERP("empiricalPrior.erp")
var model = function(){
    var x = sample(priorERP)
    factor(x > 0 ? 0 : -Infinity)
    return x
}

This will be useful in the following case:
Let's suppose you have a program with 2 models for data from 2 experiments. The first experiment is very straightforward to model and the second one is a bit trickier (i.e. involves a lot more chopping, hacking, and reformulating). In this case, it would be nice to run inference on model1 (experiment 1), and save the result, so that you don't have to rerun it every time you are testing a new version of model2.

Also, this would aid in reproducibility for our scientific projects. (Also, in our code development.)

[null-a]

I removed the CSV reading/writing example from the README while documenting the new package system. @stuhlmueller suggests creating a webppl-csv package as part of this issue to compensate.

[hawkrobe]

Yes! I agree that a separate webppl-csv package is the way to go. I'll leave this issue here until we start a repo for that project, then we can move the specs over there. Whenever it's done, we could use it in the README as an example of how to import and use external packages.

[mhtess]

Is it feasible to get webppl to read files from the web? R allows you to put in a website for reading files. This would be useful for teaching Bayesian data analysis using a webppl web-book. Right now, just copying a csv into the a code box quickly blows the stack (is this right?)

[stuhlmueller]

This is feasible in the browser using a CORS proxy such as crossorigin.me. Example:

$.ajax({
  url:'http://crossorigin.me/http://viewportsizes.com/devices.csv',
  type: "GET",
  error: function(req, message) {
    alert(message);
  },
  success: function(data) {
    alert(data);
  },
  dataType: "text"
});

One would have to be a little careful in how exactly one makes this available to webppl (since it's asynchronous), but it would be feasible.

[hawkrobe]

I recently implemented a function making asynchronous API calls from webppl. You had to do a cool and kinda tricky thing with the continuation and trampoline to get it to return properly (which essentially forces it to be synchronous, as far as webppl is concerned). That code is here, if it’s useful!

[ngoodman]

I believe most of the ideas in this thread are covered by packages now. possibly with the exceptions of reading/writing distributions? (though if distributions are serializable, then we can deal with reading/writing them as json....)

I don't think anything else in here needs to happen for the ppaml summer school, so i'll move this to someday...