Data processing

[eggerdj]

Co-authored-by: Naoki Kanazawa nkanazawa1989@gmail.com

Summary

This PR introduces the data processing package which was carved out from PR #20.

Details and comments

Data processing is the steps required to prepare the measured data for analysis. This is done using a DataProcessor which is a chain of DataActions, i.e. transformations applied to the data in place. A user can specify the actions to apply on the data to process it. For example, the code

processor = DataProcessor()
processor.append(Kernel(my_kernel))
processor.append(ToReal(scale=1e-3))

Creates a data processor that would take level 0 data, apply a kernel to create IQ data, and then take the real part of this IQ Data while scaling it by a factor 1e-3. Similarly, the data processor

processor = DataProcessor()
processor.append(Discriminator(my_discriminator))
processor.append(Population())

would take IQ data as input, discriminate it into counts and then convert these counts to a population.
An instance of DataProcessor is then used to process data by doing, for example,

data = exp_data.data[0]
processor.format_data(data)

Here, exp_data is an instance of ExperimentData. The data processor will modify data, an instance of Dict[str, Any], in place. Each node in the processor looks for the type of data it uses as input in data. For example, the Population() node will use data['counts'] to create populations which it will insert into data by doing data['populations'] = .... This makes an instance of DataProcessor reusable on different input data. Furthermore, since the different steps are contained in the data we can easily check the outcome of each processing step. Finally, each node in the DataProcessor defines the key under which it stores its output data so that we can easily retrieve, from the processor, the output data. Indeed processor.output_key() returns the data output key of the last node in the processing chain. For the examples above this output key would be memory_real and populations, respectively.

[chriseclectic]

@eggerdj Can you post some comments on the design and code examples of how this is intended to be used to make this easier to parse?

[nkanazawa1989]

Thanks @eggerdj , this looks really great. Few comments for implementation details.

[nkanazawa1989]

LGTM! thanks for addressing all of my concerns and comments.

[chriseclectic]

Looks good. I have several suggestions that we can discuss more. My main ones are to do with making this perhaps a little more Functional (as in programming), so that individual nodes or the whole data processor could be used interchangeably with regular functions / callables in the code base. (Maybe this is similar to how numpy ufuncs are actually classes that store metadata about their input and output dimensions and dtypes.)

[nkanazawa1989]

Just minor comments before approval, overall looking good.