Using processDirs to run arbitrary command

[shihchengyen]

Is there a way to have processDirs go through directories and run an arbritrary command? We want to be able to go through a bunch of directories to create an object and save it, but we think we are running into memory issues in creating the cumulative object. So we would like to create and save the object, but not add them together. We used to use the ProcessLevel('nptLevelCmd') option to do something like this.

[grero]

I'm working on something like this in #53. Again, I have forgotten how processLevel used to do this. Would we pass in a function, or a string of code that then then got eval'ed? The way the DPT code currently works is that you'd create a function that does whatever you need it do to, and then you'd decorate that function with DPT.misc.processDirs(dirs), where dirs contains the list of directories you want to process. For example

import DataProcessingTools as DPT

@DPT.misc.processDirs(dirs)
def create_obj(Obj, *args, **kwargs):
    Obj(*args, **kwargs)

This would basically just create the object Obj with arguments *args and **kwargs in each directory of of dirs.
We can discuss a bit around how we want do do this. In particular, I think this requires the code to be written as a function, like the example I gave.

[grero]

We could also do it simpler, by just having a processLevel function that accepts a function func as its first argument, and then we'd do

DPT.misc.processLevel(func, dirs, *args, **kwargs)

[shihchengyen]

I think we used to just pass a string to that got eval'ed. Essentially we would like to be able to do something like:
DPT.objects.processDirs(None, Cmd=['session', 'pyh.Unity(saveLevel=1)'])
which will specify that the command pyh.Unity(saveLevel=1) should be run from the session directories that are found under the current directory.
Not sure whether it will be easy to do, but another thing that we used to in NPT is to loop over directories and return user specified data without having to create objects:
[nd,data] = ProcessDirs(nd,'nptDirCmd','[n,x,nbins,xcenter,ycenter] ...
= hist(eyejitter(''auto''),''BinSize'',1);data = [data; xcenter ...
ycenter];');
This gave us flexibility to extract data that was useful for each directory, add it to data, and then have data returned by the function.

[grero]

Latest #53 now has the function DPT.levels.processLevel(level, cmd) which I think does this. It will run the code in the stringcmd for each directory corresponding to level under the current directory. The code can optionally append to the variable data, which will be returned, along with the directories visited.

[grero]

Latest master now allows this.