Allow users to supply more metadata, and pass it to executors.

[lgray]

Fixes #479

Still a bit of a WIP.
At the very least this implementation does pass the metadata to the processors correctly.

[lgray]

@nsmith- take a look. Lemme know if there's a better place to put things.

[lgray]

also @alexander-held this is what you were more or less asking for right?

[alexander-held]

Yes, this looks exactly like what I had in mind. Thanks for the work on this!

[lgray]

Cool, could you give this branch a try and see if you immediately run into some problem?
I'd like to implement this more correctly, but if it works as it is now and doesn't break things... Well why not.

[alexander-held]

I gave this a try and it worked as expected. I find this quite useful for bookkeeping, as it allows me to gather metadata in one central place as opposed to having to pass it through the processor constructor.

[lgray]

@nsmith- you ok with the hacky version for now?

[nsmith-]

Can we encapsulate the metadata instead of putting it at top level at least?
e.g.

fileset = {
    "mydataset": {
        "files": ["blah.root"],
        "metadata": {"thing1": 1},
    }
}

[lgray]

Sure, had been on the fence about it. Makes some things cleaner too.

[alexander-held]

I guess the access would still be events["metadata"]["thing1"] instead of events["metadata"]["metadata"]["thing1"]?

[nsmith-]

Right, in that sense it is a little confusing since dataset is set (presumably if "dataset" were specified in the metadata it would override the current value?)

[lgray]

can just make dataset a protected name...

[lgray]

@nsmith- this look ok-enough for you?
@alexander-held please give this a try real quick.

Now you do:

filelist = {
        "DummyBad": {"treename": "Events", "files": [osp.abspath("tests/samples/non_existent.root")]},
        "ZJets": {"treename": "Events", "files": [osp.abspath("tests/samples/nano_dy.root")],
                  "metadata": {"checkusermeta": True, "someusermeta": "hello"} },
        "Data" : {"treename": "Events", "files": [osp.abspath("tests/samples/nano_dimuon.root")],
                  "metadata": {"checkusermeta": True, "someusermeta2": "world"} }
    }

and it will complain at you if you try to use any reserved words.

[alexander-held]

I gave this a try, it worked as expected. One minor suggestion: In the error message

ValueError: Reserved word "treename" in fileset dictionary, please rename this entry!

it might be useful to include "metadata"? Otherwise it may sound as if the top-level "treename": "Events" is to blame.

[lgray]

@nsmith- good from your side?