XENON RunDB Frontend

[tunnell]

(This is XENON-specific, still need to do #48)

This addresses Issue #73. There is a frontend that looks for data in a run database. Currently, no default is given, so the MongoURL has to be fed in. For example:

import strax

st = strax.Context(
    storage=strax.xenon.RunDB(mongo_url='mongodb://rundb-rw:XXXXX@rundbcluster-shard-00-00-cfaei.gcp.mongodb.net:27017,rundbcluster-shard-00-01-cfaei.gcp.mongodb.net:27017,rundbcluster-shard-00-02-cfaei.gcp.mongodb.net:27017/test?ssl=true&replicaSet=RunDBCluster-shard-0&authSource=admin&retryWrites=true'),
    register_all=strax.xenon.plugins,
    register=strax.xenon.pax_interface.RecordsFromPax
)

d = st.get_array('170321_0620', 'raw_records')

The way it works is by searching the RunDB for data of interest (this is the only RunDB call!). If a run document exists, then it tries to determine if it knows how to read it. S3 can be read anywhere. A fuzzy hacky match is used (like in hax) to determine if you're running on dali, where other XENON data is. At the moment, this interface cannot write RunDB aware. This is reserved for snax.

This relies on a RunDB mirror hosted through MongoDB Atlas on Google Cloud Platform. Which data is available can also be seen here.

[tunnell]

I just realized: this PR also includes a cleanup and fix of the S3 interface.

[tunnell]

This is one use.

[tunnell]

This is another.

[briedel]

To understand this correctly, strax only queries for the location of the data of a certain type?

[tunnell]

At present, yes. Writing to the RunDB will be more centralized I'm guessing (cax/snax) since not everybody needs to write. This will be more complicated and I can link there (just factored runDB calls into one place to make easier to replace).

The real reason it doesn't write is because writing to the RunDB happens in multiple threads and we couldn't figure out how to do it just once (yet). @briedel

[JelleAalbers]

@briedel We will probably add some more rundb calls later for fetching of run-level metadata and possibly some configuration.

RunDB write support should come soon, but not in this first concept:

• We need to add a hook when data is done writing that calls back to the storage frontend (mongodb in this case). First of all we'll have to change the storage base classes a bit for this.
• Next, we'll hit a problem where we can't put this callback it in the most obvious place (Saver) in the most obvious way (attribute) because Savers can get forked over multiple processes (in low-level multiprocessing, the ParallelSourcePlugin), and you can't put unpickleable attributes (like callbacks, mongo clients, etc.) on things that will be forked. This isn't really a problem, as the forked versions don't need the callback, so we just have to erase it just before forking.

Even if we only use snax for writing, it's still useful to have write support in the strax frontend, to simplify snax's life.

[tunnell]

PR good?

[JelleAalbers]

Was this accidentally commented out or do we no longer need it?

[JelleAalbers]

Could you catch this in get_metadata and raise DataNotAvailable there? This would fix #90. (Then here, you'd catch DataNotAvailable again if you're writing.)

The S3 frontend is unusual in that _find relies on get_metadata rather than the other way around. But the context relies on DataNotAvailable being raised if get_metadata fails, so it can try other storage frontends (https://github.com/AxFoundation/strax/blob/master/strax/context.py#L632).

[JelleAalbers]

Actually it turns out #90 has already been fixed, so never mind.

[JelleAalbers]

Maybe add the caveats about writing you wrote in the PR here. If I remember correctly, writing is supported, but it doesn't get registered back in the rundb.

Also perhaps worth noting the chosen write backend is files if you're on dali and S3 otherwise (right?).

[JelleAalbers]

Looks good! I left a few comments that might be useful to look at (one would fix #90 I think), but feel free to merge.