Context options, storage conversion, and fuzzy data matching

[JelleAalbers]

This adds three things:

Context options

These options control how Contexts behave, e.g. on data matching and conversion. They are different from the existing normal config options, which specify what exact data you want instead. You set these new context options by passing additional keyword arguments to Context.__init__, or any function that can make contexts on the fly, such as get_df and the other get_xxx functions. If you call .show_config() on a context, you get a DataFrame with available context options and their current values:

option	default	current	data_type	help
storage_converter	False		Context	If True, save data that is loaded from one frontend through all willing other storage frontends.
fuzzy_for	()		Context	Tuple of plugin names for which no checks for version, providing plugin, and config will be performed when looking for data.
fuzzy_for_options	()		Context	Tuple of config options for which no checks will be performed when looking for data.

(OK, the word 'data_type' above is an artifact from re-using the config system for plugins. Probably it should be 'applies to' instead, or just be gone altogether.)

Storage conversion

Strax saves newly computed data through every willing storage frontend, but data that exists in one frontend was not so easy to transfer to another -- see issue #25.

A boolean context option storage_converter can now be set. If True, it will save any data loaded from one frontend, using regular data loading commands, through all willing other storage frontends. Try it with e.g st.get_df('event_info', storage_converter=True) if you have two storage frontends registered.

Fuzzy matching of data

Previously, strax could only load exactly the data specified by the context (registered plugins / specified config options). For example, to load data produced by a non-standard raw_records maker, such as the pax converter plugin, you'd first have to register that plugin. Sometimes this is too much trouble and you want strax to be more lenient. Two context options are available:

• fuzzy_for: Tuple of plugin names for which no checks for versions, providing plugin, and config will be performed when looking for data.
• fuzzy_for_options: Tuple of config options for which no checks will be performed when looking for data.

All new functionality has associated tests in test_core.

A few minor changes:

• I split core.py into two files: context.py and config.py. The former holds only the Context class, the latter various utilities regarding configuration.
• the search_config method is now called show_config.

[JelleAalbers]

The file splitting causes our code linters to complain:

• CodeFactor doesn't like our import *s in __init__.py (even though this is standard practice when combined with __all__, see e.g. https://github.com/numpy/numpy/blob/master/numpy/__init__.py#L143), and due to the splitting we now have one more.
• Codacy seems to flag all its existing beefs with context.py as 'new issues' now that they have moved to two new files.

[tunnell]

I told it to ignore those import * in future. Maybe just on that one file?

[tunnell]

Sweet, will try out. Looks good. Only thing I'm realizing is that this fuzzy matching means that I have to upload this S3 PR.

[tunnell]

Will this mean that the strax key changes based on whether or not convertor? Just wanting to make sure that it doesn't change lineage.

[JelleAalbers]

It won't change the lineage. Context options are not tracked with the data, as they change how you load but not what you load.

[tunnell]

Does this mean that I need to update my S3 PR? Or just raise NotImplementedError?

[JelleAalbers]

Yes, if you don't have fuzzy matching implemented in a frontend, you can raise NotImplementedError whenever someone tries to use these options.

Unfortunately the best implementation of fuzzy matching is frontend specific: for filesystem directories we have to laboriously scan through directories and load jsons until we find one that matches (unless the exact match with the hash succeeds of course), for the run db we can use a mongo query.

[tunnell]

Same question, do I have to update other frontends?

[tunnell]

Merge and minor? Then I'll tell the S3 bindings more with this before merge.

[tunnell]

@JelleAalbers I'd look at the Codacy issues though. Those are real. Unused variable for example.

[JelleAalbers]

Thanks for the review! I'll merge, maybe we can release a new version at the end of the week just before the workshop?

Yeah, some of the issues are real and we should have a look through them. It was just confusing they come up with this PR just because we moved some code to another file.

[tunnell]

Sure we can release later. I've just been using the releases for processing, but I guess I won't update yet. The pip install is still broken, but will fix next release. Need to update HISTORY.md