Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multiple outputs and restructured ParallelSourcePlugin #190

Merged
merged 12 commits into from Jun 15, 2019

Conversation

JelleAalbers
Copy link
Member

This allows strax plugins to have multiple outputs. If you want to return incompatible kinds of rows, e.g. both records and hits, or peaks and some metadata for each chunk, multi-output support is essential.

To return multiple outputs from a plugin:

  • The provides tuple should have multiple elements, listing the provided data type names
  • The dtype and data_kind attributes should be dictionaries mapping data type names to dtypes and data kind names, respectively. The values of these dicts can be specified in the same way as the entire attribute would be for a single-output plugin
  • The compute method must return a dictionary mapping data types to results (structured numpy arrays or field/array dictionaries).
    For more info, see the updated docs, and tests/test_multi_output.py provides an example.

This also rewrites ParallelSourcePlugin. We should no longer subclass these plugins directy (as e.g. straxen's DAQReader does); instead, they are automatically created by strax when a parallel='process' plugin is followed by other plugins that can become inlined. Again, further description is in the docs. This should speed up processing from records onwards on machines with many cores, since records is then picked up by ParallelSourcePlugin just like raw_records usually is. See also the new docs and #150.

@JelleAalbers JelleAalbers merged commit 8e4afe5 into AxFoundation:master Jun 15, 2019
@JelleAalbers JelleAalbers deleted the multi_output_2 branch September 16, 2019 09:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant