Major rework of BEAST files

[karllark]

From the Feb 2019 BEAST HackDay, the idea emerged that a major rework of the BEAST file data formats could provide significant benefits. One particular issue is the challenges of using the HDF5 format as it has hard for multiple BEAST developers to understand or manipulate. The benefits could be easier to understand/manipulate, less code for reading/writing -> less code maintenance, and better suited for Mega-BEAST (and BEAST) needs.

Currently, the formats are a mix of ASCII csv, hdf5, and fits files.

So, what would the format(s) of the BEAST files look like w/o the considering the current formats? In other words, if we started with a clean slate, what would we do? We understand the needs of the BEAST and Mega-BEAST much better now.

Please comment with ideas/proposals.

Existing issues that touch on this topic:
Output file format: #53
Size of the files: #295
HDF5 issues; #186, #262
unclosed files: #64
eztables: #9, #10

[karllark]

A file format that is more understandable than hdf5 would be nice. Possibilities are FITS or ASDF. But concerns about speed.

[karllark]

Fewer files would be easier to manage. Possibilities

1. Merge spec and spec_w_priors
2. Merge physics and observation models, but then harder to have separate observation (noisemodels) for a single physicsgrid file.
3. Merge output stats, pdf1d, and lnp files into a single file. Ideally have one "record" for each star fit.

[karllark]

For tables, we should use astropy.tables to take advantage of the extensive work done by the larger community.

[lea-hagen]

I'd like to change the input file format: make datamodel a text file rather than .py (similar to what is done now for the megabeast).

[karllark]

Good point. Would be great to be able to just us a text file. One impact would mean it would be easy to setup the beast to run from any directory with the BEAST scripts installed on the system.

[karllark]

HDF5 info

python h5py closer to numpy than pytables:
http://docs.h5py.org/en/stable/faq.html#what-s-the-difference-between-h5py-and-pytables

pytables more database orientated and includes non-numpy data types:
http://www.pytables.org/FAQ.html#how-does-pytables-compare-with-the-h5py-project

[karllark]

ASDF info

standard:
https://asdf-standard.readthedocs.io/en/stable/

python implementation:
https://github.com/spacetelescope/asdf

[karllark]

@meredith-durbin : I have a memory of you comparing asdf and hdf5 for speed, but can't find where your results were documented (an issue mabye?). Can you provide a pointer to your results? Or am I remembering something incorrectly and you did not do this?

[lea-hagen]

Another possibly relevant piece of information: I got this warning when running one of the large files for PHAT production runs. I have no idea if switching file formats would make this better.

/user/lhagen/miniconda3/envs/ac/lib/python3.5/site-packages/tables/group.py:489: PerformanceWarning: group ``/`` is exceeding the recommended maximum number of children (16384); be ready to see PyTables asking for *lots* of memory and possibly slow I/O.

[karllark]

That does not sound good. Any more information of what file this is for? Maybe the lnp file?

[karllark]

Notes at:
https://github.com/BEAST-Fitting/beast/wiki/DRAFT:-new-file-formats