Extract metadata from FITS files for searching and filtering

[zonca]

          > Thanks! I would like to extract metadata from FITS header and store in the portal for querying and filtering.

Where is the best location for the metadata?

I suggest creating creating JSON files along side the FITS and DAT files. Like datareleases/npipe6v20/lowres/npipe6v20_030_map_ns0016.fits would have a file datareleases/npipe6v20/lowres/npipe6v20_030_map_ns0016.fits.json.

And put these in the Globus collection with the files. We can then aggregate some of the metadata from the individual files into the dataset metadata, along with the core elements, like title, creation date, and publisher. E.g., datareleases/npipe6v20/lowres/metadata.json. This metadata can be combined to update the dataset descriptions, table of files, and populate a searchable JSON catalog to use with lunr.js or similar.

Originally posted by @rpwagner in #4 (comment)

[rpwagner]

@zonca do you want to have a short call next week to talk about dataset organization? Right now, there's a lot of files in each directory and I don't know enough about the data to suggest how to break them up. I'm interested in what's a reasonable breakdown so that users can find the datasets and files to do their analysis or other work.

[zonca]

We plan to have another dataset ready soon to be added to the portal.
I think it will be easier to design an overall organization for the data with 2 datasets. So let's wait for that.