mat2netcdf - a toolbox to convert .mat files to HDF5

[sophieclayton]

can we write a super handy toolbox to make it easier to convert our .mat files to netcdf so that we can analyse them using xarray and other fun python toolboxes? :)

[leewujung]

Yes! some commonly encountered dimensions and attributes could potentially be automatically detected and attached to the variables.

[sophieclayton]

yes, it should be specific to datasets with time/space dimensions to make it easier to build and most useful.

[dcherian]

A twist on this or second outcome could be to write a tutorial for the Xarray website showing how to create a nice Xarray object from old-style mat files (using scipy.io.loadmat) and new-style mat files (using h5py). Here's an example one showing how to reformat a not-so-nice netCDF file.

(also hi sophie!)

[sophieclayton]

hi deepak! nice tutorial :)

[mayajakes]

Hi, I have written a function to do this that is specific to EM-APEX float data, converting it from .mat to .nc with all the attributes etc..
I am already part of another project, but I am happy to share the function as an example if you like. It would have to be made more general to work with other .mat files.

[j08lue]

Sounds like the perfect inspiration for an example for the Xarray docs or a notebook to host somewhere. Please share, @mayajakes.

[abkfenris]

We've added some functionality to the OHW site this year so that we can show off demo notebooks.

[j08lue]

@sophieclayton - to define a target to work towards, how about we formulate the problem here a bit more open to various solutions and then I could interview a few potential users (you, @leewujung, @mayajakes, others from the Slack thread ?) about their requirements regarding source datasets and usage? A bit of discovery...

The solution might be a couple of code examples for the Xarray docs, a tool, a Python library - or maybe even taking to the producers of some dataset about publishing in a more self-documenting and platform-independent format or suggesting it for Pangeo Forge or so...

What do you think? How would you formulate the challenge in general terms? Like "How to turn .mat-based datasets into resources to analyze with Xarray?" or so?

[sophieclayton]

yes, this sounds great!

so, one example: @flane00 and I are working on some glider data provided in .mat files by a colleague, I think that it is a good use case here as the files have the "right" kind of variables for xarray: time, lat, long, temperature, salinity, chlorophyll, etc...
the data is organised into several individual deployments, but they each have the same structure.

it would be super handy for us to work with this data in python, so making a painless data conversion package would be AWESOME

[cathmmitchell]

@s-pinkham and I did something similar for some Slocum glider data in the Gulf of Maine using .mat files that were made by a colleague. For working with xarray (and for other reasons) we binned the data by longitude and depth which were our xarray coords (our transect is basically along an east-west line of constant latitude). It was/is pretty specific for our time series (a not quite complete repo is here), but depending on the format of your data, you might need to make some decisions about binning or reformatting for an xarray structure

[chuckwondo]

@sophieclayton, I'm interested in helping out, but at this point I see that nobody has signed up on the project page. Do you plan to work on this project this week?

[sophieclayton]

@chuckwondo no one has signed up, so this one will stay on the backburner, but it is something that I may pick up in future. I'll ping you if I work on it!

[chuckwondo]

Yes, please do ping me. I don't have any background in the domain, but I'm a software engineer interested and willing to help out, if you would like.