[FEATURE-REQUEST] Support netCDF-4 files

[snitramodranoel]

Any suggestions on how to make vaex support nc-4 files, which are themselves backed by hdf5?

When attempting to read a netCDF-4 file with a given variable in (x,y,z,t) coordinates this is what I get:

ValueError: array is of length 360, while the length of the DataFrame is 2928

This netCDF-4 file contains 360 latitude values, and 2928 time points, as evidenced by ncdump -h:

netcdf GSWP3.BC.Rainf.3hrMap.1940 {
dimensions:
	time = UNLIMITED ; // (2928 currently)
	lat = 360 ;
	lon = 720 ;
variables:
	double time(time) ;
		time:long_name = "Time" ;
		time:standard_name = "time" ;
		time:calendar = "proleptic_gregorian" ;
		time:units = "hours since 1871-01-01 00:00:00" ;
	float lat(lat) ;
		lat:long_name = "Latitude" ;
		lat:standard_name = "latitude" ;
		lat:axis = "Y" ;
		lat:units = "degrees_north" ;
	float lon(lon) ;
		lon:long_name = "Longitude" ;
		lon:standard_name = "longitude" ;
		lon:axis = "X" ;
		lon:units = "degrees_east" ;
	float Rainf(time, lat, lon) ;
		Rainf:_fillvalue = 1.e+20 ;
		Rainf:long_name = "Rainfall rate" ;
		Rainf:standard_name = "rainfall_flux" ;
		Rainf:alma_name = "Rainf" ;
		Rainf:amip_name = "prra" ;
		Rainf:units = "kg m-2 s-1" ;
// global attributes:
		:title = "Global Soil Wetness Project 3 <http://hydro.iis.u-tokyo.ac.jp/GSWP3> EXP1 forcing data" ;
		:id = "GSWP3.BC.Rainf.3hrMap.1940" ;
		:creator = "Hyungjun Kim <hjkim@iis.u-tokyo.ac.jp>" ;
		:institution = "Institute of Industrial Science, The University of Tokyo" ;
		:contributors = "Satoshi Watanabe, Eun-Chul Chang, Nobuyuki Utsumi, Kei Yoshimura, Gilbert Compo, Hirabayashi Yukiko, James Famiglietti, and Taikan Oki" ;
		:conventions = "ALMA-3, CF-1.6, AMIP" ;
		:history = "| 0.5b | 2014-06-22 | first beta release\n| 0.6b | 2014-07-18 | remove negative values from Qair\n| 0.7b | 2014-10-15 | add netCDF description\n|                   | beta version release\n| 0.8b | 2017-05-12 | correct wind using CRU-CL2.0\n| 0.9b | 2017-05-13 | correct bias of reference Prcp (GPCC v7) in early 20th Century\n|                   | reproduce Rainf/Snowf corresponding to updated wind speed\n| 1.0  | 2017-06-01 | version 1 official release\n| 1.05 | 2018-02-17 | timespan extended (1901-2014)\n|                   | fill wind values over ocean pixels with downscaled reanalysis\n| 1.06 | 2018-07-06 | change variable naming convention: amip_name [wss] -> [sfcWind]" ;
}

[norlandrhagen]

Seconded! This support would be really useful for weather/climate/oceanography data.

[maartenbreddels]

Yeah, should be possible. Once we have #865 in, it should also be easier to add new file formats. Would be a good example/usecase on how to add that. Anyone willing to cooperate/work on this?

[Hossein-Madadi]

Yeah, should be possible. Once we have #865 in, it should also be easier to add new file formats. Would be a good example/usecase on how to add that. Anyone willing to cooperate/work on this?

I have many nc files and after many search to select between Dask and Vaex, I decided to use vaex with xarray in my research. Based on your comment it is possible to read nc files with vaex. But I can not do that!. is it possible to help me to do that?

[maartenbreddels]

Hi Hossein,

thanks for approaching us. This is certainly possible. I can point you to code that might help you:
Internally (so it's still a private API, but that will change), we use a Dataset class that wraps the real data, an example of wrapping an arrow dataset can be found here:

vaex/packages/vaex-core/vaex/arrow/dataset.py

Line 60 in 9289005

class DatasetArrow(vaex.dataset.Dataset):

This example is a bit complex, and I could possibly make an example that would be easier to digest. For instance a lazy csv loading dataset could be an example.

The unittest for the Dataset API can be found here:
https://github.com/vaexio/vaex/blob/master/tests/dataset_test.py

which might help.

Take a look at it, and if you want to have a higher bandwidth channel (Slack/Video), let me know so I can get you up to speed.

Regards,

Maarten

[Hossein-Madadi]

Sorry Maarten, I'm a little confused about how to change the code you introduced. It would be nice if you could teach me how to read a NetCDF (nc) file with an example.

Best Regards
Hossein

[maartenbreddels]

Hi Hossein,

I assumed that you wanted to contribute, my wrong. I was wondering what the use case is. Isn't NetCDF multidimensional data? And is xarray not better for that?

Regards,

Maarten

[Hossein-Madadi]

Hi Maarten,
I'm sorry, I think I could not describe my problem correctly. My NetCDF file is multidimensional data but when I want to use many of NetCDF files with xarray, I have memory problem. My search shows the vaex have many advantages for analysis and visualization in compared with Dask. My few days of googling eventually led me to this page.
I hope support netcdf format add as soon as possible in future version.
Thanks for your attention.

Best Regards
Hossein