Error with the test data set

[abhisekocean]

HI,

I am new to the glidertool and processing of glider data. Recently we have started a project with SLOCUM G3 gliders and wanted to explore this tool for higher-level processing.

So, I installed this tool using conda and downloaded the test/data directory from the Github. While running the test case as suggested in "https://glidertools.readthedocs.io/", I have now encountered an error as given below and not sure how to solve it.

ds_dict = gt.load.seaglider_basestation_netCDFs( filenames, names, return_merged=True, keep_global_attrs=False )

It is showing the below error:

DIMENSION: sg_data_point
{latitude, ctd_pressure, ctd_time, eng_wlbb2flvmt_wl700sig, aanderaa4330_dissolved_oxygen, temperature, longitude, eng_wlbb2flvmt_Chlsig, ctd_depth, eng_wlbb2flvmt_wl470sig, salinity}
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14/14 [00:00<00:00, 80.27it/s]

DIMENSION: qsp2150_data_point
{eng_qsp_PARuV, aanderaa4330_results_time}
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████| 14/14 [00:00<00:00, 123.11it/s]

Merging dimensions on time indicies: sg_data_point, qsp2150_data_point,

ValueError Traceback (most recent call last)
/tmp/ipykernel_681/556921051.py in
----> 1 ds_dict = gt.load.seaglider_basestation_netCDFs(
2 filenames, names,
3 return_merged=True,
4 keep_global_attrs=False
5 )

~/miniconda3/lib/python3.9/site-packages/glidertools/load/seaglider.py in load_multiple_vars(files, variable_names, return_merged, verbose, keep_global_attrs, netcdf_attrs, keep_variable_attrs)
495 df_other = data[other]
496 print(other, end=", ")
--> 497 df_merged = merge_dimensions(df_merged, df_other, interp_lim=1)
498 data["merged"] = df_merged
499 drop_names = list(data["merged"].filter(regex="_drop").columns)

~/miniconda3/lib/python3.9/site-packages/glidertools/utils.py in merge_dimensions(df1, df2, interp_lim)
130 if same_type:
131 df = df1.join(df2, sort=True, how="outer", rsuffix="_drop")
--> 132 df = df.interpolate(limit=interp_lim).bfill(limit=interp_lim)
133 return df.loc[df1.index]
134 else:

~/miniconda3/lib/python3.9/site-packages/pandas/util/_decorators.py in wrapper(*args, **kwargs)
309 stacklevel=stacklevel,
310 )
--> 311 return func(*args, **kwargs)
312
313 return wrapper

~/miniconda3/lib/python3.9/site-packages/pandas/core/frame.py in interpolate(self, method, axis, limit, inplace, limit_direction, limit_area, downcast, **kwargs)
10707 **kwargs,
10708 ) -> DataFrame | None:

10709 return super().interpolate(
10710 method,
10711 axis,

~/miniconda3/lib/python3.9/site-packages/pandas/core/generic.py in interpolate(self, method, axis, limit, inplace, limit_direction, limit_area, downcast, **kwargs)
6888 "those NaNs before interpolating."
6889 )
-> 6890 new_data = obj._mgr.interpolate(
6891 method=method,
6892 axis=axis,

~/miniconda3/lib/python3.9/site-packages/pandas/core/internals/managers.py in interpolate(self, **kwargs)
375
376 def interpolate(self: T, **kwargs) -> T:
--> 377 return self.apply("interpolate", **kwargs)
378
379 def shift(self: T, periods: int, axis: int, fill_value) -> T:

~/miniconda3/lib/python3.9/site-packages/pandas/core/internals/managers.py in apply(self, f, align_keys, ignore_failures, **kwargs)
325 applied = b.apply(f, **kwargs)
326 else:
--> 327 applied = getattr(b, f)(**kwargs)
328 except (TypeError, NotImplementedError):
329 if not ignore_failures:

~/miniconda3/lib/python3.9/site-packages/pandas/core/internals/blocks.py in interpolate(self, method, axis, inplace, limit, fill_value, **kwargs)
1368 new_values = values.T.fillna(value=fill_value, method=method, limit=limit).T
1369 else:
-> 1370 new_values = values.fillna(value=fill_value, method=method, limit=limit)
1371 return self.make_block_same_class(new_values)
1372

~/miniconda3/lib/python3.9/site-packages/pandas/core/arrays/_mixins.py in fillna(self, value, method, limit)
216 self: NDArrayBackedExtensionArrayT, value=None, method=None, limit=None
217 ) -> NDArrayBackedExtensionArrayT:
--> 218 value, method = validate_fillna_kwargs(
219 value, method, validate_scalar_dict_value=False
220 )

~/miniconda3/lib/python3.9/site-packages/pandas/util/_validators.py in validate_fillna_kwargs(value, method, validate_scalar_dict_value)
370 raise ValueError("Must specify a fill 'value' or 'method'.")
371 elif value is None and method is not None:
--> 372 method = clean_fill_method(method)
373
374 elif value is not None and method is None:

~/miniconda3/lib/python3.9/site-packages/pandas/core/missing.py in clean_fill_method(method, allow_nearest)
118 expecting = "pad (ffill), backfill (bfill) or nearest"
119 if method not in valid_methods:
--> 120 raise ValueError(f"Invalid fill method. Expecting {expecting}. Got {method}")
121 return method
122

ValueError: Invalid fill method. Expecting pad (ffill) or backfill (bfill). Got linear

Any help will be much appreciated.

Thanks,
Abhisek

[dhruvbalwada]

Glad you found this package.

Could you please give some more details about your problem (e.g. as suggested as here: http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports).

At the very least tell us what the variables you are passing to the function contain? Are you running the code from the example?

Have you looked at the demos here: https://github.com/GliderToolsCommunity/glidertools-demo ?

[abhisekocean]

Dear @dhruvbalwada

Thanks for your quick response.

I am following the steps given in
https://glidertools.readthedocs.io/en/latest/loading.html

I found that the referred demo directory listed the same steps given in the above link. Anyways downloaded the demo directory and tried to execute the "Importing glider data.ipynb". But, getting the same error. Not sure if this is a problem with the installation.

Note that I am using Python 3.9.1 version. Is that could be a problem?

I wanted to successfully execute the glidertools with these example datasets to make sure that everything is working fine.

Thanks,
Abhisek

[jbusecke]

Hi @abhisekocean, I think this is in fact not a user error on your side. Our CI is failing with a similar error.

Let me try to figure out what is going on here.

[jbusecke]

This behavior might have been caused by a new pandas version. Previous runs using pandas 1.2.5 succeeded. Can you confirm your version of pandas installed? If you have >1.2.5, can you downgrade and see if the error resolves itself?

[abhisekocean]

Dear @jbusecke

Thanks for the suggestion.

I could now run the test code successfully (tested till plotting the "original data") after downgrading the pandas version to 1.2.5. Earlier I was using version 1.3.0 which resulted in that error.

Thank you very much