Lat / Lon not correctly read from forcing

[StephanThober]

Hi all,

first of all, thank you for developing this nice tool. I try to use for one catchment I am studying, but get the following error:

./ms darling_1979.conf 
Traceback (most recent call last):
  File "./ms", line 117, in <module>
    main()
  File "./ms", line 109, in main
    ms = MetSim(setup)
  File "/Users/thober/py_venvs/python3.6.4_venv/lib/python3.6/site-packages/metsim/metsim.py", line 152, in __init__
    self.met_data = io.read_met_data(self.params, self.domain)
  File "/Users/thober/py_venvs/python3.6.4_venv/lib/python3.6/site-packages/metsim/io.py", line 46, in read_met_data
    return process_funcs[params['forcing_fmt']](params, domain)
  File "/Users/thober/py_venvs/python3.6.4_venv/lib/python3.6/site-packages/metsim/io.py", line 83, in process_nc
    calendar=params['calendar'], var_dict=params.get('forcing_vars', None))
  File "/Users/thober/py_venvs/python3.6.4_venv/lib/python3.6/site-packages/metsim/io.py", line 157, in read_netcdf
    ds = ds.sel(**{d: domain[d] for d in iter_dims})
  File "/Users/thober/py_venvs/python3.6.4_venv/lib/python3.6/site-packages/xarray/core/dataset.py", line 1472, in sel
    tolerance, **indexers)
  File "/Users/thober/py_venvs/python3.6.4_venv/lib/python3.6/site-packages/xarray/core/coordinates.py", line 346, in remap_label_indexers
    obj, v_indexers, method=method, tolerance=tolerance
  File "/Users/thober/py_venvs/python3.6.4_venv/lib/python3.6/site-packages/xarray/core/indexing.py", line 237, in remap_label_indexers
    dim_indexers = get_dim_indexers(data_obj, indexers)
  File "/Users/thober/py_venvs/python3.6.4_venv/lib/python3.6/site-packages/xarray/core/indexing.py", line 205, in get_dim_indexers
    % invalid)
ValueError: dimensions or multi-index levels ['lat', 'lon'] do not exist

I attach for you here my pip list:
pip_list.txt

and the tarball of config and input files:
example.zip

Thanks in advance for your help! If you have any questions, please let me know.

Stephan

[jhamman]

Thanks @StephanThober for the report. I suspect this is one of two things.

1. We have not tested metsim with pandas==0.22 and xarray==0.10.3 so there could be a regression there.
2. One thing I noticed from your previous email to me was that you have a time dimension in your lat/lon variables in your domain file:

    	float lat(time, y, x) ;
		lat:missing_value = "-9999." ;
		lat:description = "upscaled lat from 1km to 50km using arithmetic mean" ;
	float lon(time, y, x) ;
		lon:missing_value = "-9999." ;
		lon:description = "upscaled lon from 1km to 50km using arithmetic mean" ; 

This could also be causing a problem.

[jhamman]

@arbennett may have other ideas though...

[arbennett]

Having a time dimension in the lat/lon variables may cause issues, but that doesn't appear to be what's happening here (from the traceback at least).

@StephanThober I looked at your example, but the netcdf files were not in the input directory. My guess is there's just a minor formatting error in them. Could you either post what you get from ncdump -hor a link to your input files?

[StephanThober]

Sorry, for not uploading these. I removed the time dependency in the lat and lon variable, but the error persists. Please find the zip of the input attached.
input.zip

@jhamman Which versions of pandas and xarray should work?

Thanks for your help!

[arbennett]

@StephanThober thanks for sending along your input. I looked into this and there are a couple of things that need to be done to get you up and running.

First, I would recommend making sure that you have the most up to date version of MetSim by running:
pip install git+https://github.com/UW-Hydro/MetSim@develop

There a couple of changes to the configuration file that are necessary. The issue with the lat/lon that you originally posted is because your data variables are actually indexed on x/y. This is a simple fix, you can just add the line iter_dims = ['x', 'y'] in the main section of your configuration file. This will tell MetSim to loop over those dimensions. Then one of your variable name mapping sections was reversed. You just need to change your state_vars section to:

[state_vars]
pre = prec
tmax = t_max
tmin = t_min

The domain file static.nc has a time dimension that currently will cause an error, you can simply drop it.

Finally, there is one other thing that you will need to change to make everything work. You are currently using forcing_orig.nc as both your input and state file. This file currently has one calendar years worth of data. The issue that you'll run into is that the state file requires a 90 window preceding the start date that you set in the configuration. So, for example, since your start is set to 1979/1/1 your state file would need to contain at least the dates 1978/10/3 - 1978/12/31. Note that you can still use the same file for the input and state, just that there needs to be 90 days of data before your start date (this is used to calculate some rolling seasonal averages).

Let me know if you have any other questions!

[StephanThober]

@arbennett Thanks for your advice!

I installed the latest version, corrected the config file (sorry for that), removed the time variable from the static file and changed the start time to 1979/5/1:0.

I also had to add a variable swe in the forcing_orig.nc file. In disaggregate.py in line 62, I had to change df_daily.index[-1][0] to df_daily.index[-1] and in line 64, I had to change df_daily.index[0][0] to df_daily.index[0]. Now, I get the following error:

Traceback (most recent call last):
File "./ms", line 117, in
main()
File "./ms", line 113, in main
ms.run()
File "/Users/thober/py_venvs/python3.6.4_venv/lib/python3.6/site-packages/metsim/metsim.py", line 238, in run
self.state.sel(**locs), self.disagg, times, label)
File "/Users/thober/py_venvs/python3.6.4_venv/lib/python3.6/site-packages/metsim/metsim.py", line 539, in wrap_run
df_complete = disaggregate(df, params, sg, t_begin, t_end)
File "/Users/thober/py_venvs/python3.6.4_venv/lib/python3.6/site-packages/metsim/disaggregate.py", line 81, in disaggregate
t_begin, t_end)
File "/Users/thober/py_venvs/python3.6.4_venv/lib/python3.6/site-packages/metsim/disaggregate.py", line 196, in temp
interp = scipy.interpolate.PchipInterpolator(time, temp, extrapolate=True)
File "/Users/thober/py_venvs/python3.6.4_venv/lib/python3.6/site-packages/scipy/interpolate/_cubic.py", line 100, in init
dk = self._find_derivatives(xp, yp)
File "/Users/thober/py_venvs/python3.6.4_venv/lib/python3.6/site-packages/scipy/interpolate/_cubic.py", line 147, in _find_derivatives
mk = (y[1:] - y[:-1]) / hk
ValueError: operands could not be broadcast together with shapes (493,1) (0,1)

The problem seems to be that df_daily contains no values at this location, so the error message makes sense. Is it a problem that I have cells with no data values in the forcing file?

[arbennett]

If you had to add swe to forcing_orig.nc then it seems like the latest version didn't install correctly. I would recommend uninstalling MetSim and reinstalling with:
pip install git+https://github.com/UW-Hydro/MetSim@develop

Alternatively, usually the error you posted happens with you have data points with t_min > t_max. You can try checking for that.

[arbennett]

@StephanThober just wanted to give you an update. We have released version 1.1.0, so you should be able to reinstall MetSim with pip install metsim, rather than the way I have posted previously.

[StephanThober]

@arbennett I did this, but now get another error related to Numba, that you find below. But again, it seems that the program tries to work on an empty data frame because the basin only covers a part of the netcdf file?

Is metsim able to deal with this?

Thanks in advance!

The Numba error looks like this:

./ms darling_1979.conf
/Users/thober/py_venvs/python3.6.4_venv/lib/python3.6/site-packages/dask/local.py:271: RuntimeWarning: invalid value encountered in less
return func(*args2)
Traceback (most recent call last):
File "./ms", line 11, in
sys.exit(main())
File "/Users/thober/py_venvs/python3.6.4_venv/lib/python3.6/site-packages/metsim/cli/ms.py", line 109, in main
ms.run()
File "/Users/thober/py_venvs/python3.6.4_venv/lib/python3.6/site-packages/metsim/metsim.py", line 327, in run
self.state.sel(**locs), self.disagg, times, label)
File "/Users/thober/py_venvs/python3.6.4_venv/lib/python3.6/site-packages/metsim/metsim.py", line 637, in wrap_run
sg = solar_geom(elev, lat, params['lapse_rate'])
File "/Users/thober/py_venvs/python3.6.4_venv/lib/python3.6/site-packages/numba/dispatcher.py", line 344, in _compile_for_args
reraise(type(e), e, None)
File "/Users/thober/py_venvs/python3.6.4_venv/lib/python3.6/site-packages/numba/six.py", line 658, in reraise
raise value.with_traceback(tb)
numba.errors.TypingError: Failed at nopython (nopython frontend)
Invalid usage of Function() with parameters (array(float64, 1d, C), int64)

• parameterized
  In definition 0:
  All templates rejected
  In definition 1:
  All templates rejected
  [1] During: resolving callee type: Function()
  [2] During: typing of call at /Users/thober/py_venvs/python3.6.4_venv/lib/python3.6/site-packages/metsim/physics.py (231)

File "../../py_venvs/python3.6.4_venv/lib/python3.6/site-packages/metsim/physics.py", line 231:
def solar_geom(elev: float, lat: float, lr: float) -> tuple:

sinegeom = sinlat * sindecl
coshss = min(max(-sinegeom / cosegeom, -1), 1)
^

This is not usually a problem with Numba itself but instead often caused by
the use of unsupported features or an issue in resolving types.

To see Python/NumPy features supported by the latest release of Numba visit:
http://numba.pydata.org/numba-doc/dev/reference/pysupported.html
and
http://numba.pydata.org/numba-doc/dev/reference/numpysupported.html

For more information about typing errors and how to debug them visit:
http://numba.pydata.org/numba-doc/latest/user/troubleshoot.html#my-code-doesn-t-compile

If you think your code should work with Numba, please report the error message
and traceback, along with a minimal reproducer at:
https://github.com/numba/numba/issues/new

[arbennett]

Hi @StephanThober sorry for the late reply. I have been unable to reproduce this issue, are you still experiencing it?

[StephanThober]

@arbennett : Unfortunately, yes. :( Did you use my configuration file that I provided? Could you share your pip list? I updated my numba version to 0.39.0, but this also did not help.

Just for clarification, metsim is able to deal with domains that contain entire cells with missing values?

[arbennett]

I have been testing more on your data and was able to reproduce your issue. I think the root cause was a versioning issue in xarray, but I'm not sure. The issue was that when selecting the latitude for solar geometry calculations MetSim was generating an array, when it should have just been a number. I've created a branch that guards against this, and will get it merged into the main MetSim repo ASAP.

In the mean time, if you would like to try I have my fix at https://github.com/arbennett/MetSim/tree/bugfix/lat_list and you can install it with:
pip install git+https://github.com/arbennett/MetSim@bugfix/lat_list

Let me know if this works!

[StephanThober]

@arbennett That did it! Thanks a lot Andrew. I can finally start working with this tool :)