Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUG: xarray data access #471

Closed
jklenzing opened this issue Jul 16, 2020 · 4 comments · Fixed by #647
Closed

BUG: xarray data access #471

jklenzing opened this issue Jul 16, 2020 · 4 comments · Fixed by #647
Assignees
Labels
Milestone

Comments

@jklenzing
Copy link
Member

jklenzing commented Jul 16, 2020

There is a possible bug with how we access xarray data found in #468. While that specific issue was solved by using xarray notation to bypass the index acces via pysat, the error analysis is documented here to help with future flake8 compliance pulls and potential bugs in other instruments. The issue arises when trying to set 2D data via indices, ie,

inst = pysat.Instrument(platform='icon', name='mighti', tag='vector_wind_green', clean_level='none')
inst.load(2020, 2)
idx, idy, = np.where(inst['Wind_Quality'] != 1)
inst[idx, idy, 'Zonal_Wind'] = np.nan

This can be traced to the except path here:

pysat/pysat/_instrument.py

Lines 565 to 569 in ec81344

try:
self.data[key[-1]].loc[indict] = in_data
except:
indict['time'] = self.index[indict['time']]
self.data[key[-1]].loc[indict] = in_data

The error triggered at 568 is

IndexError: arrays used as indices must be of integer or boolean type

It's possible that the real error is masked by the bare except statement. If I remove the try/except statement to force the main line, I get the following error at 566

KeyError: "not all values found in index 'Altitude'"

Solution needed: sort out what error the except statement is supposed to catch, then debug from there.

@jklenzing
Copy link
Member Author

Tracked down the source of the error. When passing >1D data, the except statement winds up trying to pass mixed indices (in the case above, a datetime index and integers). This needs to be updated using the latest xarray standards.

https://xarray.pydata.org/en/stable/indexing.html#

@jklenzing jklenzing added this to the 3.0.0 Release milestone Aug 14, 2020
@jklenzing
Copy link
Member Author

NOTE: this issue affects setitem and getitem in the Instrument class

@rstoneback
Copy link
Collaborator

Worth reviewing the whole setitem and getitem codes while working on this one.

@jklenzing jklenzing self-assigned this Dec 22, 2020
@jklenzing jklenzing mentioned this issue Jan 21, 2021
10 tasks
@jklenzing jklenzing linked a pull request Jan 25, 2021 that will close this issue
10 tasks
@jklenzing
Copy link
Member Author

Closing with merge of #647

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants