Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUG: xarray and multi_file_day and data padding. #1005

Closed
rstoneback opened this issue May 5, 2022 · 4 comments · Fixed by #1092
Closed

BUG: xarray and multi_file_day and data padding. #1005

rstoneback opened this issue May 5, 2022 · 4 comments · Fixed by #1092
Assignees
Labels
Milestone

Comments

@rstoneback
Copy link
Collaborator

Describe the bug
Checks on the time index aren't consistent across pandas and xarray. Things work for pandas, they don't for xarray.

To Reproduce
Steps to reproduce the behavior:

  1. gold = pysat.Instrument("ses14", "gold", "nmax")
  2. gold.multi_file_day = True
  3. gold.load(2020, 1)
  4. See error
File ~/Code/pysat/pysat/_instrument.py:2935, in Instrument.load(self, yr, doy, end_yr, end_doy, date, end_date, fname, stop_fname, verifyPad, use_header, **kwargs)
   2933     self._prev_data.sort_index(inplace=True)
   2934 if not self._index(self._curr_data).is_monotonic_increasing:
-> 2935     self._curr_data.sort_index(inplace=True)
   2936 if not self._index(self._next_data).is_monotonic_increasing:
   2937     self._next_data.sort_index(inplace=True)

AttributeError: 'Dataset' object has no attribute 'sort_index'

The error is on _curr_data and not _prev_data since there is no data for _prev_data

Expected behavior
Gold data goes past date boundaries. Trying to clean that up with existing pysat features. Lack of testing got us here :(

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • OS: MacOS
  • Version [e.g. 22] Python 3.9
  • Other details about your setup that could be relevant

Additional context
🐬

@rstoneback
Copy link
Collaborator Author

This is only a problem if the times aren't monotonic.

@rstoneback
Copy link
Collaborator Author

rstoneback commented Jun 8, 2022

Short term fix, put in type check and put in correct code for xarray.

@rstoneback rstoneback added this to the 3.1.0 Release milestone Jun 8, 2022
@jklenzing jklenzing added the bug label Mar 6, 2023
@jklenzing
Copy link
Member

xarray.Dataset does not have the sort_index method. It needs data.sortby('time')

@jklenzing jklenzing self-assigned this Mar 28, 2023
@jklenzing
Copy link
Member

I've got a fix in the queue, but testing is going to be a nightmare. We currently do not test for a non-monotonic index in the padding tests. Adding the malformed_index kwarg into the tests currently has 12 broken tests, 6 passing tests per class.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants