Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixed XRSTimeSeries quality flags from breaking parsing of some XRS data. #6410

Merged
merged 5 commits into from
Sep 22, 2022

Conversation

alasdairwilson
Copy link
Member

PR Description

Closes #6409

Currently #6260 checks for 2 possible netcdf names for the XRS quality flag column 'xrs[a/b]_flags' and '[a/b]_flag' , this stops TimeSeries from loading in some XRS data files due to KeyError when neither of those exist.

This PR adds support for a 3rd flag column name: 'xrs[a/b]_flag' but maybe there is a 4th (or more) out there though

@alasdairwilson alasdairwilson requested a review from a team as a code owner September 16, 2022 12:27
@nabobalis nabobalis added BugFix timeseries Affects the timeseries submodule labels Sep 16, 2022
@@ -235,8 +235,12 @@ def _parse_netcdf(filepath):
flux_name_b = flux_name_a.replace("a", "b")
xrsa = np.array(h5nc[flux_name_a])
xrsb = np.array(h5nc[flux_name_b])
xrsa_quality = np.array(h5nc[flux_name_a.replace("flux", "flags")])
xrsb_quality = np.array(h5nc[flux_name_b.replace("flux", "flags")])
flux_flag_a = h5nc.variables.get("a_flags") or h5nc.variables.get("xrsa_flags") or h5nc.variables.get("xrsa_flag")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is quite annoying that they vary at least 3 ways.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know the stuff has been running for 60 years or something so changes are excused but there is so much stuff like this with GOES which make it a pain.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should perhaps think about this some more and maybe have specific 1m/ 2s/background timeseries classes or something

Copy link
Member Author

@alasdairwilson alasdairwilson Sep 16, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Even within the same data series they are inconsistent with file names/column names/metadata so while it might be an idea, it doesn't make this kind of thing any tidier.

As an aside I have a scraper for 1m data (plus reader for pre 2009 1m data) but its ugly (secondary client rather than using attrs to select series) so haven't tried to put it in sunpy yet.

Co-authored-by: Nabil Freij <nabil.freij@gmail.com>
Copy link
Member

@dstansby dstansby left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks 👍 - I don't suppose there are some small files we can add to our test suite to make sure this doesn't break in the future?

@alasdairwilson
Copy link
Member Author

Looks 👍 - I don't suppose there are some small files we can add to our test suite to make sure this doesn't break in the future?

So I made a test file from a 1m avg goes file and added it to the test suite. I am happy with the PR now as long as you lot are.

@nabobalis nabobalis merged commit d08c48c into sunpy:main Sep 22, 2022
@lumberbot-app
Copy link

lumberbot-app bot commented Sep 22, 2022

Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

  1. Checkout backport branch and update it.
git checkout 4.0
git pull
  1. Cherry pick the first parent branch of the this PR on top of the older branch:
git cherry-pick -x -m1 d08c48c32e6ac774c6d7b32242041bd55c6eec9c
  1. You will likely have some merge/cherry-pick conflict here, fix them and commit:
git commit -am 'Backport PR #6410: Fixed XRSTimeSeries quality flags from breaking parsing of some XRS data.'
  1. Push to a named branch:
git push YOURFORK 4.0:auto-backport-of-pr-6410-on-4.0
  1. Create a PR against branch 4.0, I would have named this PR:

"Backport PR #6410 on branch 4.0 (Fixed XRSTimeSeries quality flags from breaking parsing of some XRS data.)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the Still Needs Manual Backport label once the PR gets merged.

If these instructions are inaccurate, feel free to suggest an improvement.

@lumberbot-app lumberbot-app bot added the Still Needs Manual Backport This PR needs manually backporting label Sep 22, 2022
@nabobalis nabobalis mentioned this pull request Oct 12, 2022
@nabobalis nabobalis removed the Still Needs Manual Backport This PR needs manually backporting label Oct 12, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
timeseries Affects the timeseries submodule
Projects
None yet
Development

Successfully merging this pull request may close these issues.

XRSTimeSeries broken for some netcdf files due to reading quality flags
4 participants