Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for extra NOAA JSON timeseries #4346

Open
nabobalis opened this issue Jul 11, 2020 · 5 comments
Open

Add support for extra NOAA JSON timeseries #4346

nabobalis opened this issue Jul 11, 2020 · 5 comments
Labels
Effort Medium Requires a moderate time investment. Feature Request New feature wanted. Package Novice Requires little technical knowledge of the package. Priority Low Rapid action not required. timeseries Affects the timeseries submodule

Comments

@nabobalis
Copy link
Contributor

From #4340 (comment):

I wonder if it would be an interesting feature/extension for the nooa TimeSeries class, to can handle also other json files (with t.s.) from https://services.swpc.noaa.gov/json/. For example https://services.swpc.noaa.gov/json/goes/ (/primary/xrays-1-day.json) contain json timeseries for soft x-rays provided in real time (contrary to the goes.py class where the fits files are available after 1-2 days). I think this may be something interesting for the sunpy user to have. I attach a simple example of how I process and plot this xrays-1-day.json file, similarly it works for for the other xrays-?-day.json files. Something similar could be tailored to the noaa class, right?
GOES_JSON_FLARE.zip

@nabobalis nabobalis added Feature Request New feature wanted. Package Novice Requires little technical knowledge of the package. Priority Low Rapid action not required. Effort Medium Requires a moderate time investment. timeseries Affects the timeseries submodule labels Jul 11, 2020
@AthKouloumvakos
Copy link

@nabobalis thank you for opening this issue. I was trying to see some details for the products provided and information detailed by the https://services.swpc.noaa.gov/json/. So for the GOES json files that are provided under the /primary/ or /secondary/ sub-directories, the TimeSeries class could probably handle the following products:

  1. GOES json files (main-url: https://services.swpc.noaa.gov/json/goes/):

    • GOES X-ray Flux (more info at: https://www.swpc.noaa.gov/products/goes-x-ray-flux)
      X-ray Flux, 1-minute averaged: xrays-1-day.json, xrays-3-day.json, xrays-6-hour.json, xrays-7-day.json (note: the files are for different time spans, i.e. now-?days/hours)

    • GOES Proton Flux (more info at: https://www.swpc.noaa.gov/products/goes-proton-flux)
      Integral-proton flux, 5-minutes averaged: integral-protons-1-day.json, integral-protons-3-day.json, integral-protons-6-hour.json, integral-protons-7-day.json (and integral-protons-plot-1-day.json, integral-protons-plot-3-day.json, integral-protons-plot-6-hour.json, integral-protons-plot-7-day.json). The difference between the integral-protons-* and integral-protons-plot-* json files is the number of proton channels contained in each case. The integral-protons-plot-* files have the >=10 MeV, >=50 MeV, >=100 MeV, >=500 MeV integrated proton flux as issued by noaa so far, in the integral-protons-* files the >=1 MeV, >=5 MeV, >=30 MeV, >=50 MeV, >=60 MeV have been also added.

      Differential-proton flux, 5-minutes averaged: differential-protons-1-day.json, differential-protons-3-day.json, differential-protons-6-hour.json, differential-protons-7-day.json

      Integral-proton fluence, 5-minutes averaged: integral-proton-fluence-7-day.json

    • GOES Electron Flux (https://www.swpc.noaa.gov/products/goes-electron-flux)
      Integral-electron flux, 5-minutes averaged: integral-electrons-1-day.json, integral-electrons-3-day.json, integral-electrons-6-hour.json, integral-electrons-7-day.json

      Differential-electron flux, 5-minutes averaged: differential-electrons-1-day.json, differential-electrons-3-day.json, differential-electrons-6-hour.json, differential-electrons-7-day.json

    • GOES Magnetometer data (more info at: https://www.swpc.noaa.gov/products/goes-magnetometer)
      Components and Total magnetic field, 1-minute averaged: magnetometers-1-day.json, magnetometers-3-day.json, magnetometers-6-hour.json, magnetometers-7-day.json.

For the GOES X-ray Flux case I am currently working on an e.g. NOAAGoesSXRTimeSeries class that can handle the sxr .json files and it is based on the revised noaa TimeSeries class #4339. I will conclude my efforts next week, I would like to do some reading first of the sunpy Developer’s Guide at especially the pull request part since it is going to be my first one.

@wtbarnes
Copy link
Member

wtbarnes commented Nov 4, 2020

Given the comments on #4359, I think this should possibly be closed? The conclusion there was, at least for the XRS data, these JSON files are not the canonical data product and it would be contradictory and confusing to have multiple data sources for a single instrument. I do not know if this applies to the other data products mentioned here as well.

@hayesla
Copy link
Member

hayesla commented Nov 5, 2020

Yes I agree, it doesn't need to be supported.

However, it could be worth it to add an example of how to do a quick-look at the GOES 1 minute data? Nice example of how to create TimeSeries and may be probably useful for space-weathery people (i.e. looking at NRT data).

e.g.

import pandas as pd 
from sunpy.time import parse_time
from sunpy import timeseries as ts 
from collections import OrderedDict
from astropy import units as u 

data = pd.read_json("https://services.swpc.noaa.gov/json/goes/primary/xrays-7-day.json")

data_short = data[data["energy"]=="0.05-0.4nm"]
data_long = data[data["energy"]=="0.1-0.8nm"]
time_array = parse_time(data_short["time_tag"])

units = OrderedDict([("xrsa", u.W/u.m**2), ("xrsb", u.W/u.m**2)])
meta = OrderedDict({"instrument":"GOES X-ray sensor", "measurments": "primary", "type": "quicklook"}) # or whatever

goes_data = pd.DataFrame({"xrsa": data_short["flux"].values, "xrsb": data_long["flux"].values}, index=time_array.datetime)
test_ts = ts.TimeSeries(goes_data, meta, units, source="xrs")
test_ts.peek()

Figure_1

@m11dedhia
Copy link
Contributor

Are only examples left to be added or the implementation for support of NOAA JSON timeseries as well?

@nabobalis
Copy link
Contributor Author

I guess just an example using the code that Laura posted above.

@dstansby dstansby removed the Hacktoberfest Issues that could be the focus of GSoC or Hacktoberfests label Apr 16, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Effort Medium Requires a moderate time investment. Feature Request New feature wanted. Package Novice Requires little technical knowledge of the package. Priority Low Rapid action not required. timeseries Affects the timeseries submodule
Projects
None yet
Development

No branches or pull requests

7 participants