Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Allow read_surfrad to parse http files #1459

Merged
merged 11 commits into from
Jul 19, 2022
Merged

Allow read_surfrad to parse http files #1459

merged 11 commits into from
Jul 19, 2022

Conversation

AdamRJensen
Copy link
Member

@AdamRJensen AdamRJensen commented May 19, 2022 •
edited by kandersolar
Loading

  • I am familiar with the contributing guidelines
  • Tests added
  • [ ] Updates entries in docs/sphinx/source/reference for API changes.
  • Adds description and name entries in the appropriate "what's new" file in docs/sphinx/source/whatsnew for all changes. Includes link to the GitHub Issue with :issue:`num` or this Pull Request with :pull:`num`. Includes contributor name and/or GitHub username (link with :ghuser:`user`).
  • New code is fully documented. Includes numpydoc compliant docstrings, examples, and comments where necessary.
  • Pull request is nearly complete and ready for detailed review.
  • Maintainer: Appropriate GitHub Labels and Milestone are assigned to the Pull Request and linked Issue.

A StackOverflow user pointed out that the SUFRAD files are not only accessible from the FTP server but also via HTTPS links. This PR updates the read_surfrad function to allow for parsing remote HTTP files.

@AdamRJensen AdamRJensen added this to the 0.9.2 milestone May 19, 2022
@AdamRJensen
Copy link
Member Author

@kanderso-nrel Is a whatsnew entry needed?

@kandersolar
Copy link
Member

@kanderso-nrel Is a whatsnew entry needed?

I can imagine a user being interested in this change, so I'd vote yes to a whatsnew entry :)

@AdamRJensen
Merge branch 'master' into surfrad_http
fa8b9e4
@AdamRJensen
Squashed commit of the following:
b313c64 
commit 5047b26
Author: Prajwal Borkar <48290911+PrajwalBorkar@users.noreply.github.com>
Date:   Tue May 17 19:14:53 2022 +0530

    Updated get_cams protocol to https pvlib#1457 (pvlib#1458)

    * Updated get_cams protocol to https pvlib#1457

    * Updated instances of http to https. pvlib#1457

    * Updated documentation links to https

    * Added Contributor

commit a0812b1
Author: roger-lcc <58332996+roger-lcc@users.noreply.github.com>
Date:   Wed May 4 20:01:42 2022 +0800

    CI asv check (pvlib#1454)

    * CI asv check

    * added CI asv check

    * CI asv check

    * CI asv check

    * updated CI asv check

    * Update docs/sphinx/source/whatsnew/v0.9.2.rst

    updated v0.9.2.rst

    Co-authored-by: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com>

    Co-authored-by: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com>

commit 83e379a
Author: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com>
Date:   Thu Apr 28 19:26:09 2022 -0400

    Bump pandas to 0.25.0; test updates (pvlib#1448)

    * bump pandas min from 0.22.0 to 0.25.0

    * fix buggy test__check_pandas_assert_kwargs

    don't use monkeypatch and mocker in the same test function.  pytest-dev/pytest-mock#289

    * fix psm3 test (apparent_zenith -> solar_zenith)

    * whatsnew

    * better UTC conversion in sun_rise_set_transit_ephem

    * helpful comments

    * more whatsnew

    * '3.0' -> '3' in read_crn test?

    * apply dtypes during parsing in read_crn

    * move dropna() post-processing into read_fwf call

    * fix read_crn for pandas<1.2.0

    * Update pvlib/solarposition.py

    Co-authored-by: Will Holmgren <william.holmgren@gmail.com>

    * nix pytz

    * UTC -> utc

    * address simd arccos issue in tracking.singleaxis

    Co-authored-by: Will Holmgren <william.holmgren@gmail.com>

commit 8d0f863
Author: Naman Priyadarshi <77211855+Naman-Priyadarshi@users.noreply.github.com>
Date:   Tue Apr 12 22:55:58 2022 +0530

    Advance numba from 0.36.1 to 0.40.0 in asv py3.6 environment (pvlib#1440)

    * Advance numba from 0.36.1 to 0.40.0

    * Advance numba from 0.36.1 to 0.40.0

    * Updated whatsnew.rst

commit 5cb695d
Author: Naman Priyadarshi <77211855+Naman-Priyadarshi@users.noreply.github.com>
Date:   Wed Apr 6 23:58:03 2022 +0530

    Remove unnecessary **kwargs from spa_python and get_total_irradiance (pvlib#1437)

    * Update Solarposition.py

    Removed **kwargs from pvlib.solarposition.spa_python

    * Added v0.9.2.rst, changes in pvlib/irradiance.py and pvlib/location.py

    Made new v0.9.2.rst and removed **kwargs from pvlib/irradiance.py (Line 309) and pvlib/location.py (Line 234-235)

    * Update docs/sphinx/source/whatsnew/v0.9.2.rst

    * Update docs/sphinx/source/whatsnew/v0.9.2.rst

    Co-authored-by: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com>

commit 8460b36
Author: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com>
Date:   Tue Mar 29 15:31:25 2022 -0600

    Finalize 0.9.1 (pvlib#1431)

    * fix heading levels in user_guide/bifacial.rst

    * whatsnew cleanup

    * fix readme html

    missing tag, maybe unicode zero-width spaces?

    * readme: link to universal zenodo doi

    * readme: update installation link for pvlib#1173

    * whatsnew date

    * additional contributors

    * delete errant space

commit edbf2a6
Author: RoyCoding8 <92641125+RoyCoding8@users.noreply.github.com>
Date:   Wed Mar 30 01:58:18 2022 +0530

    Updated plot_singlediode.py (pvlib#1434)

    * Update plot_singlediode.py

    Changed the unit from C to degree C (°C)

    * Update plot_singlediode.py

    Changed to LaTeX \degree symbol in matplotlib which avoids any encoding issues with using Unicode characters.

    * Update v0.9.1.rst

    Added name to the contributors' list

    * Update v0.9.1.rst

commit cf4a8ad
Author: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com>
Date:   Tue Mar 29 14:04:40 2022 -0600

    Update sphinx to 4.5.0 (pvlib#1435)

    * bump sphinx and pydata-sphinx-theme versions

    * clean up sphinx conf.py

    * fix distutils strangeness, maybe

    * use freshly-released sphinx==4.5.0

commit 884a153
Author: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com>
Date:   Wed Mar 23 13:41:35 2022 -0600

    Clarify delta_t docstring descriptions (pvlib#1429)

    * clarify delta_t docstrings

    * whatsnew

commit c243183
Author: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com>
Date:   Thu Mar 17 12:01:57 2022 -0600

    Deprecate pvlib.forecast (pvlib#1426)

    * deprecate pvlib.forecast classes

    * catch warnings in tests

    * add warning admonition to forecasts.rst

    * whatsnew

    * stickler

    * pin pytest < 7.1.0

    * pin pytest in the right place this time

    * more warning suppression in tests

    * unpin pytest

    * Update docs/sphinx/source/whatsnew/v0.9.1.rst

    * copy warning to reference/forecasting.rst

commit e3baa12
Author: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com>
Date:   Thu Mar 17 11:28:56 2022 -0600

    Fix conditional dependency on dataclasses (pvlib#1422)

    * better conditional dependency on dataclasses

    * whatsnew

commit 27cba7a
Author: Naman Priyadarshi <77211855+Naman-Priyadarshi@users.noreply.github.com>
Date:   Thu Mar 17 22:48:08 2022 +0530

    Added asv benchmarking badge to the table of badges in the main README. (pvlib#1427)

    * Update Readme.md

    Added benchmarks asv badge to the badge section

    * Updated v.0.9.1.rst

    Added my name to the list of Contributers.

commit 1893b20
Author: Adam R. Jensen <39184289+AdamRJensen@users.noreply.github.com>
Date:   Mon Mar 14 18:37:58 2022 +0100

    Add variable mapping of psm3 (pvlib#1374)

    * Add variable mapping of psm3

    * Add enhancement entry in whatsnew

    * Fix stickler

    * Map keys in metadata dict

    * Remove double spaces in docs

    * Fix stickler

    * Doc update

    Co-authored-by: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com>

    * Reformatting - changes by kanderso-nrel

    * Update docstring table with 2020

    * Add deprecation warning test coverage

    * Rename to VARIABLE_MAP

    * Change apparent_zenith to solar_zenith

    Based on the decision in pvlib#1403

    * Update attributes docstring

    * Change elevation to altitude when mapping variables

    * Update psm3 variable mapping test

    Co-authored-by: Kevin Anderson <57452607+kanderso-nrel@users.noreply.github.com>
@AdamRJensen
Revert "Squashed commit of the following:"
3d63ac3 
This reverts commit b313c64.
@AdamRJensen
Merge remote-tracking branch 'upstream/master' into surfrad_http
ec55285
@AdamRJensen
Update whatsnew
5c01bc4
kandersolar
kandersolar approved these changes May 19, 2022
View reviewed changes
Copy link
Member

@kandersolar kandersolar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does NOAA document somewhere that the HTTP files are identical to the FTP files? The READMEs in the HTTP index only mention the FTP locations: https://gml.noaa.gov/aftp/data/radiation/surfrad/Penn_State_PA/README

I'm also curious why the HTTP index seems to have two directories for each station (e.g. both fpk and Fort_Peck_MT).

@AdamRJensen
Copy link
Member Author

Here's the response I got from Chris Cornwall from NOAA in regards to the reason for the SURFRAD files being available on both the FTP server and the new HTTP index:

HTTP access to our FTP data was set up a couple years ago because more and more web browsers ended support for the FTP protocol, in the interest of security. As you noticed, the same data are available on our aftp.cmdl.noaa.gov server as through the web link https://gml.noaa.gov/aftp/data/radiation/surfrad/
The reason for duplicate directories for the SURFRAD permanent locations is for backward compatibility to the old FTP data directory structure. The 3-letter site abbreviations are actually symbolic links on the server, pointing to their longer-named counterparts: dra -> Desert_Rock_NV, for example.

@AdamRJensen AdamRJensen requested a review from mikofski June 3, 2022 08:59
wholmgren
wholmgren approved these changes Jul 19, 2022
View reviewed changes
Copy link
Member

@wholmgren wholmgren left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

any reason this is still sitting here?

@AdamRJensen
Copy link
Member Author

any reason this is still sitting here?

I think it was just needing a second review. Feel free to merge.

@kandersolar kandersolar merged commit c3016f1 into pvlib:master Jul 19, 2022
@kandersolar
Copy link
Member

Thanks @AdamRJensen!

@AdamRJensen AdamRJensen mentioned this pull request Aug 5, 2022
Open
@AdamRJensen AdamRJensen deleted the surfrad_http branch October 23, 2024 17:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kandersolar kandersolar kandersolar approved these changes

@wholmgren wholmgren wholmgren approved these changes

@mikofski mikofski Awaiting requested review from mikofski

Assignees
No one assigned
Labels
enhancement io
Projects
None yet
Milestone
0.9.2
Development

Successfully merging this pull request may close these issues.
3 participants
@AdamRJensen @kandersolar @wholmgren