Add module for parsing NOAA GOES data

[rhroberts]

Finally getting around to finishing this :) I picked up where Paul left off since he is no longer with TWDB.

• usgs.eddn module has been removed
• noaa.goes module and associated unit tests have been added to replace it

#173 can be closed in favor of this PR. Thanks!

[emiliom]

Great, thank you @rhroberts ! I'll get to it within a few days. I've been meaning to issue a new release, so this will be a good motivator.

[rhroberts]

Thanks, @emiliom! Much appreciated!

[emiliom]

Note: Probably should add from . import noaa to ulmo/__init__.py. I'll look closer.

[emiliom]

@rhroberts sorry for taking so long to review this PR. I'm finally working on it.

I tested your PR locally by running the goes tests. All but one of the tests succeeded. The one that fails is test_parse_dcp_message_timestamp. The final error message is:

urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='dcs1.noaa.gov', port=443): 
  Max retries exceeded with url: 
  /Account/FieldTestData?addr=C5149430&hours=12 
  (Caused by SSLError(SSLError("bad handshake: SysCallError(32, 'EPIPE')",),))

and the stack trace looks something like this:

test/noaa_goes_test.py:32: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
ulmo/noaa/goes/core.py:112: in get_data
    messages = _fetch_url(params)
ulmo/noaa/goes/core.py:137: in _fetch_url
    r = requests.post(dcs_url, params=params, timeout=60)
/home/mayorga/.local/lib/python3.6/site-packages/requests/api.py:119: in post
    return request('post', url, data=data, json=json, **kwargs)
/home/mayorga/.local/lib/python3.6/site-packages/requests/api.py:61: in request
    return session.request(method=method, url=url, **kwargs)
/home/mayorga/.local/lib/python3.6/site-packages/requests/sessions.py:530: in request
    resp = self.send(prep, **send_kwargs)
/home/mayorga/.local/lib/python3.6/site-packages/requests/sessions.py:643: in send
    r = adapter.send(request, **kwargs)

Examining noaa_goes_test.py, it looks like test_parse_dcp_message_timestamp would try to loop through the 3 list elements in message_test_sets, failing on the first one. I tried to see if the other two might work by commenting out first the first element, then both the first and second elements, but the same error type occurred.

Are you able to run that test successfully?

Note: As you can see this was run with the Python 3.6 conda environment created by conda_environment.yml. I've been reworking the environment creation workflow to, among other things, remove dependencies no longer used and remove old version pinning that were causing the Python version to be locked at 3.6. Let me know what Python version you used when running your tests.

[rhroberts]

Thanks for checking it out @emiliom! I seem to recall running into this test failure awhile back, but I wasn't able to replicate it when I tested today using the conda_environment.yml with Python 3.6.13. I'll dig into it more on Monday though.

Output of pip list for reference:

Package                       Version             Location
----------------------------- ------------------- ----------------------
alabaster                     0.7.12
appdirs                       1.4.4
attrs                         20.3.0
Babel                         2.9.0
beautifulsoup4                4.9.3
brotlipy                      0.7.0
certifi                       2020.12.5
cffi                          1.14.5
chardet                       4.0.0
colorama                      0.4.4
coverage                      5.5
coveralls                     3.0.1
cryptography                  3.4.6
docopt                        0.6.2
docutils                      0.16
flake8                        3.9.0
freezegun                     1.1.0
future                        0.18.2
geojson                       2.5.0
html5lib                      0.9999999
httpretty                     0.8.10
idna                          2.10
imagesize                     1.2.0
importlib-metadata            3.7.3
iniconfig                     1.1.1
isodate                       0.6.0
Jinja2                        2.11.3
lxml                          4.6.2
MarkupSafe                    1.1.1
mccabe                        0.6.1
mock                          4.0.3
more-itertools                8.7.0
numexpr                       2.7.3
numpy                         1.19.5
numpydoc                      1.1.0
packaging                     20.9
pandas                        1.0.5
pip                           21.0.1
pluggy                        0.13.1
py                            1.10.0
pycodestyle                   2.6.0
pycparser                     2.20
pyflakes                      2.2.0
Pygments                      2.8.1
pyOpenSSL                     20.0.1
pyparsing                     2.4.7
PySocks                       1.7.1
pytest                        6.2.2
pytest-cov                    2.11.1
pytest-runner                 5.3.0
python-dateutil               2.8.1
pytz                          2021.1
PyYAML                        5.4.1
requests                      2.25.1
setuptools                    49.6.0.post20210108
six                           1.15.0
snowballstemmer               2.1.0
soupsieve                     2.0.1
Sphinx                        3.5.2
sphinxcontrib-applehelp       1.0.2
sphinxcontrib-devhelp         1.0.2
sphinxcontrib-htmlhelp        1.0.3
sphinxcontrib-jsmath          1.0.1
sphinxcontrib-qthelp          1.0.3
sphinxcontrib-serializinghtml 1.1.4
suds-jurko                    0.6
tables                        3.6.1
toml                          0.10.2
typing-extensions             3.7.4.3
ulmo                          0.8.7.dev0          /home/ubuntu/Code/ulmo
urllib3                       1.26.4
webencodings                  0.5.1
wheel                         0.36.2
zipp                          3.4.1

[rhroberts]

Hey @emiliom, I'm still not able to replicate this test failure on my machine. Based on this issue, it looks like it might be an issue with the versions of HTTPretty and pyOpenSSL being used. The HTTPretty dependency is from the tox.ini, not from conda_environment.yml, so it might be that we have different versions installed when running the tests? Could you let me know what versions you are using? Thanks!

[emiliom]

Thanks. I'm going to try running the tests using the new dependency setup I've implemented, using requirements.txt and requirements-dev.txt, where I've removed the pinning of httpretty to a very old version. I'll report back. Regardless, I'm glad to hear you're able to successfully run the tests.

[emiliom]

Since the tests work for you, and the test set up for ulmo is already improved, I'll just merge your PR and re-run the goes tests after merging. I'll keep you posted via #178. Thanks again!

[rhroberts]

Awesome, thanks Emilio!