New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

https FDSN web service end points not working #1874

Closed
megies opened this Issue Sep 2, 2017 · 15 comments

Comments

Projects
None yet
5 participants
@megies
Member

megies commented Sep 2, 2017

When checking the InsecureRequestWarning in #1779, I noticed that those FDSNWS end points we have that go to https addresses don't seem to work..

In [1]: from obspy.clients.fdsn import Client

In [2]: client = Client('GFZ')
---------------------------------------------------------------------------
FDSNException                             Traceback (most recent call last)
<ipython-input-2-1d7f32e7dd73> in <module>()
----> 1 client = Client('GFZ')

/home/megies/git/obspy-master/obspy/clients/fdsn/client.pyc in __init__(self, base_url, major_versions, user, password, user_agent, debug, timeout, service_mappings, force_redirect)
    260             print("Request Headers: %s" % str(self.request_headers))
    261 
--> 262         self._discover_services()
    263 
    264     def get_events(self, starttime=None, endtime=None, minlatitude=None,

/home/megies/git/obspy-master/obspy/clients/fdsn/client.pyc in _discover_services(self)
   1479                    "be due to a temporary service outage or an invalid FDSN "
   1480                    "service address." % self.base_url)
-> 1481             raise FDSNException(msg)
   1482 
   1483         # Cache.

FDSNException: No FDSN services could be discovered at 'https://geofon.gfz-potsdam.de'. This could be due to a temporary service outage or an invalid FDSN service address.

In [3]: client = Client('SCEDC')
---------------------------------------------------------------------------
FDSNException                             Traceback (most recent call last)
<ipython-input-3-7c19ea139f48> in <module>()
----> 1 client = Client('SCEDC')

/home/megies/git/obspy-master/obspy/clients/fdsn/client.pyc in __init__(self, base_url, major_versions, user, password, user_agent, debug, timeout, service_mappings, force_redirect)
    260             print("Request Headers: %s" % str(self.request_headers))
    261 
--> 262         self._discover_services()
    263 
    264     def get_events(self, starttime=None, endtime=None, minlatitude=None,

/home/megies/git/obspy-master/obspy/clients/fdsn/client.pyc in _discover_services(self)
   1479                    "be due to a temporary service outage or an invalid FDSN "
   1480                    "service address." % self.base_url)
-> 1481             raise FDSNException(msg)
   1482 
   1483         # Cache.

FDSNException: No FDSN services could be discovered at 'https://service.scedc.caltech.edu'. This could be due to a temporary service outage or an invalid FDSN service address.

In [4]: client = Client('USGS')
---------------------------------------------------------------------------
FDSNException                             Traceback (most recent call last)
<ipython-input-4-2d8335176e68> in <module>()
----> 1 client = Client('USGS')

/home/megies/git/obspy-master/obspy/clients/fdsn/client.pyc in __init__(self, base_url, major_versions, user, password, user_agent, debug, timeout, service_mappings, force_redirect)
    260             print("Request Headers: %s" % str(self.request_headers))
    261 
--> 262         self._discover_services()
    263 
    264     def get_events(self, starttime=None, endtime=None, minlatitude=None,

/home/megies/git/obspy-master/obspy/clients/fdsn/client.pyc in _discover_services(self)
   1479                    "be due to a temporary service outage or an invalid FDSN "
   1480                    "service address." % self.base_url)
-> 1481             raise FDSNException(msg)
   1482 
   1483         # Cache.

FDSNException: No FDSN services could be discovered at 'https://earthquake.usgs.gov'. This could be due to a temporary service outage or an invalid FDSN service address.

@megies megies added this to the 1.1.0 milestone Sep 2, 2017

@megies

This comment has been minimized.

Member

megies commented Sep 2, 2017

@krischer

This comment has been minimized.

Member

krischer commented Sep 20, 2017

Is this still a problem? I just tried and I can access all HTTPS services just fine.

@megies

This comment has been minimized.

Member

megies commented Sep 20, 2017

Hmm.. strange, can't reproduce right now. So I guess it works now..? But really strange because hard to believe it was simultaneously fixed on three servers??

@krischer

This comment has been minimized.

Member

krischer commented Sep 20, 2017

yea - maybe it was some kind of certificate issue? next time you encounter this: can you run with debug=True?

Maybe we should also revert all default services to use HTTP? Lots of Python installations have outdated crypto libraries and https will not work I guess. But I'm no expert in these manners. Opinions? @barsch @QuLogic @markcwill

@markcwill

This comment has been minimized.

Contributor

markcwill commented Sep 20, 2017

@krischer @megies in my experience it's usually a cert issue. We only see crypto issues on really old libs like embedded stuff in routers & radios, or unless someone's trying to do something fancy like restrict what algos can be used. It could have been something weird like they all expired at the same time, but unlikely, since it looks like the 3 services in this thread (USGS, Caltech, etc) are signed by different root authorities.

@megies, related to #1779, my quick-and-dirty test is to do a handshake, e.g.

$ openssl s_client -connect "earthquake.usgs.gov:443" -showcerts -servername "earthquake.usgs.gov"

That one looks to be good for me right now. To contrast, it looks as though the service at IRIS is not set up properly, e.g. maybe not sending the full chain of certs (can't verify with intermediate certs), so the handshake is failing for me right now.

$ openssl s_client -connect "service.iris.edu:443" -showcerts -servername "service.iris.edu"

I think I agree with @krischer there should be a policy about this, either default to http or use https only if signed off by the datacenter and/or the http endpoint doesn't really exist (like if 80 automatically redirects to 443).

@krischer

This comment has been minimized.

Member

krischer commented Sep 20, 2017

That one looks to be good for me right now. To contrast, it looks as though the service at IRIS is not set up properly, e.g. maybe not sending the full chain of certs (can't verify with intermediate certs), so the handshake is failing for me right now.

$ openssl s_client -connect "service.iris.edu:443" -showcerts -servername "service

@chad-iris

@krischer

This comment has been minimized.

Member

krischer commented Sep 20, 2017

I think I agree with @krischer there should be a policy about this, either default to http or use https only if signed off by the datacenter and/or the http endpoint doesn't really exist (like if 80 automatically redirects to 443).

Right now I would tend to default to HTTP. Another person also had problems connecting to the HTTPS services using ObsPy from azure cloud. Also in most cases the data is not sensitive and people could still choose to manually select HTTPS.

@chad-iris

This comment has been minimized.

Contributor

chad-iris commented Sep 20, 2017

That one looks to be good for me right now. To contrast, it looks as though the service at IRIS is not set up properly, e.g. maybe not sending the full chain of certs (can't verify with intermediate certs), so the handshake is failing for me right now.
$ openssl s_client -connect "service.iris.edu:443" -showcerts -servername "service.iris.edu"

I've asked our admin to investigate. Looks like that certificate is about to expire anyway so we'll probably just get a new one.

@chad-iris

This comment has been minimized.

Contributor

chad-iris commented Sep 26, 2017

That one looks to be good for me right now. To contrast, it looks as though the service at IRIS is not set up properly, e.g. maybe not sending the full chain of certs (can't verify with intermediate certs), so the handshake is failing for me right now.
$ openssl s_client -connect "service.iris.edu:443" -showcerts -servername "service.iris.edu"

The certs for *.iris.edu are now updated and should be up to modern snuff. Thanks for the nudge.

@megies

This comment has been minimized.

Member

megies commented Sep 27, 2017

Well, I guess we can close this. If there's problems in the future, users can always use the explicit URL to the http endpoint anyway.

@megies megies closed this Sep 27, 2017

@megies megies added the external label Sep 27, 2017

@krischer

This comment has been minimized.

Member

krischer commented Sep 28, 2017

I'm reopening this as a reminder to change back to HTTP. I think we want this as just too many python installations have outdated crypto libraries/certificates and I don't see a big advantage for using HTTPS for waveform data.

Using HTTP should also not be an issue with authenticated queries as HTTP digest auth does encrypt the credentials.

@krischer krischer reopened this Sep 28, 2017

@Jollyfant

This comment has been minimized.

Contributor

Jollyfant commented Sep 28, 2017

Technically, FDSN specifications impose that services are available on :80. Defaulting to HTTP seems the best choice for now IMHO.

@megies

This comment has been minimized.

Member

megies commented Sep 28, 2017

How about having all short URL mappings go to HTTP at port 80 and optionally we could add a kwarg ensure_https=False that raises on any attempted connection to HTTP? That could help people paranoid about their credentials to ensure they stay super safe? (But I guess thats just my paranoia coming through and most people wont care, most likely)

In any case, I'm not the HTTP/HTTPS security expert, and if HTTP digest is secure enough then this additional check is probably not necessary.

krischer added a commit to krischer/obspy that referenced this issue Sep 29, 2017

Default the fdsnws mappings to HTTP.
Main reasons are (see obspy#1874):

* The FDSN specifications impose that services are on :80.
* The crypto libraries and certificates in many Python installations are
  outdated making it fail in non-obvious ways to users.
@krischer

This comment has been minimized.

Member

krischer commented Sep 29, 2017

Done in #1908.

Feel free to add the ensure_https kwargs, but if someone is paranoid, they could just explicitly specify the https url of the web service.

@megies megies closed this in #1908 Sep 29, 2017

@megies

This comment has been minimized.

Member

megies commented Sep 29, 2017

if someone is paranoid, they could just explicitly specify the https url of the web service.

Yeah fair enough, no need to add more maintenance burden.

nackerley pushed a commit to nackerley/obspy that referenced this issue May 8, 2018

Default the fdsnws mappings to HTTP.
Main reasons are (see obspy#1874):

* The FDSN specifications impose that services are on :80.
* The crypto libraries and certificates in many Python installations are
  outdated making it fail in non-obvious ways to users.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment