New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
obspy.iris: Use new FDSN web services and retire deprecated services #537
Conversation
just a rant: we should try to autogenerate our client by using the delivered WADL files (e.g. http://service.iris.edu/fdsnws/event/1/application.wadl ) to keep our clients up-to-date with IRIS - not dynamically on each client run, but once everytime we release a new version of ObsPy |
+1 for the WADL autogeneration if it is possible. Not sure though. For some query options they only specify the default value and not the possible options. Furthermore they do not seem to use Also they now appear to be using a QuakeML version > 1.0. No version number is included for some reason, but the origin depth unit appears to be meter. Judging from their documentation is appears to be QuakeML 1.2RC3. |
Before committing to using IRIS's WADL, you should know that the WADLs are in flux. More specifically, IRIS would love to hear recommendations for improving our WADLs and making them more useful. If we perfect the WADL that is used to describe our existing FDSN services, then hopefully future FDSN-based services will also provide better WADLs. Each service implementation may vary from each other in some minor way: some query parameters might not be relevant, or some may be augmented. As a specific example, here the IRIS FDSN event service differs from the spec. in that IRIS offers a format parameter with two options, text and xml. However, the FDSN spec says nothing about format. Another example might be if a data center doesn't store a particular type of metadata. Then the parameter used to filter upon that data will likely not be implemented. Hopefully a good WADL could be used to determine that ahead of time. If you have any questions, feel free to contact us, as we would like to help make it easier for ObsPy to interface with the web services. Now that there are FDSN standardized versions of our core services, ObsPy should be able to nearly automatically / seamlessly interface with the equivalent services at other data centers. Thanks, |
Hi Celso, what is the best way to contact you? It is ok, if we discuss it here? We are of course interested in being able to automatically access data from centers implementing the FDSN specification. Also if anyone at IRIS is interested in changing parts of obspy.iris feel free to do it and send us a pull request. I am no expert in WADL and do not know how feasible my suggestions are but they should be valid WADL. From our point of view the following changes would facilitate the autogeneration of code based on your WADLs. It is just a short snippets based on your event WADL. <request>
<!--Document the different fields, just done for one example here-->
<param name="starttime" style="query" type="xs:date">
<doc xml:lang="english" title="limit to events occurring on or after the specified start time.">
Example: starttime=2012-11-29T00:00:00
</doc>
</param>
<param name="endtime" style="query" type="xs:date"/>
<!--Indicate the used default values. Only done for some examples here-->
<param name="minlatitude" style="query" type="xs:double" default="-90.0"/>
<param name="maxlatitude" style="query" type="xs:double" default="90.0"/>
<param name="minlongitude" style="query" type="xs:double" default="-180.0"/>
<param name="maxlongitude" style="query" type="xs:double" default="180.0"/>
<param name="minmagnitude" style="query" type="xs:double"/>
<param name="maxmagnitude" style="query" type="xs:double"/>
<param name="magtype" style="query" type="xs:string"/>
<param name="catalog" style="query" type="xs:string"/>
<param name="contributor" style="query" type="xs:string"/>
<param name="maxdepth" style="query" type="xs:double"/>
<param name="mindepth" style="query" type="xs:double"/>
<param name="latitude" style="query" type="xs:double"/>
<param name="longitude" style="query" type="xs:double"/>
<param name="maxradius" style="query" type="xs:double"/>
<param name="minradius" style="query" type="xs:double"/>
<param name="orderby" style="query" type="xs:string" default="time"/>
<param name="updatedafter" style="query" type="xs:date"/>
<param name="eventid" style="query" type="xs:long"/>
<param name="originid" style="query" type="xs:long"/>
<!--Use booleans variables for flags-->
<param name="includearrivals" style="query" type="xs:boolean" default="false"/>
<param name="includeallmagnitudes" style="query" type="xs:boolean" default="false"/>
<param name="includeallorigins" style="query" type="xs:boolean" default="false"/>
<param name="limit" style="query" type="xs:int" default="1"/>
<param name="offset" style="query" type="xs:int" default="1"/>
<param name="output" style="query" type="xs:string" default="xml"/>
<!--Indicate the available options. Just done for one example here-->
<param name="format" style="query" default="xml">
<option value="xml"/>
<option value="text"/>
</param>
</request> Best wishes, Lion |
Here's an example of a beefed up event wadl. It doesn't match the wadl we have on line yet, but we're working on getting everything synced. Nothing's written in stone yet (it can and probably will change), but how's this for wadl documentation of the FDSN event service? <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<application xmlns="http://wadl.dev.java.net/2009/02">
<doc title="FDSN event web service 1.0"/>
<resources base="http://service.iris.edu/fdsnws/event/1/">
<resource path="/">
<method id="index" name="GET">
<response>
<representation mediaType="text/plain"/>
<representation mediaType="application/xml"/>
</response>
</method>
<resource path="/counts">
<method id="counts" name="GET">
<request>
<param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="starttime" style="query" type="xs:date">
<doc xml:lang="english" title="limit to events occurring on or after the specified start time">
Examples: starttime=2012-11-29 or 2012-11-29T00:00:00 or 2012-11-29T00:00:00.000
</doc>
</param>
<param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="endtime" style="query" type="xs:date">
<doc xml:lang="english" title="limit to events occurring on or before the specified end time"/>
</param>
<param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="minlatitude" style="query" type="xs:double" default="-90.0">
<doc xml:lang="english" title="southern boundary"/>
</param>
<param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="maxlatitude" style="query" type="xs:double" default="90.0">
<doc xml:lang="english" title="northern boundary"/>
</param>
<param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="minlongitude" style="query" type="xs:double" default="-180.0">
<doc xml:lang="english" title="western boundary"/>
</param>
<param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="maxlongitude" style="query" type="xs:double" default="180.0">
<doc xml:lang="english" title="eastern boundary"/>
</param>
<param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="minmagnitude" style="query" type="xs:double">
<doc xml:lang="english" title="Limit to events with a magnitude larger than or equal to the specified minimum"/>
</param>
<param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="maxmagnitude" style="query" type="xs:double">
<doc xml:lang="english" title="Limit to events with a magnitude smaller than or equal to the specified maximum"/>
</param>
<param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="magtype" style="query" type="xs:string">
<doc xml:lang="english" title="type of Magnitude used to test minimum and maximum limits (case insensitive)">
Examples: Ml,Ms,mb,Mw"</doc>
</param>
<param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="catalog" style="query" type="xs:string" default="NEIC+PDE">
<doc xml:lang="english" title="Specify the catalog from which origins and magnitudes will be retrieved"/>
</param>
<param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="contributor" style="query" type="xs:string">
<doc xml:lang="english" title="Limit to events contributed by a specified contributor"/>
</param>
<param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="maxdepth" style="query" type="xs:double">
<doc xml:lang="english" title="Limit to events with depths equal to or less than the specified depth"/>
</param>
<param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="mindepth" style="query" type="xs:double">
<doc xml:lang="english" title="Limit to events with depths equal to or greater than the specified depth"/>
</param>
<param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="latitude" style="query" type="xs:double" default="0.0">
<doc xml:lang="english" title="Specify the central latitude point for a radial search"/>
</param>
<param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="longitude" style="query" type="xs:double" default="0.0">
<doc xml:lang="english" title="Specify the central longitude point for a radial search"/>
</param>
<param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="maxradius" style="query" type="xs:double" default="180.0">
<doc xml:lang="english" title="Specify maximum distance from the geographic point defined by latitude and longitude"/>
</param>
<param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="minradius" style="query" type="xs:double" default="0.0">
<doc xml:lang="english" title="Specify minimum distance from the geographic point defined by latitude and longitude"/>
</param>
<param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="updatedafter" style="query" type="xs:date">
<doc xml:lang="english" title="Limit to events updated after the specified time"/>
</param>
<!-- <param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="output" style="query" type="xs:string" default="xml"/> -->
<param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="format" style="query" type="xs:string" default="xml">
<doc xml:lang="english" title=""/>
<option value="xml" mediaType="application/xml"/>
<option value="text" mediaType="text/plain"/>
</param>
</request>
<response>
<representation mediaType="text/plain"/>
<representation mediaType="application/xml"/>
</response>
</method>
</resource>
<resource path="/query">
<method id="query" name="GET">
<request>
<param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="starttime" style="query" type="xs:date">
<doc xml:lang="english" title="limit to events occurring on or after the specified start time">
Examples: starttime=2012-11-29 or 2012-11-29T00:00:00 or 2012-11-29T00:00:00.000
</doc>
</param>
<param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="endtime" style="query" type="xs:date">
<doc xml:lang="english" title="limit to events occurring on or before the specified end time"/>
</param>
<param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="minlatitude" style="query" type="xs:double" default="-90.0">
<doc xml:lang="english" title="southern boundary"/>
</param>
<param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="maxlatitude" style="query" type="xs:double" default="90.0">
<doc xml:lang="english" title="northern boundary"/>
</param>
<param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="minlongitude" style="query" type="xs:double" default="-180.0">
<doc xml:lang="english" title="western boundary"/>
</param>
<param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="maxlongitude" style="query" type="xs:double" default="180.0">
<doc xml:lang="english" title="eastern boundary"/>
</param>
<param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="minmagnitude" style="query" type="xs:double">
<doc xml:lang="english" title="Limit to events with a magnitude larger than or equal to the specified minimum"/>
</param>
<param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="maxmagnitude" style="query" type="xs:double">
<doc xml:lang="english" title="Limit to events with a magnitude smaller than or equal to the specified maximum"/>
</param>
<param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="magtype" style="query" type="xs:string">
<doc xml:lang="english" title="type of Magnitude used to test minimum and maximum limits (case insensitive)">
Examples: Ml,Ms,mb,Mw"</doc>
</param>
<param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="catalog" style="query" type="xs:string" default="NEIC+PDE">
<doc xml:lang="english" title="Specify the catalog from which origins and magnitudes will be retrieved"/>
</param>
<param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="contributor" style="query" type="xs:string">
<doc xml:lang="english" title="Limit to events contributed by a specified contributor"/>
</param>
<param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="maxdepth" style="query" type="xs:double">
<doc xml:lang="english" title="Limit to events with depths equal to or less than the specified depth"/>
</param>
<param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="mindepth" style="query" type="xs:double">
<doc xml:lang="english" title="Limit to events with depths equal to or greater than the specified depth"/>
</param>
<param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="latitude" style="query" type="xs:double" default="0.0">
<doc xml:lang="english" title="Specify the central latitude point for a radial search"/>
</param>
<param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="longitude" style="query" type="xs:double" default="0.0">
<doc xml:lang="english" title="Specify the central longitude point for a radial search"/>
</param>
<param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="maxradius" style="query" type="xs:double" default="180.0">
<doc xml:lang="english" title="Specify maximum distance from the geographic point defined by latitude and longitude"/>
</param>
<param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="minradius" style="query" type="xs:double" default="0.0">
<doc xml:lang="english" title="Specify minimum distance from the geographic point defined by latitude and longitude"/>
</param>
<param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="orderby" style="query" type="xs:string" default="time">
<doc xml:lang="english" title="Specify the ordering of the returned results"/>
<option value="time">
<doc xml:lang="english" title="Sort by time, descending"/>
</option>
<option value="magnitude">
<doc xml:lang="english" title="Sort by magnitude, descending"/>
</option>
<option value="time-asc">
<doc xml:lang="english" title="Sort by time, ascending"/>
</option>
<option value="magnitude-asc">
<doc xml:lang="english" title="Sort by magnitude, ascending"/>
</option>
</param>
<param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="updatedafter" style="query" type="xs:date">
<doc xml:lang="english" title="Limit to events updated after the specified time"/>
</param>
<param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="eventid" style="query" type="xs:long">
<doc xml:lang="english" title="Retrieve an event based on the unique ID numbers assigned by the IRIS DMC"/>
</param>
<param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="originid" style="query" type="xs:long">
<doc xml:lang="english" title="Retrieve an event based on the unique origin ID numbers assigned by the IRIS DMC"/>
</param>
<param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="includearrivals" style="query" type="xs:boolean" default="false">
<doc xml:lang="english" title="Specify if phase arrivals should be included"/>
</param>
<param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="includeallmagnitudes" style="query" type="xs:sboolean" default="false">
<doc xml:lang="english" title="Retrieve all magnitudes for the event, or only the primary magnitude from within the catalog"/>
</param>
<param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="includeallorigins" style="query" type="xs:boolean" default="false">
<doc xml:lang="english" title="Retrieve all origins or only the primary origin associated with each event in a catalog"/>
</param>
<param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="limit" style="query" type="xs:int">
<doc xml:lang="english" title="Limit the results to the specified number of events"/>
</param>
<param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="offset" style="query" type="xs:int" default="1">
<doc xml:lang="english" title="Return results starting at the event count specified"/>
</param>
<!-- <param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="output" style="query" type="xs:string" default="xml"/> -->
<param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="format" style="query" type="xs:string" default="xml">
<doc xml:lang="english" title="Specify output format. This is an IRIS extension to the FDSN specification"/>
<option value="xml" mediaType="application/xml"/>
<option value="text" mediaType="text/plain"/>
</param>
<param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="nodata" style="query" type="xs:int" default="204">
<doc xml:lang="english" title="Specify which HTML Status code is returned when no data is found. This is an IRIS extension to the FDSN specification"/>
<option value="204"/>
<option value="404"/>
</param>
</request>
<response>
<representation mediaType="text/plain"/>
<representation mediaType="application/xml"/>
</response>
</method>
</resource>
<resource path="/version">
<method id="version" name="GET">
<response>
<representation mediaType="text/plain"/>
</response>
</method>
</resource>
<resource path="/catalogs">
<method id="catalogs" name="GET">
<response>
<representation mediaType="application/xml"/>
</response>
</method>
</resource>
<resource path="/contributors">
<method id="contributors" name="GET">
<response>
<representation mediaType="application/xml"/>
</response>
</method>
</resource>
</resource>
</resources>
</application> |
Hi Celso, this look very good in my opinion and should enable us to query the webservice without specifying every option by hand. We might encounter more subtle problems as soon as we start working on an implementation. @barsch: Right now I would actually prefer to create a generic FDSN webservice that is able to query all datacenters providing one. The corresponding WADL can be downloaded on-the-fly during client initialization. @celso-iris: I am currently working on StationXML support for ObsPy and noticed that the returned StationXML files do not validate against the schema. This should require only some very minor changes. Examples:
The QuakeML files are fine :) |
@krischer: obspy.fdsn is definitively the way to go - I wanted to propose that anyway ;) |
+1 Orfeus is AFAIK currently also working on an implementation. |
@krischer: did you validate requested QuakeML files against the official schemas/relaxNG? So far, those accepted a lot that was not conform with the UML/pdf documentation, from my experience... |
Was thinking about starting work on this at some point. Just to make sure, this is supposed to go into a new module obspy.fdsn and the client should basically take an url at initialization which will default to the IRIS ws for now. Did I get this right? |
@megies : That sounds correct to me. http://comcat.cr.usgs.gov/fdsnws/event/1/ and RESIF has data and metadata services up and running: These might be a good places to try alternate URLs. The USGS happens to have many extensions to the FDSN spec. I don't know how you might handle it. In the IRIS-WS java library, we added an "add" routine to the EventCriteria object that takes an arbitrary string pair, and adds the results to the query url. |
I would suggest that we directly parse the WADL files upon client instantiation. We could also provide some kind of mapping for known services. Something like this might work: WADL_MAPPINGS = {
"IRIS": "url to iris wadl",
"USGS": "url to usgs wadl",
"RESIF": "url to resif wadl",
...
}
class Client(object):
def __init__(self, WADL):
if WADL in WADL_MAPPINGS:
self._parse_wadl(WADL_MAPPINGS[WADL])
else:
self._parse_wadl(WADL)
# Usage
from obspy.fdsn import Client
c = Client("IRIS")
... |
I thought we consented to not using WADL in the initial implementation? edit: talked with @krischer, should probably parse the wadls right from the start.. |
Now maps the short names of the parameters to their proper long names when necessary. Furthermore a dynamic help function for the different services is now available.
Added a sensible exception in case no services are being discovered at the chosen endpoint.
be removed again, when IRIS updates the event WADL (and after updating data/event.wadl)
tests were running fine when executing the corresponding test. tests were only failing with `obspy-runtests fdsn` because of prior warnings.
I'm OK with merging this. Open points are tracked in #646. |
@celso-iris I found a small bug: The event HTML page does not describe the |
Now only print derivations from the standard. Also some minor other changes.
ObsPy now has support for the FDSN webservices.
The IRIS DMC has implemented the International Federation of Digital Seismograph Networks (FDSN) versions of their core web services that support access to time series data, related metadata and event parameters. The equivalent DMC services are now deprecated and will be retired, read more on that at http://www.iris.washington.edu/pipermail/webservices/2013-March/000378.html.
The FDSN web services are available from a dedicated host name with new web documentation: http://service.iris.edu/
Deprecation of equivalent IRIS services
With the release of the FDSN services they are deprecating the equivalent services.
These services will be turned off in approximately 6 months:
Differences between FDSN and the services they are replacing
The FDSN web services return HTTP status code 204 to indicate "No data" as opposed to status code 404 used by the services they are replacing. The behavior of returning a 404 can be restored by including the 'nodata=404' parameter and value in the request.
fdsnws-dataselect is a combination of the capabilities of ws-bulkdataselect and ws-dataselect:
fdsnws-station is equivalent to ws-station with these important changes:
The FDSN StationXML schema (http://www.fdsn.org/xml/station/) is generally equivalent to the XML format previously used but it is a distinctly different schema that cannot be parsed with the same software parsing the XML from ws-station.
fdsnws-event is equivalent to ws-event with these differences: