Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

obspy.iris: Use new FDSN web services and retire deprecated services #537

Merged
merged 58 commits into from Oct 4, 2013

Conversation

megies
Copy link
Member

@megies megies commented Jun 27, 2013

The IRIS DMC has implemented the International Federation of Digital Seismograph Networks (FDSN) versions of their core web services that support access to time series data, related metadata and event parameters. The equivalent DMC services are now deprecated and will be retired, read more on that at http://www.iris.washington.edu/pipermail/webservices/2013-March/000378.html.

The FDSN web services are available from a dedicated host name with new web documentation: http://service.iris.edu/

Deprecation of equivalent IRIS services

With the release of the FDSN services they are deprecating the equivalent services.

These services will be turned off in approximately 6 months:

Differences between FDSN and the services they are replacing

The FDSN web services return HTTP status code 204 to indicate "No data" as opposed to status code 404 used by the services they are replacing. The behavior of returning a 404 can be restored by including the 'nodata=404' parameter and value in the request.

fdsnws-dataselect is a combination of the capabilities of ws-bulkdataselect and ws-dataselect:

  1. enhancement: the network, station, location and channel selections may be comma-separated lists (e.g. BHE,BHN)
  2. enhancement: the network, station, location and channel selections may contain * and ? wildcards
  3. should be a drop-in replacement for ws-bulkdataselect (accepts the same requests via POST).
  4. a near complete replacement for ws-dataselect (accepts GET requests with parameters) except that it does not support the 'ref' parameter and does not support caching in the internal ICAB system.

fdsnws-station is equivalent to ws-station with these important changes:

  1. the XML output is returned as FDSN StationXML, a different schema
  2. new 'matchtimeseries' parameter to limit results to channels with high probability of data at the DMC
  3. the 'output' parameter name is changed to 'format'

The FDSN StationXML schema (http://www.fdsn.org/xml/station/) is generally equivalent to the XML format previously used but it is a distinctly different schema that cannot be parsed with the same software parsing the XML from ws-station.

fdsnws-event is equivalent to ws-event with these differences:

  1. the FDSN service returns event depths in meters to conform to QuakeML 1.2 (the ws-event service used kilometers)
  2. the 'output' parameter name is changed to 'format'
  3. only a single catalog can be selected per-request, the default catalog is 'NEIC PDE'
  4. in addition to the default order of descending results, the 'orderby' parameter can be 'time-asc' or 'magnitude-asc' to set the order to ascending.

@barsch
Copy link
Member Author

barsch commented Mar 28, 2013

just a rant: we should try to autogenerate our client by using the delivered WADL files (e.g. http://service.iris.edu/fdsnws/event/1/application.wadl ) to keep our clients up-to-date with IRIS - not dynamically on each client run, but once everytime we release a new version of ObsPy

@krischer
Copy link
Member

krischer commented Apr 3, 2013

+1 for the WADL autogeneration if it is possible. Not sure though. For some query options they only specify the default value and not the possible options. Furthermore they do not seem to use xs:boolean for their flags but a string with either "yes" or "no".

Also they now appear to be using a QuakeML version > 1.0. No version number is included for some reason, but the origin depth unit appears to be meter. Judging from their documentation is appears to be QuakeML 1.2RC3.

@celso-iris
Copy link

Before committing to using IRIS's WADL, you should know that the WADLs are in flux. More specifically, IRIS would love to hear recommendations for improving our WADLs and making them more useful. If we perfect the WADL that is used to describe our existing FDSN services, then hopefully future FDSN-based services will also provide better WADLs. Each service implementation may vary from each other in some minor way: some query parameters might not be relevant, or some may be augmented.

As a specific example, here the IRIS FDSN event service differs from the spec. in that IRIS offers a format parameter with two options, text and xml. However, the FDSN spec says nothing about format.

Another example might be if a data center doesn't store a particular type of metadata. Then the parameter used to filter upon that data will likely not be implemented. Hopefully a good WADL could be used to determine that ahead of time.

If you have any questions, feel free to contact us, as we would like to help make it easier for ObsPy to interface with the web services. Now that there are FDSN standardized versions of our core services, ObsPy should be able to nearly automatically / seamlessly interface with the equivalent services at other data centers.

Thanks,
Celso

@krischer
Copy link
Member

krischer commented Apr 5, 2013

Hi Celso,

what is the best way to contact you? It is ok, if we discuss it here?

We are of course interested in being able to automatically access data from centers implementing the FDSN specification. Also if anyone at IRIS is interested in changing parts of obspy.iris feel free to do it and send us a pull request.

I am no expert in WADL and do not know how feasible my suggestions are but they should be valid WADL. From our point of view the following changes would facilitate the autogeneration of code based on your WADLs. It is just a short snippets based on your event WADL.

<request>
  <!--Document the different fields, just done for one example here-->
  <param name="starttime" style="query" type="xs:date">
      <doc xml:lang="english" title="limit to events occurring on or after the specified start time.">
          Example: starttime=2012-11-29T00:00:00
      </doc>
  </param>

  <param name="endtime" style="query" type="xs:date"/>

  <!--Indicate the used default values. Only done for some examples here-->
  <param name="minlatitude" style="query" type="xs:double" default="-90.0"/>
  <param name="maxlatitude" style="query" type="xs:double" default="90.0"/>
  <param name="minlongitude" style="query" type="xs:double" default="-180.0"/>
  <param name="maxlongitude" style="query" type="xs:double" default="180.0"/>

  <param name="minmagnitude" style="query" type="xs:double"/>
  <param name="maxmagnitude" style="query" type="xs:double"/>

  <param name="magtype" style="query" type="xs:string"/>
  <param name="catalog" style="query" type="xs:string"/>
  <param name="contributor" style="query" type="xs:string"/>
  <param name="maxdepth" style="query" type="xs:double"/>
  <param name="mindepth" style="query" type="xs:double"/>
  <param name="latitude" style="query" type="xs:double"/>
  <param name="longitude" style="query" type="xs:double"/>
  <param name="maxradius" style="query" type="xs:double"/>
  <param name="minradius" style="query" type="xs:double"/>
  <param name="orderby" style="query" type="xs:string" default="time"/>
  <param name="updatedafter" style="query" type="xs:date"/>
  <param name="eventid" style="query" type="xs:long"/>
  <param name="originid" style="query" type="xs:long"/>

  <!--Use booleans variables for flags-->
  <param name="includearrivals" style="query" type="xs:boolean" default="false"/>
  <param name="includeallmagnitudes" style="query" type="xs:boolean" default="false"/>
  <param name="includeallorigins" style="query" type="xs:boolean" default="false"/>

  <param name="limit" style="query" type="xs:int" default="1"/>
  <param name="offset" style="query" type="xs:int" default="1"/>
  <param name="output" style="query" type="xs:string" default="xml"/>

  <!--Indicate the available options. Just done for one example here-->
  <param name="format" style="query" default="xml">
   <option value="xml"/>
   <option value="text"/>
  </param>

</request>

Best wishes,

Lion

@celso-iris
Copy link

Here's an example of a beefed up event wadl. It doesn't match the wadl we have on line yet, but we're working on getting everything synced. Nothing's written in stone yet (it can and probably will change), but how's this for wadl documentation of the FDSN event service?

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<application xmlns="http://wadl.dev.java.net/2009/02">
   <doc title="FDSN event web service 1.0"/>
   <resources base="http://service.iris.edu/fdsnws/event/1/">
      <resource path="/">
         <method id="index" name="GET">
            <response>
               <representation mediaType="text/plain"/>
               <representation mediaType="application/xml"/>
            </response>
         </method>
         <resource path="/counts">
            <method id="counts" name="GET">
               <request>
                  <param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="starttime" style="query" type="xs:date">
                     <doc xml:lang="english" title="limit to events occurring on or after the specified start time">
                        Examples: starttime=2012-11-29 or 2012-11-29T00:00:00 or 2012-11-29T00:00:00.000
                     </doc>
                  </param>
                  <param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="endtime" style="query" type="xs:date">
                     <doc xml:lang="english" title="limit to events occurring on or before the specified end time"/>
                  </param>
                  <param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="minlatitude" style="query" type="xs:double" default="-90.0">
                     <doc xml:lang="english" title="southern boundary"/>
                  </param>
                  <param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="maxlatitude" style="query" type="xs:double" default="90.0">
                     <doc xml:lang="english" title="northern boundary"/>
                  </param>
                  <param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="minlongitude" style="query" type="xs:double" default="-180.0">
                     <doc xml:lang="english" title="western boundary"/>
                  </param>
                  <param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="maxlongitude" style="query" type="xs:double" default="180.0">
                     <doc xml:lang="english" title="eastern boundary"/>
                  </param>
                  <param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="minmagnitude" style="query" type="xs:double">
                     <doc xml:lang="english" title="Limit to events with a magnitude larger than or equal to the specified minimum"/>
                  </param>
                  <param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="maxmagnitude" style="query" type="xs:double">
                     <doc xml:lang="english" title="Limit to events with a magnitude smaller than or equal to the specified maximum"/>
                  </param>
                  <param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="magtype" style="query" type="xs:string">
                     <doc xml:lang="english" title="type of Magnitude used to test minimum and maximum limits (case insensitive)">
                     Examples: Ml,Ms,mb,Mw"</doc>
                  </param>
                  <param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="catalog" style="query" type="xs:string" default="NEIC+PDE">
                     <doc xml:lang="english" title="Specify the catalog from which origins and magnitudes will be retrieved"/>
                  </param>
                  <param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="contributor" style="query" type="xs:string">
                     <doc xml:lang="english" title="Limit to events contributed by a specified contributor"/>
                  </param>
                  <param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="maxdepth" style="query" type="xs:double">
                     <doc xml:lang="english" title="Limit to events with depths equal to or less than the specified depth"/>
                  </param>
                  <param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="mindepth" style="query" type="xs:double">
                     <doc xml:lang="english" title="Limit to events with depths equal to or greater than the specified depth"/>
                  </param>
                  <param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="latitude" style="query" type="xs:double" default="0.0">
                     <doc xml:lang="english" title="Specify the central latitude point for a radial search"/>
                  </param>
                  <param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="longitude" style="query" type="xs:double" default="0.0">
                     <doc xml:lang="english" title="Specify the central longitude point for a radial search"/>
                  </param>
                  <param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="maxradius" style="query" type="xs:double" default="180.0">
                     <doc xml:lang="english" title="Specify maximum distance from the geographic point defined by latitude and longitude"/>
                  </param>
                  <param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="minradius" style="query" type="xs:double" default="0.0">
                     <doc xml:lang="english" title="Specify minimum distance from the geographic point defined by latitude and longitude"/>
                  </param>
                  <param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="updatedafter" style="query" type="xs:date">
                     <doc xml:lang="english" title="Limit to events updated after the specified time"/>
                  </param>
                  <!-- <param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="output" style="query" type="xs:string" default="xml"/> -->
                  <param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="format" style="query" type="xs:string" default="xml">
                     <doc xml:lang="english" title=""/>
                     <option value="xml" mediaType="application/xml"/>
                     <option value="text" mediaType="text/plain"/>
                  </param>
               </request>
               <response>
                  <representation mediaType="text/plain"/>
                  <representation mediaType="application/xml"/>
               </response>
            </method>
         </resource>

         <resource path="/query">
            <method id="query" name="GET">
               <request>
                  <param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="starttime" style="query" type="xs:date">
                     <doc xml:lang="english" title="limit to events occurring on or after the specified start time">
                        Examples: starttime=2012-11-29 or 2012-11-29T00:00:00 or 2012-11-29T00:00:00.000
                     </doc>
                  </param>
                  <param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="endtime" style="query" type="xs:date">
                     <doc xml:lang="english" title="limit to events occurring on or before the specified end time"/>
                  </param>
                  <param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="minlatitude" style="query" type="xs:double" default="-90.0">
                     <doc xml:lang="english" title="southern boundary"/>
                  </param>
                  <param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="maxlatitude" style="query" type="xs:double" default="90.0">
                     <doc xml:lang="english" title="northern boundary"/>
                  </param>
                  <param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="minlongitude" style="query" type="xs:double" default="-180.0">
                     <doc xml:lang="english" title="western boundary"/>
                  </param>
                  <param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="maxlongitude" style="query" type="xs:double" default="180.0">
                     <doc xml:lang="english" title="eastern boundary"/>
                  </param>
                  <param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="minmagnitude" style="query" type="xs:double">
                     <doc xml:lang="english" title="Limit to events with a magnitude larger than or equal to the specified minimum"/>
                  </param>
                  <param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="maxmagnitude" style="query" type="xs:double">
                     <doc xml:lang="english" title="Limit to events with a magnitude smaller than or equal to the specified maximum"/>
                  </param>
                  <param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="magtype" style="query" type="xs:string">
                     <doc xml:lang="english" title="type of Magnitude used to test minimum and maximum limits (case insensitive)">
                     Examples: Ml,Ms,mb,Mw"</doc>
                  </param>
                  <param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="catalog" style="query" type="xs:string" default="NEIC+PDE">
                     <doc xml:lang="english" title="Specify the catalog from which origins and magnitudes will be retrieved"/>
                  </param>
                  <param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="contributor" style="query" type="xs:string">
                     <doc xml:lang="english" title="Limit to events contributed by a specified contributor"/>
                  </param>
                  <param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="maxdepth" style="query" type="xs:double">
                     <doc xml:lang="english" title="Limit to events with depths equal to or less than the specified depth"/>
                  </param>
                  <param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="mindepth" style="query" type="xs:double">
                     <doc xml:lang="english" title="Limit to events with depths equal to or greater than the specified depth"/>
                  </param>
                  <param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="latitude" style="query" type="xs:double" default="0.0">
                     <doc xml:lang="english" title="Specify the central latitude point for a radial search"/>
                  </param>
                  <param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="longitude" style="query" type="xs:double" default="0.0">
                     <doc xml:lang="english" title="Specify the central longitude point for a radial search"/>
                  </param>
                  <param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="maxradius" style="query" type="xs:double" default="180.0">
                     <doc xml:lang="english" title="Specify maximum distance from the geographic point defined by latitude and longitude"/>
                  </param>
                  <param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="minradius" style="query" type="xs:double" default="0.0">
                     <doc xml:lang="english" title="Specify minimum distance from the geographic point defined by latitude and longitude"/>
                  </param>
                  <param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="orderby" style="query" type="xs:string" default="time">
                     <doc xml:lang="english" title="Specify the ordering of the returned results"/>
                     <option value="time">
                        <doc xml:lang="english" title="Sort by time, descending"/>
                     </option>
                     <option value="magnitude">
                        <doc xml:lang="english" title="Sort by magnitude, descending"/>
                     </option>
                     <option value="time-asc">
                        <doc xml:lang="english" title="Sort by time, ascending"/>
                     </option>
                     <option value="magnitude-asc">
                        <doc xml:lang="english" title="Sort by magnitude, ascending"/>
                     </option>
                  </param>
                  <param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="updatedafter" style="query" type="xs:date">
                     <doc xml:lang="english" title="Limit to events updated after the specified time"/>
                  </param>
                  <param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="eventid" style="query" type="xs:long">
                     <doc xml:lang="english" title="Retrieve an event based on the unique ID numbers assigned by the IRIS DMC"/>
                  </param>
                  <param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="originid" style="query" type="xs:long">
                     <doc xml:lang="english" title="Retrieve an event based on the unique origin ID numbers assigned by the IRIS DMC"/>
                  </param>
                  <param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="includearrivals" style="query" type="xs:boolean" default="false">
                     <doc xml:lang="english" title="Specify if phase arrivals should be included"/>
                  </param>
                  <param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="includeallmagnitudes" style="query" type="xs:sboolean" default="false">
                     <doc xml:lang="english" title="Retrieve all magnitudes for the event, or only the primary magnitude from within the catalog"/>
                  </param>
                  <param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="includeallorigins" style="query" type="xs:boolean" default="false">
                     <doc xml:lang="english" title="Retrieve all origins or only the primary origin associated with each event in a catalog"/>
                  </param>
                  <param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="limit" style="query" type="xs:int">
                     <doc xml:lang="english" title="Limit the results to the specified number of events"/>
                  </param>
                  <param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="offset" style="query" type="xs:int" default="1">
                     <doc xml:lang="english" title="Return results starting at the event count specified"/>
                  </param>
                  <!-- <param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="output" style="query" type="xs:string" default="xml"/> -->
                  <param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="format" style="query" type="xs:string" default="xml">
                     <doc xml:lang="english" title="Specify output format. This is an IRIS extension to the FDSN specification"/>
                     <option value="xml" mediaType="application/xml"/>
                     <option value="text" mediaType="text/plain"/>
                  </param>
                  <param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="nodata" style="query" type="xs:int" default="204">
                     <doc xml:lang="english" title="Specify which HTML Status code is returned when no data is found. This is an IRIS extension to the FDSN specification"/>
                     <option value="204"/>
                     <option value="404"/>
                  </param>
               </request>
               <response>
                  <representation mediaType="text/plain"/>
                  <representation mediaType="application/xml"/>
               </response>
            </method>
         </resource>
         <resource path="/version">
            <method id="version" name="GET">
               <response>
                  <representation mediaType="text/plain"/>
               </response>
            </method>
         </resource>
         <resource path="/catalogs">
            <method id="catalogs" name="GET">
               <response>
                  <representation mediaType="application/xml"/>
               </response>
            </method>
         </resource>
         <resource path="/contributors">
            <method id="contributors" name="GET">
               <response>
                  <representation mediaType="application/xml"/>
               </response>
            </method>
         </resource>
      </resource>
   </resources>
</application>

@krischer
Copy link
Member

Hi Celso,

this look very good in my opinion and should enable us to query the webservice without specifying every option by hand. We might encounter more subtle problems as soon as we start working on an implementation.

@barsch: Right now I would actually prefer to create a generic FDSN webservice that is able to query all datacenters providing one. The corresponding WADL can be downloaded on-the-fly during client initialization.

@celso-iris: I am currently working on StationXML support for ObsPy and noticed that the returned StationXML files do not validate against the schema. This should require only some very minor changes. Examples:

The QuakeML files are fine :)

@barsch
Copy link
Member Author

barsch commented Apr 16, 2013

@krischer: obspy.fdsn is definitively the way to go - I wanted to propose that anyway ;)

@krischer
Copy link
Member

+1

Orfeus is AFAIK currently also working on an implementation.

@megies
Copy link
Member

megies commented Apr 17, 2013

@krischer: did you validate requested QuakeML files against the official schemas/relaxNG? So far, those accepted a lot that was not conform with the UML/pdf documentation, from my experience...

@megies
Copy link
Member

megies commented Jun 23, 2013

Was thinking about starting work on this at some point. Just to make sure, this is supposed to go into a new module obspy.fdsn and the client should basically take an url at initialization which will default to the IRIS ws for now. Did I get this right?

@celso-iris
Copy link

@megies : That sounds correct to me.
You might like to know that the USGS has their version of the FDSN event webservice up and running now.

http://comcat.cr.usgs.gov/fdsnws/event/1/
http://comcat.cr.usgs.gov/fdsnws/event/1/application.wadl

and RESIF has data and metadata services up and running:

http://ws.resif.fr/

These might be a good places to try alternate URLs. The USGS happens to have many extensions to the FDSN spec. I don't know how you might handle it. In the IRIS-WS java library, we added an "add" routine to the EventCriteria object that takes an arbitrary string pair, and adds the results to the query url.

@krischer
Copy link
Member

I would suggest that we directly parse the WADL files upon client instantiation. We could also provide some kind of mapping for known services.

Something like this might work:

WADL_MAPPINGS = {
    "IRIS": "url to iris wadl",
    "USGS": "url to usgs wadl",
    "RESIF": "url to resif wadl",
    ...
}


class Client(object):
    def __init__(self, WADL):
        if WADL in WADL_MAPPINGS:
            self._parse_wadl(WADL_MAPPINGS[WADL])
        else:
            self._parse_wadl(WADL)


# Usage
from obspy.fdsn import Client
c = Client("IRIS")
...

@megies
Copy link
Member

megies commented Jun 25, 2013

I thought we consented to not using WADL in the initial implementation?

edit: talked with @krischer, should probably parse the wadls right from the start..

@megies megies mentioned this pull request Oct 4, 2013
7 tasks
@megies
Copy link
Member

megies commented Oct 4, 2013

I'm OK with merging this. Open points are tracked in #646.

@krischer
Copy link
Member

krischer commented Oct 4, 2013

@celso-iris I found a small bug: The event HTML page does not describe the originid parameter, but the WADL does.

krischer added a commit that referenced this pull request Oct 4, 2013
ObsPy now has support for the FDSN webservices.
@krischer krischer merged commit ff05839 into master Oct 4, 2013
@mbyt mbyt deleted the obspy.fdsn branch February 16, 2014 12:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
.clients.fdsn .clients.iris release blocker critical issues blocking the corresponding future release task
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants