ISC event syncing with seiscomp3

[basaks]

@alexgorb says we need to have a provision to use the events as reported by ISC. We can use this service for that: http://www.isc.ac.uk/iscbulletin/search/bulletin/

We can download quakemls, use obspy to convert to sc3ml and ingest into seiscomp3.

[alexgorb]

Yes indeed…

[niketchhajed]

Clarified some points with @alexgorb about narrowing the search for ISC events for the ultimate purpose of inversion.

We need to search for ISC events with epicenters that lie within the rectangular region drawn above plus a radius of 90 degrees.

The time frame needs to be starting from when the ENGDAHL events ended i.e. 31st March 2009 up-to 1st October 2014, the date upto which ISC has reviewed events, as of this writing.

Within the above region, events with only magnitude 3 and above need to be considered. And for the region outside this spherical region, we need to consider events 5 magnitude and above.

[niketchhajed]

After taking the inputs from @alexgorb, the scripts at https://github.com/GeoscienceAustralia/passive-seismic/tree/pst-12/isc-events have been fired on an aws ec2 instance in London region. The script is current running and should generate the isc events for the period starting 1st April 2009 upto 1st October 2014, unless there are any connectivity or other issues encountered. The events are being dumped for a 1 hour interval. So, in case if therea re any errors encountered, we will know what failed and those can be retried later.

[niketchhajed]

The running script hit an unexpected condition and could not finish. For one of the 1 hour slots, the quakeml response returned was:

<?xml version="1.0" encoding="UTF-8"?>

<quakeml xmlns="http://quakeml.org/xmlns/quakeml/1.2">No events were found.

Will handle it in the scripts and re-trigger.

[niketchhajed]

Some of the quakeml responses are like below:

<?xml version="1.0" encoding="UTF-8"?>

<quakeml xmlns="http://quakeml.org/xmlns/quakeml/1.2">Sorry, but your request cannot be processed at the present time. Please try again in a few minutes.
<br>International Seismological Centre<br>Pipers Lane, Thatcham, Berkshire<br>United Kingdom RG19 4NS<br>+44 (0)1635 861022, voice; +44 (0)1635 872351, fax<br>admin<img src="/images/staff/at.jpg" height="13"></img>isc.ac.uk, e-mail<br><br></pre>
</body>
</html>

Once this pass of 3 threads running in parallel is complete, will re-run for the time slots that did not return valid events quakeml.

[niketchhajed]

ubuntu@ip-172-31-11-9:~$ grep "<event " `find . -name *.xml | xargs` | wc -l
125351

125351 ISC reviewed events have been dumped into SC3ML files.

ubuntu@ip-172-31-11-9:~$ grep "returned no events" `find . -name nohup.out` | wc -l
26745

And 26745 attempts to query the isc server have not returned results. Each attempt is for 1 hour time period. Typically, one hour has 2 to 6 events. So we are looking at another ~75K events, could be more.

Will fire the scripts with the failed attempts.

[niketchhajed]

Have updated the scripts to factor in the re-running of failed attempts. Also re-factored code.

The modified script has been fired to get the missed isc events.

[niketchhajed]

The bulk of the ISC events have been downloaded:

ubuntu@ip-172-31-11-9:~/failed-attempts$ grep -r "<event " . | wc -l
274255

Only about 400 hours (or ~17 days worth of events) of more downloading remains:

ubuntu@ip-172-31-11-9:~/failed-attempts$ grep "returned no events" nohup.out | wc -l
399

Have re-triggered to download the differential.

Next step is to ingest these events and the engdahl events into the POC Seiscomp3 instance and re-review with @alexgorb. Also, the receiver function environment setup seems to be ready as well as the FDSN webservice setup should be up and running. If everything goes fine, we should be able to see the receiver functions being generated for the ISC/ENGDAHL events. But, I guess this will take some time.

[Zephyrpony]

Jira task PST-217
https://gajira.atlassian.net/browse/PST-217