tune the output information in ASDFDataSet.process() method

[wjlei1990]

Hi Lion,

I am currently using this to process asdf file and I found the output information is limited. For example, in the following output, the interpolation method raise a ValueError but I don't know what stream(or trace) is that. I think knowing the stream.id is very helpful so I can go into the hdf5 file and see what is wrong with that trace.

Do you have specific reasons to do so(simple terminal output)?

Terminal Output:

--------------------------------------------------------------------------
Processing!!! 1445269490.98
Processing!!! 1445269490.98
Processing!!! 1445269490.98
Processing!!! 1445269490.99
Process Process-2:
Traceback (most recent call last):
  File "/ccs/home/lei/anaconda/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/autofs/nccs-svm1_home1/lei/software/pyasdf/pyasdf/asdf_data_set.py", line 1680, in run
    output_stream = self.processing_function(stream, inv)
  File "/autofs/nccs-svm1_home1/lei/test/git_merge/source_inversion_wf/src/preproc_asdf/proc_util.py", line 112, in process_function
    cut_func(st, starttime, endtime)
  File "/autofs/nccs-svm1_home1/lei/test/git_merge/source_inversion_wf/src/preproc_asdf/proc_util.py", line 51, in cut_func
    tr_list.append(flex_cut_trace(tr, t1, t2))
  File "/autofs/nccs-svm1_home1/lei/test/git_merge/source_inversion_wf/src/preproc_asdf/proc_util.py", line 39, in flex_cut_trace
    return tr.slice(cut_starttime, cut_endtime)
  File "/ccs/home/lei/anaconda/lib/python2.7/site-packages/obspy/core/trace.py", line 1114, in slice
    tr.trim(starttime=starttime, endtime=endtime)
  File "/ccs/home/lei/anaconda/lib/python2.7/site-packages/obspy/core/trace.py", line 231, in new_func
    result = func(*args, **kwargs)
  File "/ccs/home/lei/anaconda/lib/python2.7/site-packages/obspy/core/trace.py", line 1075, in trim
    raise ValueError("startime is larger than endtime")

[krischer]

I'll work on improving the error reporting and handling!

[wjlei1990]

Great.

This is the output using multiprocessing. What I am going to using mpi in the future :) I think for mpi case, it is probably hard to output this kind of things, right?

Can we have something like:

try:
    process_blalbla(trace)
except Exception as err:
    print("%s: %s" % (trace.id, err))

Very primitive thoughts...Ignore me if it doesn't make sense.

[krischer]

Sure. Already at it. Should be done in the next half hour :-)

[krischer]

This took a bit longer but I also wrote some more documentation: http://seismicdata.github.io/pyasdf/parallel_processing.html

The latest master has much better output and error reporting when run with multiprocessing. Additionally you can now set the number of processors and the length of the printed traceback for the multiprocessing processing function.

I think this solves your problem :-)

$ python process_test.py
Launching processing using multiprocessing on 8 cores ...
 -> Processing approximately task 0 of 107 ...
 -> Processing approximately task 8 of 107 ...
 -> Processing approximately task 16 of 107 ...
 -> Processing approximately task 24 of 107 ...
 -> Processing approximately task 32 of 107 ...
 -> Processing approximately task 40 of 107 ...

Error during the processing of station 'IC.LSA' and tag 'synthetic' on CPU 2:
Traceback (At max 3 levels - most recent call last):
  File "/Users/lion/.miniconda3/envs/obspy_py34/lib/python3.4/threading.py", line 888, in _bootstrap
    self._bootstrap_inner()
  File "/Users/lion/.miniconda3/envs/obspy_py34/lib/python3.4/threading.py", line 920, in _bootstrap_inner
    self.run()
  File "/Users/lion/workspace/code/pyasdf/pyasdf/asdf_data_set.py", line 1785, in run
    output_stream = self.processing_function(stream, inv)
  File "process_test.py", line 9, in process
    raise ValueError("Something went wrong!!!")

ValueError: Something went wrong!!!

 -> Processing approximately task 48 of 107 ...
 -> Processing approximately task 56 of 107 ...
 -> Processing approximately task 64 of 107 ...
 -> Processing approximately task 72 of 107 ...

Error during the processing of station 'IU.DAV' and tag 'synthetic' on CPU 0:
Traceback (At max 3 levels - most recent call last):
  File "/Users/lion/.miniconda3/envs/obspy_py34/lib/python3.4/threading.py", line 888, in _bootstrap
    self._bootstrap_inner()
  File "/Users/lion/.miniconda3/envs/obspy_py34/lib/python3.4/threading.py", line 920, in _bootstrap_inner
    self.run()
  File "/Users/lion/workspace/code/pyasdf/pyasdf/asdf_data_set.py", line 1785, in run
    output_stream = self.processing_function(stream, inv)
  File "process_test.py", line 9, in process
    raise ValueError("Something went wrong!!!")

ValueError: Something went wrong!!!

 -> Processing approximately task 80 of 107 ...
 -> Processing approximately task 88 of 107 ...
 -> Processing approximately task 96 of 107 ...
 -> Processing approximately task 104 of 107 ...
Finished processing in 3.13 seconds (0.029 sec/station).

[wjlei1990]

👍

Thanks, Lion. That is great!

[krischer]

c75d65b does the same for the MPI based processing so it should be much easier to diagnose problems now.