Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Resource id refactor #2091

Merged
merged 37 commits into from Jul 5, 2018
Merged
Show file tree
Hide file tree
Changes from 23 commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
763c447
begin refactor
d-chambers Mar 15, 2018
886e923
continue refactor of ResourceID
d-chambers Mar 19, 2018
54bc75e
fix docstrings, use _debug_class_state context manager
d-chambers Mar 20, 2018
e566026
python 2 support
d-chambers Mar 20, 2018
81f244a
small style fixes
d-chambers Mar 20, 2018
a18356f
add code and tests for pickle support
d-chambers Mar 21, 2018
2f14d37
address krischer review
d-chambers Mar 27, 2018
56f56be
add test for multithreaded catalog creation
d-chambers Apr 6, 2018
1480ef4
start parent_key refactor
d-chambers Apr 8, 2018
c68311e
work on resource_id refactor
d-chambers Apr 10, 2018
d920cf9
continue refactor
d-chambers Apr 12, 2018
02d3d45
fix resource_id doctest and flake issue in test
d-chambers Apr 14, 2018
0d14ce0
cleanup test catalog creation
d-chambers Apr 14, 2018
09b8491
update resource_id docs
d-chambers Apr 14, 2018
8c878ab
refactor recursive object finder
d-chambers Apr 15, 2018
6b36a38
exclude built-ins from unique id tests
d-chambers Apr 15, 2018
5c94faa
implement custom function for resource_id recursion
d-chambers Apr 15, 2018
e61334c
py27 fix and pep8 issues
d-chambers Apr 15, 2018
3bfea0a
fix CamelCase name in test_util_misc
d-chambers Apr 15, 2018
18c9d05
rename resource_id import
d-chambers Apr 16, 2018
ba5aa11
Consistent sphinx references.
krischer Jul 3, 2018
977120c
Removing the now obsolete rlock() decorator.
krischer Jul 3, 2018
160d8dc
Also removing the _decorate_polyfill() function.
krischer Jul 3, 2018
83bca69
de-indent _yield_resource_id_parent_attr
d-chambers Jul 4, 2018
9c01708
change order of fun inputs in _yield_resource_id_parent_attr
d-chambers Jul 4, 2018
207b856
re-enable skipped doctests
d-chambers Jul 4, 2018
63e7817
add test for warnings not being raised
d-chambers Jul 4, 2018
aa26cfd
fix warning message
d-chambers Jul 4, 2018
2fcbbad
Fixing an issue introduced via the rebasing.
krischer Jul 4, 2018
5b87da9
Readding another test that got lost in the rebase.
krischer Jul 4, 2018
a9d923b
Some cleanup.
krischer Jul 4, 2018
62e52dd
Adding changelog entry.
krischer Jul 4, 2018
be8a415
Scope the resource ids for each event reading plugin.
krischer Jul 4, 2018
ba9ecaf
Learning the alphabet.
krischer Jul 4, 2018
33e0f8c
fix typo in warning
d-chambers Jul 4, 2018
dffd40d
add test for better warning
d-chambers Jul 4, 2018
fce9aa5
fix resource_id warning logic
d-chambers Jul 4, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
3 changes: 2 additions & 1 deletion obspy/core/event/__init__.py
Expand Up @@ -25,7 +25,8 @@

from .base import (
Comment, CompositeTime, ConfidenceEllipsoid, CreationInfo, DataUsed,
QuantityError, ResourceIdentifier, TimeWindow, WaveformStreamID)
QuantityError, TimeWindow, WaveformStreamID)
from obspy.core.event.resourceid import ResourceIdentifier
from .catalog import Catalog, read_events
from .event import Event, EventDescription
from .magnitude import (
Expand Down
550 changes: 7 additions & 543 deletions obspy/core/event/base.py

Large diffs are not rendered by default.

22 changes: 15 additions & 7 deletions obspy/core/event/catalog.py
Expand Up @@ -34,12 +34,12 @@
from obspy.core.util import NamedTemporaryFile, _read_from_plugin
from obspy.core.util.base import (ENTRY_POINTS, download_to_file,
sanitize_filename)
from obspy.core.util.decorator import (map_example_filename, rlock,
uncompress_file)
from obspy.core.util.decorator import map_example_filename, uncompress_file
from obspy.core.util.misc import buffered_load_entry_point
from obspy.imaging.cm import obspy_sequential

from .base import CreationInfo, ResourceIdentifier
from .base import CreationInfo
from obspy.core.event import ResourceIdentifier

from .event import Event

Expand All @@ -53,7 +53,7 @@ class Catalog(object):

:type events: list of :class:`~obspy.core.event.event.Event`, optional
:param events: List of events
:type resource_id: :class:`~obspy.core.event.base.ResourceIdentifier`
:type resource_id: :class:`~obspy.core.event.resourceid.ResourceIdentifier`
:param resource_id: Resource identifier of the catalog.
:type description: str, optional
:param description: Description string that can be assigned to the
Expand Down Expand Up @@ -84,12 +84,22 @@ def _get_resource_id(self):
return self.__dict__['resource_id']

def _set_resource_id(self, value):
if type(value) == dict:
if isinstance(value, dict):
value = ResourceIdentifier(**value)
elif type(value) != ResourceIdentifier:
value = ResourceIdentifier(value)
value.set_referred_object(self, warn=False)
self.__dict__['resource_id'] = value

def __setstate__(self, state):
"""
Reset the resource id after being unpickled to ensure they are
bound to the correct object.
"""
state['resource_id'].set_referred_object(self, warn=False,
parent=self)
self.__dict__.update(state)

resource_id = property(_get_resource_id, _set_resource_id)

def _get_creation_info(self):
Expand Down Expand Up @@ -765,7 +775,6 @@ def plot(self, projection='global', resolution='l',
return fig


@rlock
@map_example_filename("pathname_or_url")
def read_events(pathname_or_url=None, format=None, **kwargs):
"""
Expand Down Expand Up @@ -845,7 +854,6 @@ def read_events(pathname_or_url=None, format=None, **kwargs):
if len(pathnames) > 1:
for filename in pathnames[1:]:
catalog.extend(_read(filename, format, **kwargs).events)
ResourceIdentifier.bind_resource_ids()
return catalog


Expand Down
37 changes: 31 additions & 6 deletions obspy/core/event/event.py
Expand Up @@ -20,16 +20,17 @@
from __future__ import (absolute_import, division, print_function,
unicode_literals)
from future.builtins import * # NOQA

import copy

from obspy.core.event.header import (
EventType, EventTypeCertainty, EventDescriptionType)
from obspy.core.util.decorator import rlock
from obspy.core.event.resourceid import ResourceIdentifier
from obspy.core.util.misc import _yield_resource_id_parent_attr
from obspy.imaging.source import plot_radiation_pattern, _setup_figure_and_axes


from .base import (_event_type_class_factory,
CreationInfo, ResourceIdentifier)
from .base import _event_type_class_factory, CreationInfo


__Event = _event_type_class_factory(
Expand Down Expand Up @@ -57,7 +58,7 @@ class Event(__Event):
event is usually associated with one or more magnitudes, and with one or
more focal mechanism determinations.

:type resource_id: :class:`~obspy.core.event.base.ResourceIdentifier`
:type resource_id: :class:`~obspy.core.event.resourceid.ResourceIdentifier`
:param resource_id: Resource identifier of Event.
:type force_resource_id: bool, optional
:param force_resource_id: If set to False, the automatic initialization of
Expand Down Expand Up @@ -102,6 +103,10 @@ class Event(__Event):
"""
do_not_warn_on = ["_format", "extra"]

def __init__(self, *args, **kwargs):
super(Event, self).__init__(*args, **kwargs)
self.scope_resource_ids()

def short_str(self):
"""
Returns a short string representation of the current Event.
Expand Down Expand Up @@ -275,7 +280,6 @@ def plot(self, kind=[['ortho', 'beachball'], ['p_sphere', 's_sphere']],

return fig

@rlock
def __deepcopy__(self, memodict=None):
"""
reset resource_id's object_id after deep copy to allow the
Expand All @@ -287,9 +291,13 @@ def __deepcopy__(self, memodict=None):
memodict[id(self)] = result
for k, v in self.__dict__.items():
setattr(result, k, copy.deepcopy(v, memodict))
result.resource_id.bind_resource_ids() # bind all resource_ids
result.scope_resource_ids()
return result

def __setstate__(self, state_dict):
super(Event, self).__setstate__(state_dict)
self.scope_resource_ids()

def write(self, filename, format, **kwargs):
"""
Saves event information into a file.
Expand All @@ -311,6 +319,23 @@ def write(self, filename, format, **kwargs):
from .catalog import Catalog
Catalog(events=[self]).write(filename, format, **kwargs)

def scope_resource_ids(self):
"""
Ensure all resource_ids in event instance are event-scoped.

This will ensure the resource_ids refer to objects in the event
structure when possible.
"""
gen = _yield_resource_id_parent_attr(self)

for resource_id, parent, attr in gen:
if attr == 'resource_id':
resource_id.set_referred_object(parent, parent=self,
warn=False)
else:
resource_id._parent_key = self
resource_id._object_id = None
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How are things like the derived origin ids scoped after this function has been called? _parent_key would be set but _parent_id_tree and things like this might not be set. But I just be misunderstanding some things.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any object that is referenced by a resource_id should have the attribute resource_id. Lines 332-334 take care of this case, which would involve logging the identities of all the objects with resource_id attributes into the proper place in the _parent_id_tree. Then, resource ids that refer to other objects in the same event (but not their parent) such as the pick_id of the Arrival object will find the correct object when get_referred_object is called because it was previously put into _parent_id_tree. Because _object_id on has been set to None it will not raise a warning. I added another test to specifically check for warnings being raised in this case.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 Thanks for the explanation. After spending some more time I think I now fully understand your changes and how everything works. Nice job!



__EventDescription = _event_type_class_factory(
"__EventDescription",
Expand Down
38 changes: 17 additions & 21 deletions obspy/core/event/magnitude.py
Expand Up @@ -23,8 +23,9 @@

from obspy import UTCDateTime
from obspy.core.event.base import (
_event_type_class_factory, ResourceIdentifier, CreationInfo,
_event_type_class_factory, CreationInfo,
WaveformStreamID, TimeWindow)
from obspy.core.event import ResourceIdentifier
from obspy.core.event.header import (
AmplitudeCategory, AmplitudeUnit, EvaluationMode, EvaluationStatus,
ATTRIBUTE_HAS_ERRORS)
Expand Down Expand Up @@ -54,7 +55,7 @@ class Magnitude(__Magnitude):
originID. It is either a combination of different magnitude estimations, or
it represents the reported magnitude for the given event.

:type resource_id: :class:`~obspy.core.event.base.ResourceIdentifier`
:type resource_id: :class:`~obspy.core.event.resourceid.ResourceIdentifier`
:param resource_id: Resource identifier of Magnitude.
:type force_resource_id: bool, optional
:param force_resource_id: If set to False, the automatic initialization of
Expand All @@ -80,11 +81,11 @@ class Magnitude(__Magnitude):
* coda magnitude (``'Mc'``)
* ``'MH'``, ``'Mwp'``, ``'M50'``, ``'M100'``, etc.

:type origin_id: :class:`~obspy.core.event.base.ResourceIdentifier`,
:type origin_id: :class:`~obspy.core.event.resourceid.ResourceIdentifier`,
optional
:param origin_id: Reference to an origin’s resource_id if the magnitude has
an associated Origin.
:type method_id: :class:`~obspy.core.event.base.ResourceIdentifier`,
:type method_id: :class:`~obspy.core.event.resourceid.ResourceIdentifier`,
optional
:param method_id: Identifies the method of magnitude estimation. Users
should avoid to give contradictory information in method_id and
Expand Down Expand Up @@ -138,14 +139,13 @@ class StationMagnitude(__StationMagnitude):
"""
This class describes the magnitude derived from a single waveform stream.

:type resource_id: :class:`~obspy.core.event.base.ResourceIdentifier`,
optional
:type resource_id:
:class:`~obspy.core.event.resourceid.ResourceIdentifier`
:param resource_id: Resource identifier of StationMagnitude.
:type force_resource_id: bool, optional
:param force_resource_id: If set to False, the automatic initialization of
`resource_id` attribute in case it is not specified will be skipped.
:type origin_id: :class:`~obspy.core.event.base.ResourceIdentifier`,
optional
:type origin_id: :class:`~obspy.core.event.resourceid.ResourceIdentifier`
:param origin_id: Reference to an origin’s ``resource_id`` if the
StationMagnitude has an associated :class:`~obspy.core.event.Origin`.
:type mag: float
Expand All @@ -155,13 +155,12 @@ class StationMagnitude(__StationMagnitude):
:type station_magnitude_type: str, optional
:param station_magnitude_type: See
:class:`~obspy.core.event.magnitude.Magnitude`
:type amplitude_id: :class:`~obspy.core.event.base.ResourceIdentifier`,
optional
:type amplitude_id:
:class:`~obspy.core.event.resourceid.ResourceIdentifier`
:param amplitude_id: Identifies the data source of the StationMagnitude.
For magnitudes derived from amplitudes in waveforms (e.g., local
magnitude ML), amplitudeID points to publicID in class Amplitude.
:type method_id: :class:`~obspy.core.event.base.ResourceIdentifier`,
optional
:type method_id: :class:`~obspy.core.event.resourceid.ResourceIdentifier`
:param method_id: See :class:`~obspy.core.event.magnitude.Magnitude`
:type waveform_id: :class:`~obspy.core.event.base.WaveformStreamID`,
optional
Expand Down Expand Up @@ -196,7 +195,7 @@ class StationMagnitudeContribution(__StationMagnitudeContribution):
StationMagnitude objects for computing a network magnitude estimation.

:type station_magnitude_id:
:class:`~obspy.core.event.base.ResourceIdentifier`, optional
:class:`~obspy.core.event.resourceid.ResourceIdentifier`
:param station_magnitude_id: Refers to the resource_id of a
StationMagnitude object.
:type residual: float, optional
Expand Down Expand Up @@ -244,7 +243,7 @@ class Amplitude(__Amplitude):
single amplitude measurement or a measurement of the visible signal
duration for duration magnitudes.

:type resource_id: :class:`~obspy.core.event.base.ResourceIdentifier`
:type resource_id: :class:`~obspy.core.event.resourceid.ResourceIdentifier`
:param resource_id: Resource identifier of Amplitude.
:type force_resource_id: bool, optional
:param force_resource_id: If set to False, the automatic initialization of
Expand Down Expand Up @@ -292,8 +291,7 @@ class Amplitude(__Amplitude):
combinations of SI base units.
See :class:`~obspy.core.event.header.AmplitudeUnit` for allowed
values.
:type method_id: :class:`~obspy.core.event.base.ResourceIdentifier`,
optional
:type method_id: :class:`~obspy.core.event.resourceid.ResourceIdentifier`
:param method_id: Describes the method of amplitude determination.
:type period: float, optional
:param period: Dominant period in the timeWindow in case of amplitude
Expand All @@ -304,15 +302,13 @@ class Amplitude(__Amplitude):
:type time_window: :class:`~obspy.core.event.base.TimeWindow`, optional
:param time_window: Description of the time window used for amplitude
measurement. Recommended for duration magnitudes.
:type pick_id: :class:`~obspy.core.event.base.ResourceIdentifier`, optional
:type pick_id: :class:`~obspy.core.event.resourceid.ResourceIdentifier`
:param pick_id: Refers to the ``resource_id`` of an associated
:class:`~obspy.core.event.origin.Pick` object.
:type waveform_id: :class:`~obspy.core.event.base.ResourceIdentifier`,
optional
:type waveform_id: :class:`~obspy.core.event.resourceid.ResourceIdentifier`
:param waveform_id: Identifies the waveform stream on which the amplitude
was measured.
:type filter_id: :class:`~obspy.core.event.base.ResourceIdentifier`,
optional
:type filter_id: :class:`~obspy.core.event.resourceid.ResourceIdentifier`
:param filter_id: Identifies the filter or filter setup used for filtering
the waveform stream referenced by ``waveform_id``.
:type scaling_time: :class:`~obspy.core.utcdatetime.UTCDateTime`, optional
Expand Down
32 changes: 15 additions & 17 deletions obspy/core/event/origin.py
Expand Up @@ -23,8 +23,9 @@

from obspy import UTCDateTime
from obspy.core.event.base import (
_event_type_class_factory, ResourceIdentifier, CreationInfo,
_event_type_class_factory, CreationInfo,
WaveformStreamID, ConfidenceEllipsoid)
from obspy.core.event import ResourceIdentifier
from obspy.core.event.header import (
EvaluationMode, EvaluationStatus, OriginDepthType, OriginType,
OriginUncertaintyDescription, PickOnset, PickPolarity,
Expand Down Expand Up @@ -183,7 +184,7 @@ class Origin(__Origin):
earthquake hypocenter, as well as additional meta-information. Origin can
have objects of type OriginUncertainty and Arrival as child elements.

:type resource_id: :class:`~obspy.core.event.base.ResourceIdentifier`
:type resource_id: :class:`~obspy.core.event.resourceid.ResourceIdentifier`
:param resource_id: Resource identifier of Origin.
:type force_resource_id: bool, optional
:param force_resource_id: If set to False, the automatic initialization of
Expand Down Expand Up @@ -223,16 +224,15 @@ class Origin(__Origin):
:param epicenter_fixed: True if epicenter was kept fixed for computation of
Origin.
:type reference_system_id:
:class:`~obspy.core.event.base.ResourceIdentifier`, optional
:class:`~obspy.core.event.resourceid.ResourceIdentifier`
:param reference_system_id: Identifies the reference system used for
hypocenter determination. This is only necessary if a modified version
of the standard (with local extensions) is used that provides a
non-standard coordinate system.
:type method_id: :class:`~obspy.core.event.base.ResourceIdentifier`,
optional
:type method_id: :class:`~obspy.core.event.resourceid.ResourceIdentifier`
:param method_id: Identifies the method used for locating the event.
:type earth_model_id: :class:`~obspy.core.event.base.ResourceIdentifier`,
optional
:type earth_model_id:
:class:`~obspy.core.event.resourceid.ResourceIdentifier`
:param earth_model_id: Identifies the earth model used in method_id.
:type arrivals: list of :class:`~obspy.core.event.origin.Arrival`, optional
:param arrivals: List of arrivals associated with the origin.
Expand Down Expand Up @@ -326,7 +326,7 @@ class Pick(__Pick):
A pick is the observation of an amplitude anomaly in a seismogram at a
specific point in time. It is not necessarily related to a seismic event.

:type resource_id: :class:`~obspy.core.event.base.ResourceIdentifier`
:type resource_id: :class:`~obspy.core.event.resourceid.ResourceIdentifier`
:param resource_id: Resource identifier of Pick.
:type force_resource_id: bool, optional
:param force_resource_id: If set to False, the automatic initialization of
Expand All @@ -337,12 +337,10 @@ class Pick(__Pick):
:param time_errors: AttribDict containing error quantities.
:type waveform_id: :class:`~obspy.core.event.base.WaveformStreamID`
:param waveform_id: Identifies the waveform stream.
:type filter_id: :class:`~obspy.core.event.base.ResourceIdentifier`,
optional
:type filter_id: :class:`~obspy.core.event.resourceid.ResourceIdentifier`
:param filter_id: Identifies the filter or filter setup used for filtering
the waveform stream referenced by waveform_id.
:type method_id: :class:`~obspy.core.event.base.ResourceIdentifier`,
optional
:type method_id: :class:`~obspy.core.event.resourceid.ResourceIdentifier`
:param method_id: Identifies the picker that produced the pick. This can be
either a detection software program or a person.
:type horizontal_slowness: float, optional
Expand All @@ -357,7 +355,7 @@ class Pick(__Pick):
:type backazimuth_errors: :class:`~obspy.core.event.base.QuantityError`
:param backazimuth_errors: AttribDict containing error quantities.
:type slowness_method_id:
:class:`~obspy.core.event.base.ResourceIdentifier`, optional
:class:`~obspy.core.event.resourceid.ResourceIdentifier`
:param slowness_method_id: Identifies the method that was used to determine
the slowness.
:type onset: str, optional
Expand Down Expand Up @@ -423,12 +421,12 @@ class Arrival(__Arrival):
slowness and backazimuth of the observed wave—especially if derived from
array data—may further constrain the nature of the arrival.

:type resource_id: :class:`~obspy.core.event.base.ResourceIdentifier`
:type resource_id: :class:`~obspy.core.event.resourceid.ResourceIdentifier`
:param resource_id: Resource identifier of Arrival.
:type force_resource_id: bool, optional
:param force_resource_id: If set to False, the automatic initialization of
`resource_id` attribute in case it is not specified will be skipped.
:type pick_id: :class:`~obspy.core.event.base.ResourceIdentifier`
:type pick_id: :class:`~obspy.core.event.resourceid.ResourceIdentifier`
:param pick_id: Refers to the resource_id of a Pick.
:type phase: str
:param phase: Phase identification. For possible values, please refer to
Expand Down Expand Up @@ -471,8 +469,8 @@ class Pick).
:param backazimuth_weight: Weight of the backazimuth for computation of the
associated Origin. Note that the sum of all weights is not required to
be unity.
:type earth_model_id: :class:`~obspy.core.event.base.ResourceIdentifier`,
optional
:type earth_model_id:
:class:`~obspy.core.event.resourceid.ResourceIdentifier`
:param earth_model_id: Earth model which is used for the association of
Arrival to Pick and computation of the residuals.
:type comments: list of :class:`~obspy.core.event.base.Comment`, optional
Expand Down