Filters "applied" booleans should be reversed

[kkappler]

There is an inconsistency in the filter attributes. When creating an MTH5 from FDSN, filters in a ChannelTS have their "applied" attributes set to False by default, and "applied" is set to True when the instrument response is removed.

The final units of the time series, after application of remove_instrument_response (for example in mth5_driver example), seem correct but they correspond to filter "units_in" of the last stage, not filter "units_out".

Intuition says:

• If a filter was "applied", the units of the time series (to which it was applied) should be the filters "units_out".
• To apply filter to a ChannelTS, the Channel TS should be in units of "units_in".

The reason for this apparent dissonance: FDSN filters use input and output units to describe the role of the transducer according to the signal flow from external source field to archived digital record. E.g. Here is a snippet of XML from CAS04 response

<InputUnits>
<Name>mV/km</Name>
<Description>millivolt per kilometer</Description>
</InputUnits>
<OutputUnits>
<Name>count</Name>
<Description>digital counts</Description>
</OutputUnits>

Removal of the instrument response does not correspond to "applying" these filters, but to "unapplying" them. Thus we divide by the filter response (invert the filter operation), rather than multiply by it (apply the filter). This is not only an FDSN convention -- manufacturer calibration functions for magnetometers for example are provided in terms of V/nT, i.e. Output units over Input units.

We should keep this convention inside FilterBase.

Complications may arise if we want to work with more general filters. Consider the case where a user wishes to remove the channel response and then band pass the data. The band pass filter is not in the filters list and there is currently no record keeping for this filter if it is requested. For the MTH5 to be self-describing we could add a record of this band pass operation to any channel against which it has been applied (and it would seem that the "applied" boolean should be marked as True in whatever list is tracking whether it was applied). Note This means that the "state" of the filters is non-uniform at the end of the process, the response filters have applied=False, the band pass has applied=True.

The role of the filters needs to be scoped before it winds up trying to do too much. The "Filters Group" in the MTH5 is concerned with calibration and system response information and not with processing filters. Other filters, such as band pass, and notch, etc. were not intended to be handled here. The other filters captured by using a workflow pattern (such as the processing class used to compute TFs and FCs). This can be as simple as a dictionary that contains the parameters to pass to the calibration operation, which can include specification of the band pass filter.

---

---

Desired Outcome

• Channel time series describes available channel response filters in sequence of application
• A corresponding list of booleans tracks which filters were applied, and which are un-applied (corrected for).

Approaches / Conventions / Considerations

• Filters are defined with the forward (multiplicative) operation being in the direction of the signal process flow from the fundamental measurement to the archived digital record.

  • This mirrors FDSN convention and simplifies translation into or out of our convention, as no extra bookkeeping (kludging in swaps of input units and output units) is required.

• The "applied" attribute reflects whether the forward application of the filter has been carried out on the ChannelTS

• Units In /Units Out needed for all filters?

  • A bandpass filter for example, is defined only in terms of coefficients, it can operate on any time series of appropriate sample rate. While an actual application of such a filter will have input and output units, it is not obvious that we should store it with input and output units. Transducer filters on the other hand seem like they should have input and output units stored.
  • We currently support filters with undefined units, largely because there are still station XML that do not have units defined.

---

Tasks:
Players on the stage are:
"applied", "units_in", "units_out", and the units associated with the ChannelTS object

• change default boolean applied to have the value True, or change it's name to correction_applied
  • there are 90 instances of applied in mt_metadata, and 73 in mth5 :/
  • channel.filter.applied.insert(0, False) is also in aurora/sandbox/triage_metadata.py
• Add a property to the filters base class, correction_operation, default value is divide
• make sure the booleans update to the opposite polarity after calibration
  • This operation is done in ChannelTS.remove_instrument_response()
  • Correctly record which filters are removed in remove_instrument_response()
    • Not all filters are removed by default (time delay, and decimation may not be corrected)
    • Modify RemoveInstrumentResponse to allow control for include_decimation and include_time_delay
    • Figure out the best place to make list of bools to flip -- seems like complex_response method could return this, but there are 36 instances of complex_response in mt_metadata 2 in mth5, and an additional 12 in aurora :/. Instead, I added an optional argument to complex_response, that accepts a list of filters that will be corrected. This list is made by ChannelResponseFilter
• Assign the correct calibrated units to channel_ts after removing response. Note that according the the channel.json in standards, this should not be the abbreviation, but should be the full text string of the units.
• Clean the "update units" section of ChannlTS.remove_instrument_response
  • Add documentation explaining how the calibration unit are obtained
• Update tests to adhere to this convention
  Nice to have:
• Rename channel_response_filter.py to channel_response.py
• ChannelResponseFilter to ChannelResponse (25 occurrences in mth5, 14 in mt_metadata)
• Rename ChannelTS.channel_response_filter to ChannelTS.channel_response (86 occurrences in mth5)
• channel_response_filter.filter_list could be a dict instead, simply keyed by integers (list index) .. could be a list_dict, or box
  • Looked into this, but chose instead to clean up the management of list indices by making a clean, explicit, get_indices_of_filters_to_remove() method in ChannelResponse.

Still More:
While tweaking tests, it seems that units handling is still not 100% there
While ChannelTS metadata does have a units field, it doesn't interact with mt_metadata.utils.units

• add method to channel.py that returns the units object so that this can be compared with filter units in/out.

• add mixins from FilterBase to ChannelResponseFilter

  • these are not needed if we

• Manage setting "forward"/"inverse", "divide"/"multiply" attributes that defines their relationship in only one place.

  • this is done in FilterBase, and mixins are used when they are needed elsewhere

• The attributes direction (one of ["forward", "inverse"]) and the attribute application_operation in FilterBase
  maybe superfluous. We could probably do away with one of them (direction).

  • Ultimately we do not need any of these attributes, they were intended to replace the operation="divide" kwarg in ts_filters.RemoveInstrumentResponse.remove_instrument_response. Note that the filters do have "units_in", and "units_out", which makes them directional, and to the doc has now been added that they are by definition applied by multiplying in frequency domain. It would certainly be cleaner to remove any reference to filter "direction", "application_operation", "correction_operation". I will make a branch that does this and we can merge it in if we like.
  • Created fix_issue_173a branch and a corresponding branch in mth5. We need to decide if we will use fix_issue_173 or fix_issue_173a branch in the final PR.