Support custom mergeable metrics in whylogs

[andyndang]

There are three kinds of metrics that whylogs users track:

1. Tracking derived metrics from customers. Typically this is numerical data. You can use the approach above to track these metrics because they will show up as a “whylogs” column
2. Custom metrics that are mergeable: basically if you have metrics that can be “summed” or “aggregated” across different profiles, this is a feature request that we are tracking from other customers as well.
3. One-off metrics: sometimes users have one-off metrics that they want to piggy back on top of whylogs. These metrics are not aggregatable, but they want to use whylogs object to store these metrics.

[andyndang]

For 1, we're already doing this.

For 2, we'll need to support:

• Storing metrics (probably in binary form)
• A class that can handle the metric in Python
• A class that can handle the metric in Java
• These classes will implements method for serder and merging metrics. So we get something like obj.to_bytes(), obj.parse_bytes(), obj.merge(another). I believe these should be sufficient for most of the use cases

For 3, we can start with supporting numbers. But what's the behavior when merging two objects with "unmergeable" metrics? Should we throw exception? Warnings? Drop the fields?

[lalmei]

I would focus on a simple class custom metric api, like

class CustomMetric

   def track(inputs):
   def merge(self, right_metric):

if there is no merge then we cant merge, unless it inherents from something like numberTracker .

[andyndang]

That might work. We still need to:

• Decide to throw error/drop the metrics or not
• How to store this (so you'll need to convert it to bytes/from bytes)
• Store information about the class and the metrics for later parsing

[ramannanda9]

A base class should be able to handle serialization pretty easily.

@dataclass
class CustomMetric(abc.ABC):
    @abc.abstractmethod
    def track():
        pass
     @abc.abstracmethod
     def merge(self, right_metric: 'CustomMetric'):
          pass
     @abc.abstractmethod
     def name():
          pass
     def deserialize(name:str) -> 'CustomMetric'
           //implementation here by traversing subclasses of 'CustomMetric' and calling its constructor
     def serialize():
         return {"name": self.name, 'params': dataclasses.asdict(self)  }         

[lalmei]

except we need at least the protobuf de/serialization

[jamie256]

The protobuf message packing and then serialization is how our datasetprofile serializes all the associated metrics for a dataset when the logger writes, but I took your suggestion using a dataclass and doing most of that work in a base class. Here is a draft PR if you have comments: #300

[jamie256]

custom metrics participate in merging in v1:
https://github.com/whylabs/whylogs/blob/mainline/python/examples/basic/Merging_Profiles.ipynb
https://github.com/whylabs/whylogs/blob/dev/richard/custom-example/python/examples/extensions/Custom_Metrics.ipynb