[Model Monitoring] Add application metrics to V3IO tables and endpoints #5585
Conversation
There was a problem hiding this comment.
LGTM, I didn't run the tests.
I had few suggestions but in general 2 important comments:
1 - try to add more documentation. Even 1-2 lines can be super useful to understand the importance of a certain function.
2 - TBH I still not sure why not implementing these changes on top of the V3IO Connector in this PR. As long as we are using the v3io_tsdb_reader, we have to import v3io resources into the model endpoints API file, which in other words means that it is probably breaks the CE flow + anyway we will have to "rewrite" this logic on top of the abstraction, so why not now?
| if kind == mm_constants.WriterEventKind.METRIC: | ||
| # TODO : Implement the logic for writing metrics to V3IO TSDB | ||
| return | ||
| if kind == mm_schemas.WriterEventKind.METRIC: |
There was a problem hiding this comment.
As I wrote above, I would suggest to wrap this part in a different func as well (including the initialization of the index_cols_base.
|
Implements ML-6430.
Write the application metrics to V3IO KV and TSDB.
The KV store is per model endpoint, as done for results. Each key is per application + metric name, creating a schema that Grafana can consume.
The TSDB is implemented in its destined "metrics" table.
Support reading the metrics through the existing
/metricsand/metrics-valuesAPI endpoints.The testing is through some new unit tests and an extended test part in the primary system test.