Proposal: Introduce metrics reporting for lance

[yanghua]

As a table format, Lance plays a crucial role in data storage and management for AI data. A comprehensive metrics system is essential in monitoring table formats, diagnosing health status, and performance-oriented governance. This is fully reflected in Apache Iceberg.

Apache Iceberg introduced metrics reporting1. It exposes some metadata and statistical information of core table operations (such as Scan/Commit) in the form of metrics. This ensures that these core pieces of information are not hidden beneath the iceberg.

In terms of its uses, with the external exposure of these metrics, we can do many things.

• Monitoring: Includes some indicators related to load and performance. Third parties can provide some insightful dashboards based on these indicators 24;
• Diagnosis: Based on monitoring, we can provide some incremental value. One of them is the diagnosis of some performance issues, which will assist in optimizing layout or performance;
• Optimization: Based on the provided indicators, we can optimize performance at both the read and write ends.

[yanghua]

@eddyxu @westonpace @wjones127 Would like to listen to your thoughts and feedback.

[westonpace]

I think we're very much in favor of adding additional metrics. I had an initial start at this here; #3075 but I'm not sure if I'll get back to it soon. I was planning on adding some storage statistics soon as well.

I guess this issue itself is a little broad though. Is there a particular standard you are looking for? Is this something you are hoping to work on?

[yanghua]

Hi @westonpace Thanks for your feedback.

I guess this issue itself is a little broad though.

Yes, this proposal is on "idea" status currently. If the community gives positive feedback, I'd like to refine it.

Is there a particular standard you are looking for?

Actually, I am just looking for some thoughts and feedback or the status(if the community has kicked off).

I will take a look about #3075

Is this something you are hoping to work on?

Yes, I would like to give some help at least on two parts:

• introduce the framework for metrics reporting;
• enrich the statistics and metrics in some core actions on the Lance dataset, e.g. scan/take/commit..

[jaystarshot]

Maybe we can integrate with prometheus to start with?

[wjones127]

Two places I remember we have existing metrics are:

1. The IO metrics on a dataset’S object store
2. The cache metrics on a session

Those might be good examples to start with.

One key question I have is how do we decide what are the dimensions/labels of the metrics?

It would be nice if the metrics were somehow decoupled from any particular framework. Not sure if Prometheus the most generic protocol, but it is widely supported if I remember correctly. Opentelemetry might have a metrics protocol, but not sure if that’s more widely supported than Prometheus.

In LanceDB enterprises we create custom adapters between these custom metrics and metrics frameworks. For example, I think we just have a Tokio task that polls cache metrics and publishes them to a promethus endpoint.

[jaystarshot]

One key question I have is how do we decide what are the dimensions/labels of the metrics?

Thanks @wjones127 I think to start with we can first just define the metrics as static objects which only register on startup eg: {name=scan_req_counter, type=LANCE_COUNT, ...}. Then those objects could have labels at runtime.

We can have a metric framework and an exporter interface which can be consumed by multiple different backgrounds. The metric definition would be defined in lance and other backgrounds can translate that to their supported data backends.

Eg: LANCE_COUNT, LANCE_AGG, LANCE_AVG

In LanceDB enterprises we create custom adapters between these custom metrics and metrics frameworks. For example, I think we just have a Tokio task that polls cache metrics and publishes them to a promethus endpoint.

Thanks a lot! yes i think this is great to start!