[Metrics] Provide a metric for consumer lag

[polyzos]

Search before asking

• I searched in the issues and found nothing similar.

Motivation

Currently, Fluss provides currentFetchEventTimeLag, currentEmitEventTimeLag, and pendingRecords metrics in the Flink source.

However, it doesn't support out of the box a metric for consumer offset (similar to the consumer lag in Kafka), which can be useful to users.

For reference: https://www.redpanda.com/guides/kafka-performance-kafka-consumer-lag

Solution

We should support this metric in Fluss so that all the clients/connectors can leverage it.

Anything else?

No response

Willingness to contribute

• I'm willing to submit a PR!

[sauravjaiswalsj]

Can I work on this issue?

[polyzos]

@sauravjaiswalsj Thank you for your interest 🙏 Assigned to you

[wuchong]

Thanks so much, @sauravjaiswalsj, for taking this on!

That said, this isn’t a beginner-friendly task (I just removed the good first issue tag). It will likely require a series of Fluss Improvement Proposals (FIPs) to properly design and implement. For example:

• Introducing internal system tables to manage metadata,
• Implementing consumer offsets using those system tables,
• And then building consumer lag metrics on top of the offset tracking.

Familiarity with how Kafka implements consumer offsets (e.g., via internal __consumer_offsets topics) would be extremely helpful, as we’ll need a similarly robust and scalable mechanism—but adapted to Fluss’s architecture and Lakehouse-native design.

[polyzos]

@wuchong thanks for jumping in and addressing this.. seems like i did some serious underestimation about the work required 😅

[sauravjaiswalsj]

@wuchong, that's really helpful. I am still keen to solve this issue.

[wuchong]

Thank you @sauravjaiswalsj !

Let me know if you need any help or if you’d like to collaborate on drafting the first FIP!

[vamossagar12]

hi, @sauravjaiswalsj , are you still working on this? If not, can I take this up?