feat(sql): table_writer_metrics() and memory_metrics() for better observability

[jerrinot]

It allows to query:

1. table writer metrics expose these: TOTAL_COMMITS, O3_COMMITS, ROLLBACKS, COMMITTED_ROWS, PHYSICALLY_WRITTEN_ROWS

2. memory utilization metrics: with a breakdown per tag.

For #1 to work metrics have to be enabled in QuestDB configuration. #2 is always enabled.

Questions I'd like reviewers to think about:

1. This is effectively API. Are we OK to maintain compatibility? Is it ok to break it? Imagine we remove a memory tag.
2. Should we enable metrics by default? I feel stats on O3 could be pretty useful for troubleshooting
3. each table emits just a single row. more over table metrics are reset on QuestDB restart? is that OK? Alternatively, we could expose memory metrics as tuples where each row would represent a (key-value) pair. This would allow simple sorting by memory consumption.

[jerrinot]

Please don't merge it yet, there is allocation going on due to auto-boxing. thanks to Vlad for spotting it!
edit: boxing is gone.

[puzpuzpuz]

• This is effectively API. Are we OK to maintain compatibility? Is it ok to break it? Imagine we remove a memory tag.

I'd say that we should clearly state in the docs that memory tags (and any metrics) are subject to change.

• Should we enable metrics by default? I feel stats on O3 could be pretty useful for troubleshooting

+1 from me for enabling metrics by default.

• each table emits just a single row. more over table metrics are reset on QuestDB restart? is that OK? Alternatively, we could expose memory metrics as tuples where each row would represent a (key-value) pair. This would allow simple sorting by memory consumption.

I'd suggest returning tuples instead of columns. This would make any integration simpler and less coupled with our tags and metrics. WDYT?

[jerrinot]

@puzpuzpuz I agree with everything you wrote! I will change the format to return tuples and later send a separate PR to enable metrics by default.

[jerrinot]

@puzpuzpuz can you please re-check?

[ideoma]

[PR Coverage check]

😍 pass : 91 / 94 (96.81%)

file detail

	path	covered line	new line	coverage
🔵	io/questdb/griffin/engine/table/StringLongTuplesRecordCursor.java	26	29	89.66%
🔵	io/questdb/cairo/TableWriterMetrics.java	4	4	100.00%
🔵	io/questdb/griffin/engine/table/TableWriterMetricsRecordCursorFactory.java	32	32	100.00%
🔵	io/questdb/griffin/engine/functions/table/MemoryMetricsFunctionFactory.java	3	3	100.00%
🔵	io/questdb/griffin/engine/table/MemoryMetricsRecordCursorFactory.java	22	22	100.00%
🔵	io/questdb/std/MemoryTag.java	1	1	100.00%
🔵	io/questdb/griffin/engine/functions/table/TableWriterMetricsFunctionFactory.java	3	3	100.00%

[ideoma]

I'd prefer a function for all prometheus metrics, all in one, not just writer and memory separately.