LogAttribute parameters autocomplete is not working

[czephyr]

It looks like autocomplete for attribute names is not working right.
Some fields in LogAttributes will never get suggested for search and will never appear in the available filters even when the number of filters to fetch is high (its 1000 in the example).

Example: A log has the attribute "organization" in its LogAttribute structure.

The autocomplete is not suggesting the "organization" attribute.

The filters for LogAttribute show a very small subset of all the LogAttribute attributes even when the amount of filters to be fetched is 1000.
The "organization" attribute is not there and also all the other _usage params are not there.

[czephyr]

Possibly #2128 will help

[knudtty]

That PR will get us part of the way there, I have a follow on planned that will enable those MVs to be used also for filters.

Introspection queries are really intensive, it's actually a quite difficult problem. MVs should trivialize the problem of fetching all values as part of a set. Subsets may still be difficult, but we'll introduce a toggle that allows you to still surface the "exact" values.

[czephyr]

Great to see there is interest in making this work better. Thanks for the good work.

[czephyr]

Tracking #2297 and #2376 as relevant

[knudtty]

Hey @czephyr, wanted to let you know that there are some MVs you could add today to assist. It'll require you running some SQL on your CH instance and modifying your source in your team settings.

The file can be found here. #2128 is where the MVs were added, there's a video showing how you can add to the source.

[czephyr]

I've added the MVs to my deployment making them ReplicatedSummingMergeTree and pointing the HDX metadata tables to the distributed version of them.

I'm running into this error:

Code: 70. DB::Exception: Received from localhost:9000. DB::Exception: Conversion from AggregateFunction(groupUniqArray(20), String) to AggregateFunction(groupUniqArray(20), String) is not supported: While executing Remote. (CANNOT_CONVERT_TYPE)

It looks like HDX is trying to execute this:

SELECT
    ColumnIdentifier,
    Key,
    groupUniqArray(_CAST(20, 'Int32'))(Value) AS Values
FROM logs.distributed_otel_logs_kv_rollup_15m
WHERE Value != ''
GROUP BY
    ColumnIdentifier,
    Key
ORDER BY
    ColumnIdentifier = 'NativeColumn' DESC,
    ColumnIdentifier ASC,
    Key ASC
LIMIT _CAST(30, 'Int32')

By removing the cast to groupUniqArray(20)(Value) AS Values this query works.