API for fieldname suggestion/completion

[markharwood]

Like the recently added terms_enum api, there is a need for clients like Kibana to help people complete the values they are typing. This requires a high-speed API to look up field names that match parts of what the user has already typed.

It's true to say that clients like Kibana typically already have ready access to lists of the physical fields in an index but the one omission is the potentially millions of "soft" field names introduced by the flattened field type which exist as document data in the index. Previously we had considered creating an API for listing just flattened field names but for practical purposes it would be convenient for clients if there was one elasticsearch API for autocompleting all field names, flattened or otherwise.

Usability concerns

The precise details of the API will depend to a large extent on the user experience clients want to provide and these are being discussed by members of the Kibana team (cc @elastic-jb ).
There are 2 main styles of matching - infix and prefix - either matching user input anywhere in the field name (infix) or completing the end of the field name being typed (prefix). Both have their merits-

• Infix matching could be useful to find any field name regardless of document structure e.g. searching for bytes might match the field names source.http.bytes_sent and target.dns.bytes. The user only cares about fields that mention bytes.
• Prefix matching could be useful to navigate using the structure of hierarchical documents e.g. ECS organises properties under several object fields so typing dns would match dns.answers and dns.id and further selecting dns.answers would reveal dns.answers.name and dns.answers.id.

This raises the question do users want to make use of the hierarchical structure of documents or ignore it completely?

Infix matching concerns

Infix matching is more expensive to run on flattened fields with millions of values so we will either not support it at all or curtail exploration after a max number of terms scanned or time taken

Prefix matching concerns

Is this a breadth-first match of matching values or depth-first? So in ECS would prefix completion of c suggest
a) breadth-first client cloud and container OR
b) depth-first client.address, client.bytes, client.domain etc?
With a) would the client only see values for client like client.address after they had typed the ending full stop e.g. client.? This is how completion works in a unix shell when typing field names in the command line - you only see the files in subdirectories when you put the / character at the end of whatever containing folder name e.g. typing D and using tab-completion will not suggest filenames from the Downloads or Documents folder until you type Downloads/ or Documents/. Clearly the client needs to know which fields represent objects or "leaf" fields with no subvalues. In the unix shell these suggestions are displayed differently (folders are always displayed with the trailing /)

Results format

Unlike the terms_enum api, returning simple strings with the matching names is probably not sufficient. We should probably return some type information too. Again, some input from UX teams would help here.

[elasticmachine]

Pinging @elastic/es-search (Team:Search)

[tsullivan]

Hi @markharwood I'm working on a project definition for field list hierarchies in Kibana, and was led to this issue. It looks like this new API will be useful in Kibana to provide a unified experience to allow users to quickly find a field they're looking for when it is in an object or flattened structure.

• Would it allow interactions with the *_range field types as well?
• Can you confirm that the nested field type doesn't intersect with this?

[javanna]

hey @tsullivan thanks for the ping. The work on this API has been suspended. We need to get back to it but we are not sure when we will get to it. It would be good to sync up and collect all the usecases for this new API, the needs and eventually re-prioritize this effort.

[kertal]

We discussed this in the team. We currently rely on having all fields available the doing filename suggestions. Here are 2 examples:

Adding filters:

Selecting breakdown fields:

The question is, having just a field suggestion/completion would not be sufficient. We would also need to be able to filter by type, and also we would need information about the capabilities of the field. Also helpful it would be to filter out fields without values. All this sounds like something we already have ... the field_caps API. Given we need more input/output options for completion, would there be any advantage in creating a dedicated API fro filename completion/suggestion? Or wouldn't it make more sense to further improve field_caps

[javanna]

Given we need more input/output options for completion, would there be any advantage in creating a dedicated API fro filename completion/suggestion?

Speed. Field caps is not made to provide real-time suggestions. Yet if we were to create a more responsive API targeted at suggestions, we would have to accept limited functionality to keep it fast.

[kertal]

Linking more issues with use cases @lukasolson identified that could benefit from this API:
elastic/kibana#24709
elastic/kibana#111642
elastic/kibana#96150