Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ES|QL] generate function docs and definitions for agg functions #107219

Open
drewdaemon opened this issue Apr 8, 2024 · 2 comments
Open

[ES|QL] generate function docs and definitions for agg functions #107219

drewdaemon opened this issue Apr 8, 2024 · 2 comments
Assignees
@ivancea
Labels
:Analytics/ES|QL AKA ESQL >enhancement Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo)

Comments

@drewdaemon
Copy link

Description

With #106782, we will get function definitions for eval functions. We'd also like them for stats functions.
@drewdaemon drewdaemon added >enhancement Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) needs:triage Requires assignment of a team area label :Analytics/ES|QL AKA ESQL labels Apr 8, 2024
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-analytical-engine (Team:Analytics)

@elasticsearchmachine elasticsearchmachine removed the needs:triage Requires assignment of a team area label label Apr 8, 2024
@drewdaemon drewdaemon mentioned this issue May 21, 2024
Merged
2 tasks
drewdaemon added a commit to elastic/kibana that referenced this issue May 24, 2024
@drewdaemon @stratoula @kibanamachine
[ES|QL] Generate function definitions from Elasticsearch metadata (#1…
72d2a56 
…79584)

## Summary

Resolve #179634

This PR introduces a script to gather metadata from Elasticsearch and
build the function definitions for Kibana's client-side validation.

To run the script: `cd packages/kbn-esql-validation-autocomplete && yarn
makedefs path/to/elasticsearch/repo`

## Current limitations
### Things we can't (yet) gather from ES

#### Things that are currently being filled in on the Kibana side but
would be missing if we had to rely completely on ES
- function aliases (e.g. `to_str`)
- operators and aggs (issues for Elasticsearch
elastic/elasticsearch#107219 and
elastic/elasticsearch#107220)
- Kibana AST parameter settings
- `constantOnly` — whether a parameter can be a non-literal or not
(`percentile`, `auto_bucket`)
- `constantOptions`/`constantSuggestions` — specific information about
which constants are accepted or suggested for particular parameters.
- `supportsWildcard` — whether a parameter can contain `*` (only `count`
at the moment)
- `noNestingFunctions` — whether a parameter can contain nested
functions (used for all agg functions)
- Kibana date constant types
  - `chrono_literal`
  - `time_literal`

### Other considerations
- Asciidoc links need to be scrubbed (e.g. `Function
{wikipedia}/foo/bar[arccosine] returns blah blah`)
- As of elastic/elasticsearch#106454 some
descriptions reference outdated parameter names and some parameter names
are questionable:
<img width="600" alt="Screenshot 2024-03-28 at 1 07 16 PM"
src="https://github.com/elastic/kibana/assets/315764/3b9ab457-e9fd-4c5f-afb2-23ae769bb44a">

_parameter `n` is now called `number`, but the description still calls
it `n`... also, is `number` really the best name for the variable?_

### Checklist

- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios

---------

Co-authored-by: Stratoula Kalafateli <efstratia.kalafateli@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
rshen91 pushed a commit to rshen91/kibana that referenced this issue May 30, 2024
@drewdaemon @stratoula
@kibanamachine @rshen91
[ES|QL] Generate function definitions from Elasticsearch metadata (el…
ca2f1ca 
…astic#179584)

## Summary

Resolve elastic#179634

This PR introduces a script to gather metadata from Elasticsearch and
build the function definitions for Kibana's client-side validation.

To run the script: `cd packages/kbn-esql-validation-autocomplete && yarn
makedefs path/to/elasticsearch/repo`

## Current limitations
### Things we can't (yet) gather from ES

#### Things that are currently being filled in on the Kibana side but
would be missing if we had to rely completely on ES
- function aliases (e.g. `to_str`)
- operators and aggs (issues for Elasticsearch
elastic/elasticsearch#107219 and
elastic/elasticsearch#107220)
- Kibana AST parameter settings
- `constantOnly` — whether a parameter can be a non-literal or not
(`percentile`, `auto_bucket`)
- `constantOptions`/`constantSuggestions` — specific information about
which constants are accepted or suggested for particular parameters.
- `supportsWildcard` — whether a parameter can contain `*` (only `count`
at the moment)
- `noNestingFunctions` — whether a parameter can contain nested
functions (used for all agg functions)
- Kibana date constant types
  - `chrono_literal`
  - `time_literal`

### Other considerations
- Asciidoc links need to be scrubbed (e.g. `Function
{wikipedia}/foo/bar[arccosine] returns blah blah`)
- As of elastic/elasticsearch#106454 some
descriptions reference outdated parameter names and some parameter names
are questionable:
<img width="600" alt="Screenshot 2024-03-28 at 1 07 16 PM"
src="https://github.com/elastic/kibana/assets/315764/3b9ab457-e9fd-4c5f-afb2-23ae769bb44a">

_parameter `n` is now called `number`, but the description still calls
it `n`... also, is `number` really the best name for the variable?_

### Checklist

- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios

---------

Co-authored-by: Stratoula Kalafateli <efstratia.kalafateli@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
@drewdaemon drewdaemon mentioned this issue May 31, 2024
Open
@drewdaemon drewdaemon mentioned this issue Jun 26, 2024
Merged
@nik9000
Copy link
Member

nik9000 commented Jul 5, 2024 •
edited by ivancea
Loading

The last person who touched this was @ivancea. Because I asked him to. But I think that means he's probably the right person to officially assign here. I think this is basically shaped like:

Phase 1 - Infrastructure (@ivancea )

  • Build a basic test class to run some aggs tests, adapting concepts from AbstractFunctionTestCase.
  • Expand that basic test case for
    • randomized inputs
    • multiphased execution (INITIAL/FINAL, INITIAL/INTERMEDIATE/FINAL, and SINGLE or whatever we call it)
    • type errors
  • apply the basic test case for a hand full of aggs to prove it

Phase 2 - Apply all the places(shared)

  • AVG
  • COUNT
  • COUNT_DISTINCT
  • MAX
  • MEDIAN
  • MEDIAN_ABSOLUTE_DEVIATION
  • MIN
  • PERCENTILE
  • ST_CENTROID_AGG
  • SUM
  • TOP
  • VALUES

once this phase is really happening @drewdaemon can start picking up the generated files and integrate them into Kibana's editor

Phase 3 - Only semi-related

  • Infrastructure for testing surrogate expressions

@drewdaemon drewdaemon mentioned this issue Jul 16, 2024
Merged
jbudz pushed a commit to elastic/kibana that referenced this issue Jul 17, 2024
@drewdaemon
[ES|QL] temporarily exclude aggs from inline docs script (#188473)
dce0ba1 
## Summary

As ES has rolled out partial support for generating agg function
definitions and documentation, they have become mixed into the existing
scalar function documentation section. This PR excludes them for now.
Once elastic/elasticsearch#107219 is complete,
we will unskip them and turn on full automation!
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ivancea ivancea

Labels
:Analytics/ES|QL AKA ESQL >enhancement Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo)
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@nik9000 @drewdaemon @ivancea @elasticsearchmachine