Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ES|QL] Distinguish among empty and available fields #174585

Merged
merged 35 commits into from Feb 2, 2024
Merged

[ES|QL] Distinguish among empty and available fields #174585

merged 35 commits into from Feb 2, 2024

Conversation

stratoula
Copy link
Contributor

@stratoula stratoula commented Jan 10, 2024 •
edited

Summary

Closes #174587

Implements available, empty fields separation in Discover ES|QL mode

image

The ES team added a query parameter ?drop_null_columns. This changes slightly the response:

all_columns: contain all columns with empty and non-empty values
columns: contain columns which have data

This PR is using this new query parameter in Discover to make the distinction among available and empty fields. This is very important in case of indices with many empty columns (for example ECS)

Checklist

This was referenced Jan 10, 2024
Merged
Closed
@stratoula
Copy link
Contributor Author

stratoula commented Jan 23, 2024 •
edited

I will continue working on it after the async search PR has been merged #174246

@stratoula
Copy link
Contributor Author

/ci

@stratoula
Copy link
Contributor Author

/ci

@stratoula stratoula added Feature:Discover Discover Application enhancement New value added to drive a business result Team:DataDiscovery Discover App Team (Document Explorer, Saved Search, Surrounding documents, Graph) Feature:ES|QL v8.13.0 labels Feb 1, 2024
jughosta
jughosta approved these changes Feb 2, 2024
View reviewed changes
Copy link
Contributor

@jughosta jughosta left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍 Thanks!

src/plugins/data/server/search/strategies/esql_async_search/esql_async_search_strategy.ts Outdated Show resolved Hide resolved
src/plugins/data/common/search/expressions/esql.ts Outdated Show resolved Hide resolved
@stratoula
Copy link
Contributor Author

/ci

@stratoula
Copy link
Contributor Author

/ci

@kibana-ci
Copy link
Collaborator

💚 Build Succeeded

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id before after diff
@kbn/es-types 30 31 +1
@kbn/field-utils 29 30 +1
expressions 1757 1760 +3
total +5

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
cloudSecurityPosture 399.2KB 399.2KB -6.0B
discover 569.2KB 569.3KB +111.0B
lens 1.4MB 1.4MB +77.0B
logsExplorer 979.0KB 979.0KB -6.0B
total +176.0B

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id before after diff
data 415.2KB 415.6KB +381.0B
dataViews 48.3KB 48.4KB +46.0B
total +427.0B
Unknown metric groups

API count

id before after diff
@kbn/es-types 30 31 +1
@kbn/field-utils 37 38 +1
data 3234 3235 +1
dataViews 938 940 +2
expressions 2219 2222 +3
total +8

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @stratoula

@stratoula stratoula merged commit 5104183 into elastic:main Feb 2, 2024
19 checks passed
@kibanamachine kibanamachine added the backport:skip This commit does not require backporting label Feb 2, 2024
@dej611 dej611 mentioned this pull request Feb 2, 2024
Merged
@nreese nreese mentioned this pull request Feb 5, 2024
Closed
WafaaNasr pushed a commit to WafaaNasr/kibana that referenced this pull request Feb 6, 2024
@stratoula @WafaaNasr
[ES|QL] Distinguish among empty and available fields (elastic#174585)
a58f2c2
dej611 added a commit that referenced this pull request Feb 6, 2024
@dej611 @stratoula @jughosta
[ES|QL] Reduce number of iterations on null columns (#176174)
24d3619 
## Summary

Follow up on #174585

Less code, less iterations.

---------

Co-authored-by: Stratoula Kalafateli <efstratia.kalafateli@elastic.co>
Co-authored-by: Stratoula Kalafateli <stratoula1@gmail.com>
Co-authored-by: Julia Rechkunova <julia.rechkunova@gmail.com>
fkanout pushed a commit to fkanout/kibana that referenced this pull request Feb 7, 2024
@stratoula @fkanout
[ES|QL] Distinguish among empty and available fields (elastic#174585)
2cdcb19
fkanout pushed a commit to fkanout/kibana that referenced this pull request Feb 7, 2024
@dej611 @stratoula
@jughosta @fkanout
[ES|QL] Reduce number of iterations on null columns (elastic#176174)
fd64efa 
## Summary

Follow up on elastic#174585

Less code, less iterations.

---------

Co-authored-by: Stratoula Kalafateli <efstratia.kalafateli@elastic.co>
Co-authored-by: Stratoula Kalafateli <stratoula1@gmail.com>
Co-authored-by: Julia Rechkunova <julia.rechkunova@gmail.com>
CoenWarmer pushed a commit to CoenWarmer/kibana that referenced this pull request Feb 15, 2024
@stratoula @CoenWarmer
[ES|QL] Distinguish among empty and available fields (elastic#174585)
2406555
CoenWarmer pushed a commit to CoenWarmer/kibana that referenced this pull request Feb 15, 2024
@dej611 @stratoula
@jughosta @CoenWarmer
[ES|QL] Reduce number of iterations on null columns (elastic#176174)
7440039 
## Summary

Follow up on elastic#174585

Less code, less iterations.

---------

Co-authored-by: Stratoula Kalafateli <efstratia.kalafateli@elastic.co>
Co-authored-by: Stratoula Kalafateli <stratoula1@gmail.com>
Co-authored-by: Julia Rechkunova <julia.rechkunova@gmail.com>
CoenWarmer pushed a commit to CoenWarmer/kibana that referenced this pull request Feb 15, 2024
@stratoula @CoenWarmer
[ES|QL] Distinguish among empty and available fields (elastic#174585)
c32d7ea
CoenWarmer pushed a commit to CoenWarmer/kibana that referenced this pull request Feb 15, 2024
@dej611 @stratoula
@jughosta @CoenWarmer
[ES|QL] Reduce number of iterations on null columns (elastic#176174)
4d2918d 
## Summary

Follow up on elastic#174585

Less code, less iterations.

---------

Co-authored-by: Stratoula Kalafateli <efstratia.kalafateli@elastic.co>
Co-authored-by: Stratoula Kalafateli <stratoula1@gmail.com>
Co-authored-by: Julia Rechkunova <julia.rechkunova@gmail.com>
fkanout pushed a commit to fkanout/kibana that referenced this pull request Mar 4, 2024
@stratoula @fkanout
[ES|QL] Distinguish among empty and available fields (elastic#174585)
ae04101
fkanout pushed a commit to fkanout/kibana that referenced this pull request Mar 4, 2024
@dej611 @stratoula
@jughosta @fkanout
[ES|QL] Reduce number of iterations on null columns (elastic#176174)
4668932 
## Summary

Follow up on elastic#174585

Less code, less iterations.

---------

Co-authored-by: Stratoula Kalafateli <efstratia.kalafateli@elastic.co>
Co-authored-by: Stratoula Kalafateli <stratoula1@gmail.com>
Co-authored-by: Julia Rechkunova <julia.rechkunova@gmail.com>
@jughosta jughosta mentioned this pull request Mar 14, 2024
Merged
jughosta added a commit that referenced this pull request Mar 15, 2024
@jughosta
[ES|QL][Discover] Show all non-empty values in the grid (#178692)
4ae02b0 
- Followup for #174585

## Summary

This PR hides only `null`, `undefined` values and keeps `false`, `0`,
`(empty)`.

Before:
<img width="500" alt="Screenshot 2024-03-14 at 10 07 13"
src="https://github.com/elastic/kibana/assets/1415710/c5d49362-09e3-48f2-9ecc-247560950d68">

After:
<img width="500" alt="Screenshot 2024-03-14 at 10 06 25"
src="https://github.com/elastic/kibana/assets/1415710/3c1adcfd-bec9-4b4a-b94f-78b1d916e9dd">
kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Mar 15, 2024
@jughosta
[ES|QL][Discover] Show all non-empty values in the grid (elastic#178692)
1defb82 
- Followup for elastic#174585

## Summary

This PR hides only `null`, `undefined` values and keeps `false`, `0`,
`(empty)`.

Before:
<img width="500" alt="Screenshot 2024-03-14 at 10 07 13"
src="https://github.com/elastic/kibana/assets/1415710/c5d49362-09e3-48f2-9ecc-247560950d68">

After:
<img width="500" alt="Screenshot 2024-03-14 at 10 06 25"
src="https://github.com/elastic/kibana/assets/1415710/3c1adcfd-bec9-4b4a-b94f-78b1d916e9dd">

(cherry picked from commit 4ae02b0)
kibanamachine added a commit that referenced this pull request Mar 15, 2024
@kibanamachine @jughosta
[8.13] [ES|QL][Discover] Show all non-empty values in the grid (#178692…
9473042 
…) (#178815)

# Backport

This will backport the following commits from `main` to `8.13`:
- [[ES|QL][Discover] Show all non-empty values in the grid
(#178692)](#178692)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Julia
Rechkunova","email":"julia.rechkunova@elastic.co"},"sourceCommit":{"committedDate":"2024-03-15T12:14:17Z","message":"[ES|QL][Discover]
Show all non-empty values in the grid (#178692)\n\n- Followup for
#174585
Summary\r\n\r\nThis PR hides only `null`, `undefined` values and keeps
`false`, `0`,\r\n`(empty)`.\r\n\r\nBefore:\r\n<img width=\"500\"
alt=\"Screenshot 2024-03-14 at 10 07
13\"\r\nsrc=\"https://github.com/elastic/kibana/assets/1415710/c5d49362-09e3-48f2-9ecc-247560950d68\">\r\n\r\nAfter:\r\n<img
width=\"500\" alt=\"Screenshot 2024-03-14 at 10 06
25\"\r\nsrc=\"https://github.com/elastic/kibana/assets/1415710/3c1adcfd-bec9-4b4a-b94f-78b1d916e9dd\">","sha":"4ae02b02b918f42244d35482b51b506d18ac8e18","branchLabelMapping":{"^v8.14.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","Team:DataDiscovery","backport:prev-minor","Feature:ES|QL","v8.14.0"],"title":"[ES|QL][Discover]
Show all non-empty values in the
grid","number":178692,"url":"#178692
Show all non-empty values in the grid (#178692)\n\n- Followup for
#174585
Summary\r\n\r\nThis PR hides only `null`, `undefined` values and keeps
`false`, `0`,\r\n`(empty)`.\r\n\r\nBefore:\r\n<img width=\"500\"
alt=\"Screenshot 2024-03-14 at 10 07
13\"\r\nsrc=\"https://github.com/elastic/kibana/assets/1415710/c5d49362-09e3-48f2-9ecc-247560950d68\">\r\n\r\nAfter:\r\n<img
width=\"500\" alt=\"Screenshot 2024-03-14 at 10 06
25\"\r\nsrc=\"https://github.com/elastic/kibana/assets/1415710/3c1adcfd-bec9-4b4a-b94f-78b1d916e9dd\">","sha":"4ae02b02b918f42244d35482b51b506d18ac8e18"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v8.14.0","branchLabelMappingKey":"^v8.14.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/178692","number":178692,"mergeCommit":{"message":"[ES|QL][Discover]
Show all non-empty values in the grid (#178692)\n\n- Followup for
#174585
Summary\r\n\r\nThis PR hides only `null`, `undefined` values and keeps
`false`, `0`,\r\n`(empty)`.\r\n\r\nBefore:\r\n<img width=\"500\"
alt=\"Screenshot 2024-03-14 at 10 07
13\"\r\nsrc=\"https://github.com/elastic/kibana/assets/1415710/c5d49362-09e3-48f2-9ecc-247560950d68\">\r\n\r\nAfter:\r\n<img
width=\"500\" alt=\"Screenshot 2024-03-14 at 10 06
25\"\r\nsrc=\"https://github.com/elastic/kibana/assets/1415710/3c1adcfd-bec9-4b4a-b94f-78b1d916e9dd\">","sha":"4ae02b02b918f42244d35482b51b506d18ac8e18"}}]}]
BACKPORT-->

Co-authored-by: Julia Rechkunova <julia.rechkunova@elastic.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dej611 dej611 dej611 approved these changes

@jughosta jughosta jughosta approved these changes

@miltonhultgren miltonhultgren miltonhultgren approved these changes

@jloleysens jloleysens jloleysens approved these changes

Assignees

@stratoula stratoula

Labels
backport:skip This commit does not require backporting Feature:Discover Discover Application Feature:ES|QL Feature:ExpressionLanguage Interpreter expression language (aka canvas pipeline) release_note:enhancement Team:DataDiscovery Discover App Team (Document Explorer, Saved Search, Surrounding documents, Graph) v8.13.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[ES|QL] [Discover] Distinction betweem Available and empty fields
8 participants
@stratoula @elasticmachine @kibana-ci @dej611 @jughosta @miltonhultgren @jloleysens @kibanamachine