[Lens] Enables text-based languages (SQL) #140469

stratoula · 2022-09-12T07:51:17Z

Summary

Enables text based query languages in Lens.

This is the initial PR so many features are not included. It's primary goal is to set up the architecture of the text based languages in Lens. Things to have in mind:

You can create only one layer. There are some UI limitations on supporting multiple layers that we are discussing. This is going to work on the future
Moving from one datasource to another cleanups the unsaved state
All the data manipulation is now happening from the text based language. For this reason we allow only chart-specific configuration
You can't create filters by clicking on a chart
As the data manipulation is done by the query there are some visualizations that might seem "weird" depending on the query. We want to improve the UX here by adding some warnings in the future.
Discover to Lens and the opposite direction is not supported for now.
The suggestions can be improved but this is not on the scope of this PR.

How to test

The feature is hidden under an advanced setting. To enable it go to advanced settings and switch on the "Enable SQL" setting. If you navigate to Lens and open the dataview picker you will see a new entry

Flaky runners

https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/1303 25 times
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/1304 100 times

Checklist

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Unit or functional tests were updated or added to match the most common scenarios
Any UI touched in this PR is usable by keyboard only (learn more about keyboard accessibility)
Any UI touched in this PR does not create any new axe failures (run axe in browser: FF, Chrome)
This renders correctly on smaller devices using a responsive layout. (You can test this in your browser)
This was checked for cross-browser compatibility

stratoula · 2022-09-13T11:53:20Z

src/plugins/discover/server/ui_settings.ts

@@ -315,7 +315,7 @@ export const getUiSettings: (docLinks: DocLinksServiceSetup) => Record<string, U
    value: false,
    description: i18n.translate('discover.advancedSettings.enableSQLDescription', {
      defaultMessage:
-        '{technicalPreviewLabel} This tech preview feature is highly experimental--do not rely on this for production saved searches or dashboards. This setting enables SQL as a text-based query language in Discover. If you have feedback on this experience please reach out to us on {link}',
+        '{technicalPreviewLabel} This tech preview feature is highly experimental--do not rely on this for production saved searches, visualizations or dashboards. This setting enables SQL as a text-based query language in Discover and Lens. If you have feedback on this experience please reach out to us on {link}',


ℹ️ We are using the same switch with discover. I have slightly changed the text to include Lens. The setting is still on the Discover section.

stratoula · 2022-09-13T11:54:15Z

src/plugins/unified_search/public/dataview_picker/change_dataview.tsx

@@ -262,7 +271,7 @@ export function ChangeDataView({
              setIsTextBasedLangSelected(false);
              // clean up the Text based language query
              onTextLangQuerySubmit?.({
-                language: 'kql',
+                language: 'kuery',


ℹ️ This is called on cleanup (i.e. when navigating from the text based languages mode to the dataviewMode. I was wrongly setting the language to kql while it is kuery

stratoula · 2022-09-13T11:58:19Z

x-pack/plugins/lens/public/editor_frame_service/editor_frame/data_panel_wrapper.tsx

  return (
    <>
      <Easteregg query={externalContext?.query} />
-      {Object.keys(props.datasourceMap).length > 1 && (


ℹ️ This was adding a popover for changing datasources (in case of more than one). In this PR we are introducing a new datasource so this begun to be displayed but we don't want it. The datasource change is done by the dataview picker.

elasticmachine · 2022-09-23T13:19:05Z

Pinging @elastic/kibana-vis-editors @elastic/kibana-vis-editors-external (Team:VisEditors)

spalger

Bundle bump LGTM

…o sql-in-lens

flash1293 · 2022-09-26T10:34:05Z

x-pack/plugins/lens/public/visualizations/legacy_metric/visualization.tsx

@@ -158,7 +159,7 @@ const toExpression = (
          showLabels: [!attributes?.mode || attributes?.mode === 'full'],
          colorMode: !canColor ? [ColorMode.None] : [state?.colorMode || ColorMode.None],
          autoScale: [true],
-          colorFullBackground: [true],
+          colorFullBackground: [!isTextBasedLanguage],


If this is because of the possibility of multiple metrics being rendered, then I think it's fine to just ignore it.

@flash1293 I did this, because when this is set to true I get this error

Any suggestions here?

IMHO that's alright for now (especially as it's only the legacy vis)

Done 61b1af1

flash1293

On dashboards it doesn't reload when changing the time range (it does reload when changing filters though)

stratoula · 2022-09-26T15:19:20Z

@flash1293 timeRange error fixed! Thanx for the great help!

I have also removed the cleaning up of the layers on easy query update. You can see how it works now here:

ppisljar

app services code LGTM

kertal

DataDiscovery.team owned code LGTM, just a function and a Advanced setting was renamed. SQL in Discover works as expected

flash1293 · 2022-09-27T08:42:02Z

x-pack/plugins/lens/public/text_based_languages_datasource/text_based_languages.tsx

+      const layer = props.state.layers[props.layerId];
+      const selectedField = layer?.allColumns?.find(
+        (column) => column.columnId === props.columnId
+      )!;


the selected field might be not available, we should handle this case by rendering a "FIELD MISSING" label or something like that.

Done 309b68c

flash1293

This works pretty well now - just found one issue

flash1293

Tested again and everything works fine. Thanks for the change with not dropping the config on query change, it works like a charm!

stratoula · 2022-09-27T10:12:53Z

Thanx for your comments Joe!

kibana-ci · 2022-09-27T11:32:32Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: 7fdf2c3

Failed CI Steps

Rules, Alerts and Exceptions ResponseOps Cypress Tests on Security Solution

Test Failures

[job] [logs] Rules, Alerts and Exceptions ResponseOps Cypress Tests on Security Solution / Alerts detection rules table auto-refresh should disable auto refresh when any rule selected and enable it after rules unselected

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`data`	518	519	+1
`lens`	908	917	+9
total			+10

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`data`	2508	2509	+1
`lens`	558	560	+2
total			+3

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`discover`	477.4KB	477.4KB	+13.0B
`lens`	1.2MB	1.3MB	+16.5KB
`unifiedSearch`	260.8KB	260.9KB	+80.0B
total			+16.6KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`data`	432.2KB	432.5KB	+306.0B
`lens`	35.4KB	35.6KB	+275.0B
`unifiedSearch`	47.0KB	47.1KB	+66.0B
total			+647.0B

Unknown metric groups

API count

id	before	after	diff
`data`	3211	3213	+2
`lens`	646	649	+3
total			+5

History

💛 Build #75709 was flaky 1392c48
💚 Build #75606 succeeded 660d0e8
💚 Build #75543 succeeded becd0a6
💚 Build #75486 succeeded aa51f96
💛 Build #75187 was flaky 948e5ea
💚 Build #75135 succeeded 438f375

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

* [Lens] Enable text-based languages-sql * Display data * Chart switcher and further fixes * Drag and drop fixes * Small fix * Multiple improvements * Errors implementation and save and exit * Some cleanup * Fix types failures * Revert change * Fix underlying data error * Fix jest test * Fixes app test * Rename datasource to textBased * display the dataview picker component but disabled * Fix functional test * Refactoring * Populate the dataview to theembeddable * Load sync * sync load of the new dtsource * Fix * Fix bug when the dtaview is not found * Refactoring * Add some unit tests * Add some unit tests * Add more unit tests * Add a functional test * Add all files * Update lens limit * Fixes bug * Bump lens size * Fix flakiness * Further fixes * Fix check * More fixes * Fix test * Wait for query to run * More changes * Fix * Fix the function input to fetch from variable * Remove the colorFullBackground check * Fix dashboard bug when timeRange changes * Remove the clear on query update, show column error instead * Render a field missing label in case the label is not defined * Fix Co-authored-by: Joe Reuter <johannes.reuter@elastic.co>

stratoula added 13 commits September 1, 2022 11:44

[Lens] Enable text-based languages-sql

6dfd44c

Merge branch 'main' into sql-in-lens

d490bdb

Display data

df8f932

Merge branch 'main' into sql-in-lens

5f75342

Chart switcher and further fixes

44cbc8f

Drag and drop fixes

06ab19f

Merge branch 'main' into sql-in-lens

34fbd15

Small fix

469ed15

Multiple improvements

4ee6392

Errors implementation and save and exit

e250b93

Some cleanup

3594b15

Fix types failures

67c0784

Revert change

e99d722

stratoula changed the title ~~Sql in lens~~ [Lens] Enables text-based languages (SQL) Sep 12, 2022

stratoula added 7 commits September 12, 2022 12:32

Merge with main and resolve conflicts

65ff394

Fix underlying data error

6fb0465

Fix jest test

a437fdb

Merge branch 'main' into sql-in-lens

d776d87

Fixes app test

19909de

Rename datasource to textBased

6a6344c

display the dataview picker component but disabled

e0204f1

stratoula commented Sep 13, 2022

View reviewed changes

stratoula added 6 commits September 14, 2022 10:10

Fix functional test

58238c0

Refactoring

95d2d24

Populate the dataview to theembeddable

887a81f

Load sync

88831fb

sync load of the new dtsource

f69b75d

Fix

60f499c

stratoula marked this pull request as ready for review September 23, 2022 13:19

stratoula requested review from a team as code owners September 23, 2022 13:19

spalger approved these changes Sep 23, 2022

View reviewed changes

stratoula and others added 4 commits September 26, 2022 09:00

Merge branch 'main' into sql-in-lens

16f9fa6

Fix the function input to fetch from variable

69961a8

Merge branch 'sql-in-lens' of https://github.com/stratoula/kibana int…

aa51f96

…o sql-in-lens

Merge branch 'main' into sql-in-lens

becd0a6

flash1293 reviewed Sep 26, 2022

View reviewed changes

stratoula added 3 commits September 26, 2022 15:01

Remove the colorFullBackground check

61b1af1

Fix dashboard bug when timeRange changes

660d0e8

Remove the clear on query update, show column error instead

1392c48

ppisljar approved these changes Sep 27, 2022

View reviewed changes

kertal approved these changes Sep 27, 2022

View reviewed changes

Merge branch 'main' into sql-in-lens

096efe2

flash1293 reviewed Sep 27, 2022

View reviewed changes

stratoula added 2 commits September 27, 2022 11:50

Render a field missing label in case the label is not defined

309b68c

Fix

7fdf2c3

flash1293 approved these changes Sep 27, 2022

View reviewed changes

stratoula merged commit 4c3cd03 into elastic:main Sep 27, 2022

ebuildy mentioned this pull request Jun 9, 2023

Visualizing sql query in Kibana #136740

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Lens] Enables text-based languages (SQL) #140469

[Lens] Enables text-based languages (SQL) #140469

stratoula commented Sep 12, 2022 •

edited

Loading

stratoula Sep 13, 2022

stratoula Sep 13, 2022

stratoula Sep 13, 2022

elasticmachine commented Sep 23, 2022

spalger left a comment

flash1293 Sep 26, 2022

stratoula Sep 26, 2022

flash1293 Sep 26, 2022

stratoula Sep 26, 2022

flash1293 left a comment

stratoula commented Sep 26, 2022

ppisljar left a comment

kertal left a comment

flash1293 Sep 27, 2022

stratoula Sep 27, 2022

flash1293 left a comment

flash1293 left a comment

stratoula commented Sep 27, 2022

kibana-ci commented Sep 27, 2022

API count

[Lens] Enables text-based languages (SQL) #140469

[Lens] Enables text-based languages (SQL) #140469

Conversation

stratoula commented Sep 12, 2022 • edited Loading

Summary

How to test

Flaky runners

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticmachine commented Sep 23, 2022

spalger left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

flash1293 left a comment

Choose a reason for hiding this comment

stratoula commented Sep 26, 2022

ppisljar left a comment

Choose a reason for hiding this comment

kertal left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

flash1293 left a comment

Choose a reason for hiding this comment

flash1293 left a comment

Choose a reason for hiding this comment

stratoula commented Sep 27, 2022

kibana-ci commented Sep 27, 2022

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

Metrics [docs]

Module Count

Public APIs missing comments

Async chunks

Page load bundle

API count

History

stratoula commented Sep 12, 2022 •

edited

Loading