[Logs+] Add All entry for DatasetSelector #160971

tonyghiani · 2023-06-30T09:51:11Z

📓 Summary

This PR adds the entry to allow users to select a dataset that creates a logs-*-* dataview.

Although the presentational and UX changes are minimal for the user, this work lays the foundation for restoring the dataset selection from the URL and for the dataset multi-selection feature.

The core changes for this implementation consist of:

Update DatasetSelector state machine to manage two parallel states: a popover one to manage the navigation on the selector and a selection state that handles the selection modes (currently only all and single, but ready to also implement multi)

DatasetSelector is now a controlled component regarding the selection value: it will react to the injected datasetSelection property, and notify the parent component of any change with the onSelectionChange event handler.
This will allow us to always reinitialize the DatasetSelector state machine from the URL state and fall back to the chosen selection in case there is no one to restore.

all-datasets.mov

Architectural choices

The state machine will handle the two states in parallel such that we can switch to available selection modes depending on the interactions without clashing with the independent selector navigation.
The DatasetSelection data structure is now what represents the state selection in different modes.
The three available modes (counting also multi, but not implemented yet), differs in mostly any aspect:
- Internal data structure shape
- DataViewSpecs composition
- Payload to encode into a URL state
- Extraction of the presentational values (title, icons, and with multi also the datasets which are currently selected)

With all these differences but the same final purposes of creating an ad-hoc DataView and storing encoding a URL state to store, applying the concepts for the Strategy pattern seemed the most sensed option to me. This allows us to scale and add new selection modes (multi) respecting the existing strategies contract and encapsulating all the concerned transformations and parsing.

Encoding and decoding of the Dataview id that will be used for restoring from the URL are already created and will be handy for the upcoming tasks.

…lections and be controlled

elasticmachine · 2023-06-30T09:51:14Z

Pinging @elastic/infra-monitoring-ui (Team:Infra Monitoring UI)

elasticmachine · 2023-06-30T09:51:18Z

Pinging @elastic/fleet (Team:Fleet)

apmmachine · 2023-06-30T09:51:27Z

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

/oblt-deploy : Deploy a Kibana instance using the Observability test environments.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

tonyghiani · 2023-06-30T09:54:44Z

x-pack/plugins/discover_log_explorer/common/datasets/models/dataset.ts

+  public static createAllLogsDataset() {
+    return new Dataset({
+      name: 'logs-*-*' as IndexPattern,
+      title: 'All log datasets',


question: shall we translate this title or can it be a static value as the other dataset names?

I would suggest translating it since you are already translating the selector header Select dataset

tonyghiani · 2023-06-30T09:56:55Z

x-pack/plugins/discover_log_explorer/public/components/dataset_selector/dataset_selector.tsx


 export function DatasetSelector({
  datasets,
  datasetsError,
-  initialSelected,
+  datasetSelection,


The component relies on this controlled property to present what the current selection is, while the state machine will notify with a new selection instance anytime the user clicks on an entry.

tonyghiani · 2023-06-30T09:59:56Z

...gins/discover_log_explorer/public/components/dataset_selector/state_machine/state_machine.ts

+        storeAllSelection: assign((_context) => ({
+          selection: AllDatasetSelection.create(),
+        })),
+        storeSingleSelection: assign((_context, event) =>
+          'dataset' in event ? { selection: SingleDatasetSelection.create(event.dataset) } : {}


We want to store the correct DatasetSelection instance depending on the selection mode the user can select.
These actions are triggered uniquely when a new selection is applied and work as a setter for the dataset selection strategy.

…fix'

…iani/kibana into 160146-all-entry-dataset-selector

tonyghiani · 2023-06-30T10:12:34Z

x-pack/plugins/discover_log_explorer/public/customizations/custom_dataset_selector.tsx

+   * until we handle the restore/initialization of the dataview with https://github.com/elastic/kibana/issues/160425,
+   * where this value will be used to control the DatasetSelector selection with a top level state machine.
+   */
+  const [datasetSelection, setDatasetSelection] = useState<DatasetSelection>(() =>


note: as mentioned above, this state is only used to control the selection state, in the next task we'll move this logic and its initialization to a state machine that restores from the URL instead of hardcoding the AllDatasetSelection selection type.

tonyghiani · 2023-06-30T10:18:07Z

x-pack/plugins/discover_log_explorer/public/utils/dataset_selection/encoding.ts

This encoding basically does these steps in order (reversed for decoding):

Validate that the object selection input is in a valid format.

Rison encodes the selection.

compressToBase64 the rison-encoded value

Although lz-string provides methods with a higher compression rate, the resulting string would be a non-ASCII string. If there is any better option to update these encoding/decoding I'll be happy to explore the options!

tonyghiani · 2023-06-30T10:19:17Z

...plugins/discover_log_explorer/public/utils/dataset_selection/hydrate_dataset_selection.ts.ts

+import { SingleDatasetSelection } from './single_dataset_selection';
+import { DatasetSelectionPlain } from './types';
+
+export const hydrateDatasetSelection = (datasetSelection: DatasetSelectionPlain) => {


This is added in preparation for the following task, where we'll need to transform it into a DatasetSelection instance once the URL state is correctly restored.

tonyghiani · 2023-06-30T10:20:03Z

x-pack/plugins/discover_log_explorer/public/utils/dataset_selection/types.ts

+export type SingleDatasetSelectionPayload = rt.TypeOf<typeof singleDatasetSelectionPayloadRT>;
+export type DatasetSelectionPlain = rt.TypeOf<typeof datasetSelectionPlainRT>;
+
+export interface DatasetSelectionStrategy {


This is the contract definition that any existing or future selection mode should respect.

mohamedhamed-ahmed · 2023-07-03T12:46:14Z

x-pack/plugins/discover_log_explorer/common/datasets/models/dataset.ts

+
+  public static createAllLogsDataset() {
+    return new Dataset({
+      name: 'logs-*-*' as IndexPattern,


is there a way we can set remote_cluster:logs-*-* ?

At the current stage, we won't allow customizing the datasets also to access remote clusters but is something we'll maybe allow the user to customize in future.

why not set the index pattern to [remote_cluster:logs-*-*, logs-*-*] ? without any customizations

As it is for the current Logs UI implementation, it is possible that a remote cluster is not configured or reachable.
I'm not sure what is the plan in long term, but currently we are not allowing to access the remote clusters from this dataset selector on serverless enviroment.

mohamedhamed-ahmed

LGTM! Clean work
Tested it and works perfectly 👍 Do we plan on adding tests for the whole component?

mohamedhamed-ahmed · 2023-07-03T13:18:05Z

x-pack/plugins/discover_log_explorer/common/datasets/models/dataset.ts

 import { DataViewSpec } from '@kbn/data-views-plugin/common';
+import { IndexPattern } from '@kbn/io-ts-utils';


would adding type here make sense ?

It is not necessary as IndexPattern as purely a type with no runtime code, it is exported using the export type syntax, which guarantees it gets removed from the bundle.
Here is the detail on when is useful to use the import type syntax.

tonyghiani · 2023-07-03T16:07:28Z

Do we plan on adding tests for the whole component?

Absolutely, I created an issue to track what tests we need as soon as we have are done with this initial MVP set of features 👌

kibana-ci · 2023-07-04T07:03:11Z

💚 Build Succeeded

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`discoverLogExplorer`	132	140	+8

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`discoverLogExplorer`	184.2KB	192.0KB	+7.8KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`discoverLogExplorer`	4.3KB	4.4KB	+52.0B

Unknown metric groups

ESLint disabled in files

id	before	after	diff
`discoverLogExplorer`	2	3	+1

ESLint disabled line counts

id	before	after	diff
`enterpriseSearch`	14	16	+2
`securitySolution`	410	414	+4
total			+6

Total ESLint disabled count

id	before	after	diff
`discoverLogExplorer`	5	6	+1
`enterpriseSearch`	15	17	+2
`securitySolution`	489	493	+4
total			+7

History

💚 Build #139403 succeeded 6d24915
💔 Build #139372 failed de15dc6

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

## 📓 Summary Closes elastic#160146 This PR adds the entry to allow users to select a dataset that creates a `logs-*-*` dataview. Although the presentational and UX changes are minimal for the user, this work lays the foundation for [restoring the dataset selection from the URL](elastic#160425) and for the [dataset multi-selection feature](https://github.com/elastic/observability-dev/issues/2744). The core changes for this implementation consist of: - Update DatasetSelector state machine to manage two parallel states: a `popover` one to manage the navigation on the selector and a `selection` state that handles the selection modes (currently only `all` and `single`, but ready to also implement `multi`) <img width="1522" alt="state-machine" src="https://github.com/elastic/kibana/assets/34506779/c240e5d5-6a38-4d08-b893-117132477896"> - DatasetSelector is now a controlled component regarding the selection value: it will react to the injected `datasetSelection` property, and notify the parent component of any change with the `onSelectionChange` event handler. This will allow us to always reinitialize the DatasetSelector state machine from the URL state and fall back to the chosen selection in case there is no one to restore. https://github.com/elastic/kibana/assets/34506779/4887b1d4-63ba-476b-a74f-5b4a9504f939 ## Architectural choices - The state machine will handle the two states in parallel such that we can switch to available selection modes depending on the interactions without clashing with the independent selector navigation. - The `DatasetSelection` data structure is now what represents the state selection in different modes. The three available modes (counting also `multi`, but not implemented yet), differs in mostly any aspect: - Internal data structure shape - DataViewSpecs composition - Payload to encode into a URL state - Extraction of the presentational values (title, icons, and with `multi` also the datasets which are currently selected) With all these differences but the same final purposes of creating an ad-hoc DataView and storing encoding a URL state to store, applying the concepts for the Strategy pattern seemed the most sensed option to me. This allows us to scale and add new selection modes (`multi`) respecting the existing strategies contract and encapsulating all the concerned transformations and parsing. Encoding and decoding of the Dataview id that will be used for restoring from the URL are already created and will be handy for the upcoming tasks. --------- Co-authored-by: Marco Antonio Ghiani <marcoantonio.ghiani@elastic.co> Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

## 📓 Summary Closes #160425 After the [first implementation of the log-explorer profile](#159907), we wanted to restore the selection of the dataset for a user when landing on the Discover log-explorer profile. Since we create an ad-hoc data view for Discover starting from the dataset details, we needed to develop a system for intercepting the `index` query parameter (which is used by Discover as the source of truth for restoring a data view), create our ad-hoc data view and store in the URL an encoded ID with the required details to restore the selection. The following video shows the user journey for: - Landing on the log-explorer profile with no index param, nothing to restore and fallback to All log datasets. - Landing on the log-explorer profile invalid index param, notify about failure and fallback to All log datasets. - Select a different dataset, applies the new data view and update the URL. When the URL is accessed directly, restore and initialize the data view for the selection. - Navigate back and forth in the browser history, restoring the selection and data view on `index` param changes. https://github.com/elastic/kibana/assets/34506779/37a212ee-08e4-4e54-8e42-1d739c38f164 ## 💡 Reviewer hints To have better control over the page selection and the restore process, we prepared the DatasetSelector component for [being controlled by the parent component](#160971). Having that ready, we now implemented a new top-level state machine with the following responsibilities: - Re-initialize (decompress/decode) the dataset selection from the `index` query params. - Derive and set into Discover state a new ad-hoc data view. - Keep track of new dataset selection changes and update the URL state and the current data view. <img width="1224" alt="log-explorer-machine" src="https://github.com/elastic/kibana/assets/34506779/67e3ff17-dc3f-4dcf-b6c0-f40dbbea2d44"> We found a race condition between the Discover URL initialization + data view initialization against the log-explorer profile customizations being applied. To guarantee we correctly initialize the state machine and restore the selection before Discover goes through its initialization steps, we need to wait for the customization service to exist in Discover so that also the customization callbacks are successfully invoked. --------- Co-authored-by: Marco Antonio Ghiani <marcoantonio.ghiani@elastic.co> Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

## 📓 Summary Closes elastic#160425 After the [first implementation of the log-explorer profile](elastic#159907), we wanted to restore the selection of the dataset for a user when landing on the Discover log-explorer profile. Since we create an ad-hoc data view for Discover starting from the dataset details, we needed to develop a system for intercepting the `index` query parameter (which is used by Discover as the source of truth for restoring a data view), create our ad-hoc data view and store in the URL an encoded ID with the required details to restore the selection. The following video shows the user journey for: - Landing on the log-explorer profile with no index param, nothing to restore and fallback to All log datasets. - Landing on the log-explorer profile invalid index param, notify about failure and fallback to All log datasets. - Select a different dataset, applies the new data view and update the URL. When the URL is accessed directly, restore and initialize the data view for the selection. - Navigate back and forth in the browser history, restoring the selection and data view on `index` param changes. https://github.com/elastic/kibana/assets/34506779/37a212ee-08e4-4e54-8e42-1d739c38f164 ## 💡 Reviewer hints To have better control over the page selection and the restore process, we prepared the DatasetSelector component for [being controlled by the parent component](elastic#160971). Having that ready, we now implemented a new top-level state machine with the following responsibilities: - Re-initialize (decompress/decode) the dataset selection from the `index` query params. - Derive and set into Discover state a new ad-hoc data view. - Keep track of new dataset selection changes and update the URL state and the current data view. <img width="1224" alt="log-explorer-machine" src="https://github.com/elastic/kibana/assets/34506779/67e3ff17-dc3f-4dcf-b6c0-f40dbbea2d44"> We found a race condition between the Discover URL initialization + data view initialization against the log-explorer profile customizations being applied. To guarantee we correctly initialize the state machine and restore the selection before Discover goes through its initialization steps, we need to wait for the customization service to exist in Discover so that also the customization callbacks are successfully invoked. --------- Co-authored-by: Marco Antonio Ghiani <marcoantonio.ghiani@elastic.co> Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

Marco Antonio Ghiani added 5 commits June 29, 2023 10:12

feat(discover-log-explorer): wip on mode state selection

8b5af82

feat(discover-log-explorer): wip on DatasetSelection data structures

3272648

feat(discover-log-explorer): implement dataset selection strategies

d305f84

refactor(discover-log-explorer): update dataset selector to manage se…

1815c18

…lections and be controlled

tests(discover-log-explorer): add encoding/decoding tests

68e70d8

tonyghiani added Team:Infra Monitoring UI - DEPRECATED DEPRECATED - Label for the Infra Monitoring UI team. Use Team:obs-ux-infra_services release_note:skip Skip the PR/issue when compiling release notes labels Jun 30, 2023

tonyghiani requested review from a team as code owners June 30, 2023 09:51

botelastic bot added the Team:Fleet Team label for Observability Data Collection Fleet team label Jun 30, 2023

refactor(discover-log-explorer): remove empty line

dc1b427

tonyghiani removed the request for review from a team June 30, 2023 09:53

tonyghiani commented Jun 30, 2023

View reviewed changes

kibanamachine and others added 3 commits June 30, 2023 10:00

[CI] Auto-commit changed files from 'node scripts/lint_ts_projects --…

24e5437

…fix'

refactor(discover-log-explorer): improve state machine readability

f483418

Merge branch '160146-all-entry-dataset-selector' of github.com:tonygh…

de15dc6

…iani/kibana into 160146-all-entry-dataset-selector

tonyghiani commented Jun 30, 2023

View reviewed changes

Merge branch 'main' into 160146-all-entry-dataset-selector

6d24915

mohamedhamed-ahmed self-requested a review July 3, 2023 10:14

mohamedhamed-ahmed reviewed Jul 3, 2023

View reviewed changes

mohamedhamed-ahmed approved these changes Jul 3, 2023

View reviewed changes

Merge branch 'main' into 160146-all-entry-dataset-selector

90be221

tonyghiani merged commit b9e2fdb into elastic:main Jul 4, 2023
21 checks passed

kibanamachine added v8.10.0 backport:skip This commit does not require backporting labels Jul 4, 2023

tonyghiani deleted the 160146-all-entry-dataset-selector branch July 4, 2023 07:10

tonyghiani mentioned this pull request Jul 7, 2023

[Logs+] Restore Dataset selection from page URL #161144

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Logs+] Add All entry for DatasetSelector #160971

[Logs+] Add All entry for DatasetSelector #160971

tonyghiani commented Jun 30, 2023 •

edited

Loading

elasticmachine commented Jun 30, 2023

elasticmachine commented Jun 30, 2023

apmmachine commented Jun 30, 2023

tonyghiani Jun 30, 2023

mohamedhamed-ahmed Jul 3, 2023

tonyghiani Jun 30, 2023

tonyghiani Jun 30, 2023

tonyghiani Jun 30, 2023

tonyghiani Jun 30, 2023

tonyghiani Jun 30, 2023

tonyghiani Jun 30, 2023

mohamedhamed-ahmed Jul 3, 2023

tonyghiani Jul 3, 2023

mohamedhamed-ahmed Jul 3, 2023

tonyghiani Jul 3, 2023

mohamedhamed-ahmed left a comment

mohamedhamed-ahmed Jul 3, 2023

tonyghiani Jul 3, 2023

tonyghiani commented Jul 3, 2023

kibana-ci commented Jul 4, 2023

ESLint disabled in files

ESLint disabled line counts

Total ESLint disabled count

		import { DataViewSpec } from '@kbn/data-views-plugin/common';
		import { IndexPattern } from '@kbn/io-ts-utils';

[Logs+] Add All entry for DatasetSelector #160971

[Logs+] Add All entry for DatasetSelector #160971

Conversation

tonyghiani commented Jun 30, 2023 • edited Loading

📓 Summary

Architectural choices

elasticmachine commented Jun 30, 2023

elasticmachine commented Jun 30, 2023

apmmachine commented Jun 30, 2023

🤖 GitHub comments

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mohamedhamed-ahmed left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tonyghiani commented Jul 3, 2023

kibana-ci commented Jul 4, 2023

💚 Build Succeeded

Metrics [docs]

Module Count

Async chunks

Page load bundle

ESLint disabled in files

ESLint disabled line counts

Total ESLint disabled count

History

tonyghiani commented Jun 30, 2023 •

edited

Loading