[Infra] Deprecate fields returned by the `/api/metrics/source` endpoint, in favor of ad hoc data views #181954

crespocarlos · 2024-04-29T10:30:19Z

Summary

This PR lays the necessary foundation to deprecate the indices fields list attribute retrieved and returned by the api/metrics/source endpoint and affects all infra pages.

Now instead of the Metrics UI and Inventory UI relying on that attribute to build the list of suggestions in their search bars, an adhoc data view will be created from the metricsAlias returned by the same endpoint. The data view object contains everything that's needed with the benefit of allowing us to make async requests and cache, provided by Data Views Service.

This will allow us to start moving away from api/metrics/source to retrieve information critical to the loading of the pages.

It's a big PR, and the changes affect mostly the search bars, field name selectors, and charts across infra. Besides, it also contains

Storybook fix.
Some components were turned into functional components
<MetricsPageTemplate /> now encapsulates error messages and is only used in Infrastructure pages
Improvement of the Infrastructure routing code.
Data View provider common to all infra pages
SourceProvider refactor.

How to test

Start a local kibana instance pointing to an Oblt cluster
Navigate to Inventory, and all its pages, check if charts, search bars and field name autocomplete fields are working

apmmachine · 2024-04-29T10:30:31Z

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

/oblt-deploy : Deploy a Kibana instance using the Observability test environments.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

crespocarlos · 2024-04-29T14:54:50Z

/ci

… dataviews

crespocarlos · 2024-04-30T11:03:12Z

/ci

crespocarlos · 2024-04-30T11:05:46Z

..._solution/infra/public/pages/metrics/inventory_view/components/waffle/custom_field_panel.tsx

@@ -8,83 +8,76 @@
 import { EuiButton, EuiComboBox, EuiForm, EuiFormRow } from '@elastic/eui';


Made it a functional component

crespocarlos · 2024-04-30T11:06:39Z

...ion/infra/public/pages/metrics/inventory_view/components/waffle/waffle_group_by_controls.tsx

@@ -15,193 +15,176 @@ import {
 import { i18n } from '@kbn/i18n';


Made it a functional component

crespocarlos · 2024-04-30T12:14:22Z

@elasticmachine merge upstream

crespocarlos · 2024-04-30T15:05:08Z

/ci

crespocarlos · 2024-04-30T17:04:30Z

...observability_solution/infra/public/pages/metrics/settings/source_configuration_settings.tsx

    return <SourceLoadingPage />;
  }

  return (
-    <MetricsPageTemplate
+    <PageTemplate


we don't want all the validations that happen in <MetricsPageTemplate /> in the Settings page

crespocarlos · 2024-04-30T17:05:02Z

x-pack/plugins/observability_solution/infra/public/pages/404.tsx

  return (
-    <MetricsPageTemplate
+    <PageTemplate


we don't want all the validations that happen in <MetricsPageTemplate /> in the Settings page

elasticmachine · 2024-04-30T17:05:35Z

Pinging @elastic/obs-ux-infra_services-team (Team:obs-ux-infra_services)

crespocarlos · 2024-05-02T12:22:28Z

@elasticmachine merge upstream

neptunian · 2024-05-02T13:20:34Z

x-pack/plugins/observability_solution/infra/public/containers/metrics_source/metrics_view.ts

+    services: { dataViews },
+  } = useKibanaContextForPlugin();
+
+  const { source } = useSourceContext();


If we're planning to remove the index pattern from here we could replace this with something that fetches for the index pattern in the metrics data access plugin. Is that what you're thinking to do in a follow up?

exactly, this will be removed once we have a new endpoint to return the index pattern from metrics data access plugin

neptunian · 2024-05-02T13:24:12Z

x-pack/plugins/observability_solution/infra/public/alerting/inventory/components/expression.tsx

@@ -120,15 +120,12 @@ export const defaultExpression = {

 export const Expressions: React.FC<Props> = (props) => {
  const { setRuleParams, ruleParams, errors, metadata } = props;
-  const { source, createDerivedIndexPattern } = useSourceContext();
+  const { source } = useSourceContext();


how is this being used here? seems like we may not need it with the inclusion of metricsView.

yes. we need the sourceId. Should we create a ticket to stop using this attribute? I don't really see the point of having it in the current solution.

oops, missed that

@elastic/obs-ux-management-team we might want to log the ticket Carlos mentions here and get rid of the source entirely if it's no longer used/needed, in favor of data views

neptunian · 2024-05-02T13:42:59Z

x-pack/plugins/observability_solution/infra/public/components/asset_details/charts/chart.tsx

+  } = useKibanaContextForPlugin();
+
+  const { value: filters = [] } = useAsync(async () => {
+    const resolvedDataView = await resolveDataView({


what's the reasoning behind using this instead of useMetricsDataViewContext().

it's because we need to retrieve the logs data view too for the logs rate chart.

This chart component is data view agnostic and it depends on the index pattern information that the chart objects hold. This is only needed to build filters in a way that they look nicer when the user opens the chart in Lens. If it wasn't for that, this whole section could've been removed.

neptunian · 2024-05-02T15:00:48Z

Thanks for tackling this!

jennypavlova

Looks good! I added some nits/questions.

jennypavlova · 2024-05-06T09:26:06Z

...ion/infra/public/pages/metrics/inventory_view/components/waffle/waffle_group_by_controls.tsx

+    <DropdownButton
+      label={i18n.translate('xpack.infra.waffle.groupByLabel', { defaultMessage: 'Group by' })}
+      onClick={togglePopover}
+      data-test-subj={'waffleGroupByDropdown'}


Nit

Suggested change

data-test-subj={'waffleGroupByDropdown'}

data-test-subj="waffleGroupByDropdown"

jennypavlova · 2024-05-06T09:29:19Z

x-pack/plugins/observability_solution/infra/public/utils/data_view.ts

+  name?: string;
+}
+
+export async function resolveDataView({


Do we need async here?

jennypavlova · 2024-05-06T09:30:08Z

x-pack/plugins/observability_solution/infra/public/utils/filters/build.ts

@@ -12,7 +12,7 @@ import {
  type Filter,
  isCombinedFilter,
 } from '@kbn/es-query';
-import type { DataView } from '@kbn/data-views-plugin/common';
+import { DataView } from '@kbn/data-views-plugin/common';


Why is this change needed?

jennypavlova · 2024-05-06T09:35:33Z

...rvability_solution/infra/public/components/ml/anomaly_detection/anomaly_detection_flyout.tsx

@@ -17,8 +17,8 @@ import { MetricK8sModuleProvider } from '../../../containers/ml/modules/metrics_
 import { useActiveKibanaSpace } from '../../../hooks/use_kibana_space';

 export const AnomalyDetectionFlyout = ({
-  hideJobType,
-  hideSelectGroup,
+  hideJobType = false,


Why do we need those default values?

It's better than letting them be undefined in case it's not informed

I asked because we have them as optional also in other places where we check the value but it's Ok to have them set to false by default.

jennypavlova · 2024-05-06T09:37:41Z

...lity_solution/infra/public/pages/metrics/metrics_explorer/hooks/use_metrics_explorer_data.ts

+  timestamps: MetricsExplorerTimestamp;
+  enabled?: boolean;
+}) {
+  // Your code logic here


Do we need this comment?

Suggested change

// Your code logic here

…e-fields-from-source-endpoint

jennypavlova

LGTM 💯 Thanks for the fixes!

kdelemme · 2024-05-06T19:11:24Z

@crespocarlos I'm having difficulties to see data in the Inventory pages using the edge-oblt and edge-oblt-lite. Which one should I use?

crespocarlos · 2024-05-06T19:17:21Z

@crespocarlos I'm having difficulties to see data in the Inventory pages using the edge-oblt and edge-oblt-lite. Which one should I use?

Hi @kdelemme . It should work with both. If pointing your local instance to an oblt cluster, check if you've configured the index pattern in the settings page.

value: remote_cluster:metrics-*,remote_cluster:metricbeat-*

kdelemme · 2024-05-06T19:45:35Z

Ah thanks ! Forgot about this..

kdelemme

Code change LGTM

crespocarlos · 2024-05-08T13:11:38Z

@elasticmachine merge upstream

crespocarlos · 2024-05-08T16:26:10Z

@elasticmachine merge upstream

crespocarlos · 2024-05-13T12:10:23Z

@elasticmachine merge upstream

…e-fields-from-source-endpoint

kibana-ci · 2024-05-27T08:39:51Z

💚 Build Succeeded

Buildkite Build
Commit: 85b4675
Storybooks Preview
Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-181954-85b4675d1af1
Observability Deployment

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`infra`	1561	1560	-1

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`infra`	1.5MB	1.5MB	-1.1KB

Canvas Sharable Runtime

The Canvas "shareable runtime" is an bundle produced to enable running Canvas workpads outside of Kibana. This bundle is included in third-party webpages that embed canvas and therefor should be as slim as possible.

id	before	after	diff
`module count`	-	5412	+5412
`total size`	-	8.8MB	+8.8MB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`infra`	105.6KB	105.7KB	+122.0B

History

💚 Build #209505 succeeded 1117802
💚 Build #208743 succeeded aaf7253
💔 Build #208675 failed 0200627
💛 Build #208112 was flaky 6686f57

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

…nt, in favor of ad hoc data views (elastic#181954) closes [elastic#180689](elastic#180689) ## Summary This PR lays the necessary foundation to deprecate the indices [fields list attribute retrieved and returned](https://github.com/elastic/kibana/blob/main/x-pack/plugins/observability_solution/infra/server/routes/metrics_sources/index.ts#L38) by the `api/metrics/source` endpoint and affects all infra pages. Now instead of the Metrics UI and Inventory UI relying on that attribute to build the list of suggestions in their search bars, an adhoc data view will be created from the `metricsAlias` returned by the same endpoint. The data view object contains everything that's needed with the benefit of allowing us to make async requests and cache, provided by Data Views Service. This will allow us to start moving away from `api/metrics/source` to retrieve information critical to the loading of the pages. It's a big PR, and the changes affect mostly the search bars, field name selectors, and charts across infra. Besides, it also contains - Storybook fix. - Some components were turned into functional components - `<MetricsPageTemplate />` now encapsulates error messages and is only used in Infrastructure pages - Improvement of the Infrastructure routing code. - Data View provider common to all infra pages - `SourceProvider` refactor. <img width="921" alt="image" src="https://github.com/elastic/kibana/assets/2767137/ceb781ed-2464-4f67-93e4-7013786830d4"> <img width="921" alt="image" src="https://github.com/elastic/kibana/assets/2767137/169f6f6b-03da-48e3-86b8-449bad1b2035"> <img width="921" alt="image" src="https://github.com/elastic/kibana/assets/2767137/3e5b8394-ead2-40bb-ad00-609b29b179e5"> <img width="921" alt="image" src="https://github.com/elastic/kibana/assets/2767137/34639ebf-af75-46e3-840b-8aa89951ae6e"> <img width="921" alt="image" src="https://github.com/elastic/kibana/assets/2767137/b6ff3b96-219b-4daa-bc33-f9fd36912a5d"> <img width="921" alt="image" src="https://github.com/elastic/kibana/assets/2767137/437cb7b8-62a6-4e1a-b94e-41055ab71bc1"> <img width="921" alt="image" src="https://github.com/elastic/kibana/assets/2767137/72f04712-7ee1-44fc-990e-83f0fbff7115"> ## How to test - Start a local kibana instance pointing to an Oblt cluster - Navigate to Inventory, and all its pages, check if charts, search bars and field name autocomplete fields are working --------- Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

crespocarlos force-pushed the 180689-deprecate-fields-from-source-endpoint branch from 50bb70e to 63ea9bb Compare April 29, 2024 14:54

crespocarlos force-pushed the 180689-deprecate-fields-from-source-endpoint branch from 63ea9bb to 33899c1 Compare April 29, 2024 16:50

Stop using fields returned by the source endpoint, in favor of ad hoc…

2020a7d

… dataviews

crespocarlos force-pushed the 180689-deprecate-fields-from-source-endpoint branch 4 times, most recently from e56ab19 to a511639 Compare April 30, 2024 11:02

crespocarlos commented Apr 30, 2024

View reviewed changes

Fix tests

a7eb928

crespocarlos force-pushed the 180689-deprecate-fields-from-source-endpoint branch from a511639 to a7eb928 Compare April 30, 2024 12:14

kibanamachine and others added 2 commits April 30, 2024 08:14

Merge branch 'main' into 180689-deprecate-fields-from-source-endpoint

c16c022

Clean up and storybook fix

f08c945

crespocarlos commented Apr 30, 2024

View reviewed changes

crespocarlos marked this pull request as ready for review April 30, 2024 17:05

crespocarlos requested review from a team as code owners April 30, 2024 17:05

crespocarlos changed the title ~~Stop using fields returned by the source endpoint, in favor of ad hoc…~~ [Infra] Deprecate fields returned by the /api/metrics/source endpoint, in favor of ad hoc data views Apr 30, 2024

Merge branch 'main' into 180689-deprecate-fields-from-source-endpoint

c552c4b

neptunian reviewed May 2, 2024

View reviewed changes

neptunian approved these changes May 2, 2024

View reviewed changes

jennypavlova reviewed May 6, 2024

View reviewed changes

crespocarlos added 2 commits May 6, 2024 09:20

Merge branch 'main' of github.com:elastic/kibana into 180689-deprecat…

7b25dd3

…e-fields-from-source-endpoint

CR fixes

6686f57

kdelemme self-assigned this May 6, 2024

kdelemme self-requested a review May 6, 2024 14:39

kdelemme removed their assignment May 6, 2024

jennypavlova approved these changes May 6, 2024

View reviewed changes

kdelemme mentioned this pull request May 6, 2024

Stop using sourceId in Inventory rule #182719

Open

kdelemme approved these changes May 6, 2024

View reviewed changes

Merge branch 'main' into 180689-deprecate-fields-from-source-endpoint

0200627

Merge branch 'main' into 180689-deprecate-fields-from-source-endpoint

aaf7253

kibanamachine and others added 2 commits May 13, 2024 08:10

Merge branch 'main' into 180689-deprecate-fields-from-source-endpoint

1117802

Merge branch 'main' of github.com:elastic/kibana into 180689-deprecat…

85b4675

…e-fields-from-source-endpoint

crespocarlos merged commit 982dfa3 into elastic:main May 27, 2024
22 checks passed

crespocarlos deleted the 180689-deprecate-fields-from-source-endpoint branch May 27, 2024 10:14

		@@ -8,83 +8,76 @@
		import { EuiButton, EuiComboBox, EuiForm, EuiFormRow } from '@elastic/eui';

		@@ -15,193 +15,176 @@ import {
		import { i18n } from '@kbn/i18n';

	data-test-subj={'waffleGroupByDropdown'}
	data-test-subj="waffleGroupByDropdown"

[Infra] Deprecate fields returned by the /api/metrics/source endpoint, in favor of ad hoc data views #181954

[Infra] Deprecate fields returned by the /api/metrics/source endpoint, in favor of ad hoc data views #181954

Conversation

crespocarlos commented Apr 29, 2024 • edited

Summary

How to test

apmmachine commented Apr 29, 2024

🤖 GitHub comments

crespocarlos commented Apr 29, 2024

crespocarlos commented Apr 30, 2024

crespocarlos Apr 30, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

crespocarlos commented Apr 30, 2024

crespocarlos commented Apr 30, 2024

crespocarlos Apr 30, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticmachine commented Apr 30, 2024

crespocarlos commented May 2, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

neptunian May 2, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

neptunian commented May 2, 2024

jennypavlova left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jennypavlova left a comment

Choose a reason for hiding this comment

kdelemme commented May 6, 2024

crespocarlos commented May 6, 2024

kdelemme commented May 6, 2024

kdelemme left a comment

Choose a reason for hiding this comment

crespocarlos commented May 8, 2024

crespocarlos commented May 8, 2024

crespocarlos commented May 13, 2024

kibana-ci commented May 27, 2024 • edited

💚 Build Succeeded

Metrics [docs]

Module Count

Async chunks

Canvas Sharable Runtime

Page load bundle

History

[Infra] Deprecate fields returned by the `/api/metrics/source` endpoint, in favor of ad hoc data views #181954

[Infra] Deprecate fields returned by the `/api/metrics/source` endpoint, in favor of ad hoc data views #181954

crespocarlos commented Apr 29, 2024 •

edited

crespocarlos Apr 30, 2024 •

edited

crespocarlos Apr 30, 2024 •

edited

neptunian May 2, 2024 •

edited

kibana-ci commented May 27, 2024 •

edited