Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ML] Adds Field statistics embeddable as panel in Dashboard #184030

Merged
merged 50 commits into from
Jul 2, 2024

Conversation

qn895
Copy link
Member

@qn895 qn895 commented May 22, 2024

Summary

This PR adds Field statistics embeddable as panel in Dashboard

By default, it will enable the ES|QL editor for the field stats panel. It will allow for editing of the ES|QL query, and time range.

Screen.Recording.2024-07-01.at.14.40.16.mov
Screen.Recording.2024-07-01.at.14.41.01.mov

If and only if ES|QL is disabled, it will show the data view picker as a fallback.

Checklist

Delete any items that are not applicable to this PR.

Risk Matrix

Delete this section if it is not applicable to this PR.

Before closing this PR, invite QA, stakeholders, and other developers to identify risks that should be tested prior to the change/feature release.

When forming the risk matrix, consider some of the following examples and how they may potentially impact the change:

Risk Probability Severity Mitigation/Notes
Multiple Spaces—unexpected behavior in non-default Kibana Space. Low High Integration tests will verify that all features are still supported in non-default Kibana Space and when user switches between spaces.
Multiple nodes—Elasticsearch polling might have race conditions when multiple Kibana nodes are polling for the same tasks. High Low Tasks are idempotent, so executing them multiple times will not result in logical error, but will degrade performance. To test for this case we add plenty of unit tests around this logic and document manual testing procedure.
Code should gracefully handle cases when feature X or plugin Y are disabled. Medium High Unit tests will verify that any feature flag or plugin combination still results in our service operational.
See more potential risk examples

For maintainers

@qn895 qn895 added the :ml label May 22, 2024
@qn895
Copy link
Member Author

qn895 commented Jun 7, 2024

/ci

@peteharverson
Copy link
Contributor

Done some early testing of this one (against commit cb3fd16) - overall looks good.

One quick question - are you going to add in the ability to add a filter from values in the expanded rows (it's available from the panel added from a search saved in Discover)?

): UiActionsActionDefinition<FieldStatsActionContext> {
return {
id: 'create-field-stats-table',
getIconType: () => 'inspect',
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@joana-cps any thoughts on this icon?

Screenshot 2024-06-10 at 13 42 53

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This icon is already in use in dashboards to inspect elements, I'll draft up a new one more tailored for field statistics. Added to this UX issue in progress

@qn895
Copy link
Member Author

qn895 commented Jun 10, 2024

/ci


const [dataViewId, setDataViewId] = useState(initialInput?.dataViewId ?? '');
const [viewType, setViewType] = useState(
initialInput?.viewType ?? FieldStatsInitializerViewType.ESQL
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is ES|QL the default in Discover yet? (and what about Serverless)? Just wonder if the default should match the default in Discover?

@qn895 qn895 marked this pull request as ready for review June 10, 2024 20:20
@qn895 qn895 requested review from a team as code owners June 10, 2024 20:20
@elasticmachine
Copy link
Contributor

Pinging @elastic/ml-ui (:ml)

@qn895 qn895 added Feature:File and Index Data Viz ML file and index data visualizer release_note:feature Makes this part of the condensed release notes v8.15.0 labels Jun 10, 2024
Copy link
Contributor

@TinaHeiligers TinaHeiligers left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

kibana.jsonc update LGTM

@qn895 qn895 added the impact:medium Addressing this issue will have a medium level of impact on the quality/strength of our product. label Jun 27, 2024
const references: Reference[] = dataViewId
? [
{
type: 'index-pattern',
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Switching from DATA_VIEW_SAVED_OBJECT_TYPE to 'index-pattern' will not have an impact on page load bundle size. x-pack/plugins/data_visualizer/public/application/index_data_visualizer/embeddables/field_stats/field_stats_factory.tsx is imported as const { getFieldStatsChartEmbeddableFactory } = await import('./field_stats_factory'); in x-pack/plugins/data_visualizer/public/application/index_data_visualizer/embeddables/field_stats/index.ts. This means that the file is not included in page load bundle and is only imported when the callback passed to registerReactEmbeddableFactory is called. This callback is not called until the embeddable is rendered in a dashbaord.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was a test that I intended to revert. It's now been reverted.

...timeRangeApi,
...titlesApi,
...fieldStatsControlsApi,
...dataLoadingApi,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couple questions about exposing dataLoadingApi in your API. The type FieldStatisticsTableEmbeddableApi does not include keys for onRenderComplete, onLoading, and onError. Is there a reason why these keys are getting exposed via the embeddable API? Do consumers of the api use them?

Another point dataLoading$ needs to be exposed as dataLoading. There is some inconsistencies in naming that we need to fix on the interface side.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nreese Updated here 2e106ba

updateUserInput: (update: Partial<FieldStatsInitialState>) => void;
}

export type FieldStatisticsTableEmbeddableApi =
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

API should type should also include PublishesDataLoading and PublishesBlockingError.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated here 2e106ba

})
)
.subscribe((nextSelectedDataView) => {
if (nextSelectedDataView) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fetching initialDataView sets a blocking error when data view id can not be found. Here, when data view id changes and data view can not be found, the error is ignored and no blocking error is set. Is this on purpose? Should switchMap return an error in the catch? Something more like

if (fieldStatsControlsApi.dataViewId$) {
  subscriptions.add(
    fieldStatsControlsApi.dataViewId$
      .pipe(
        skip(1),
        skipWhile((dataViewId) => !dataViewId && !defaultDataViewId),
        switchMap(async (dataViewId) => {
          try {
            const dataView = await deps.data.dataViews.get(dataViewId ?? defaultDataViewId);
            return { dataView };
          } catch (error) {
            return { dataView: undefined, error };
          }
        })
      )
      .subscribe(({ dataView, error }) => {
        if (error) {
          onError(error);
        }
        dataViews$.next([dataView]);
      })
  );
}

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this should be a blocking error. The table itself doesn't need any data view to render, we are fetching data views only for dashboard filters to show the values properly.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated here 1cad878

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this should be a blocking error. The table itself doesn't need any data view to render, we are fetching data views only for dashboard filters to show the values properly.

Thanks for explaining.

I think the only change needed is that switchMap function needs to be async, switchMap(async (dataViewId) => {}. And then await dataViews.get for the return return await deps.data.dataViews.get(dataViewId ?? defaultDataViewId);

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated here 3fa980b


fieldStatsControlsApi.updateUserInput(nextUpdate);
} catch (e) {
return Promise.reject(e);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Promise.reject(e) will cause onEdit to throw. EditPanelAction.execute does not have any try/catch around onEdit calls, resulting in unhandled error. How about displaying a warning toast when editing fails instead of throwing?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated here 1cad878

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added an error toast instead of warning toast, since that means the update fails for some reason and we probably would like to see the error trace. Let me know if that's not the standard and I can update it to be a warning toast.

toasts.addWarning(ERROR_MSG.APPLY_FILTER_ERR);
return;
// eslint-disable-next-line no-console
console.error(`${ERROR_MSG.APPLY_FILTER_ERR}: APPLY_FILTER_TRIGGER is undefined`);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

console.error can be removed. Its after a return so this code will never get run

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated here 1cad878

Copy link
Contributor

@nreese nreese left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

embeddable factory LGTM
code review only

@andreadelrio
Copy link
Contributor

Design changes LGTM, thanks @qn895 !

@qn895 qn895 enabled auto-merge (squash) July 2, 2024 02:37
@kibana-ci
Copy link
Collaborator

💛 Build succeeded, but was flaky

Failed CI Steps

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id before after diff
dataVisualizer 732 788 +56

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id before after diff
@kbn/ml-query-utils 0 1 +1

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
aiops 558.9KB 558.9KB +4.0B
dataVisualizer 759.8KB 797.8KB +38.0KB
discover 809.1KB 809.3KB +130.0B
transform 632.5KB 632.5KB +4.0B
total +38.1KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id before after diff
dataVisualizer 23.0KB 24.1KB +1.0KB
Unknown metric groups

API count

id before after diff
@kbn/ml-query-utils 28 29 +1

async chunk count

id before after diff
dataVisualizer 14 20 +6

ESLint disabled line counts

id before after diff
dataVisualizer 73 76 +3

References to deprecated APIs

id before after diff
dataVisualizer 9 11 +2

Total ESLint disabled count

id before after diff
dataVisualizer 74 77 +3

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @qn895

@qn895 qn895 merged commit de027b8 into elastic:main Jul 2, 2024
19 checks passed
@kibanamachine kibanamachine added the backport:skip This commit does not require backporting label Jul 2, 2024
@peteharverson peteharverson changed the title [ML] Add Field statistics embeddable as panel in Dashboard [ML] Adds Field statistics embeddable as panel in Dashboard Jul 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport:skip This commit does not require backporting Feature:Dashboard Dashboard related features Feature:File and Index Data Viz ML file and index data visualizer impact:medium Addressing this issue will have a medium level of impact on the quality/strength of our product. loe:medium Medium Level of Effort :ml release_note:feature Makes this part of the condensed release notes Team:Presentation Presentation Team for Dashboard, Input Controls, and Canvas v8.15.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.