[ML] Adds validation of field selected for log pattern analysis #162319

jgowdyelastic · 2023-07-20T12:24:54Z

Uses the recently created category validation package to perform validation on the field selected for pattern analysis.

If the field is considered unsuitable for categorization, a warning callout is displayed which lists the reasons it is unsuitable.
If the field is suitable, no callout is displayed.

Other changes:

Adds the selected field to the URL state, so it is remembered on page refresh.
If no field is in the URL, it will look for a field called message in the data view and auto select it.
renames the ML route /jobs/categorization_field_examples to /jobs/categorization_field_validation as it is a more accurate name and it's consistent with the newly added route in AIOPs.

Log Pattern Analysis page in ML

Log Pattern Analysis flyout in Discover

…fix'

jgowdyelastic · 2023-07-25T11:34:17Z

x-pack/plugins/aiops/server/routes/error_wrapper.ts

+import { boomify, isBoom } from '@hapi/boom';
+import { ResponseError, CustomHttpResponseOptions } from '@kbn/core/server';
+
+export function wrapError(error: any): CustomHttpResponseOptions<ResponseError> {


A duplicate of the function in ML.
Could possibly be moved to a package if we ever need it in another plugin.

The same is already in multiple plugins. We already have @kbn/ml-error-utils and could move it there, but I think for now this package is only used for client side utils, so we need to refactor that to be split into server/client side code to not unnecessarily blow up client side bundle sizes.

…d-validation

qn895 · 2023-07-27T16:39:05Z

x-pack/plugins/aiops/common/api/log_categorization/schema.ts

+import { i18n } from '@kbn/i18n';
+import { isRuntimeField } from '@kbn/ml-runtime-field-utils';
+
+export const runtimeMappingsSchema = schema.object(


Wonder if we should add runtimeMappingsSchema to ml-runtime-field-utils as we do use it in several plugins 🤔

Moved in 40fb370

I've reverted this move. I caused a 200KB bundle increase in ML and Transforms.

x-pack/plugins/aiops/public/components/log_categorization/use_categorize_request.ts

peteharverson · 2023-07-27T17:02:11Z

x-pack/plugins/aiops/public/components/log_categorization/log_categorization_for_flyout.tsx

@@ -217,6 +251,8 @@ export const LogCategorizationFlyout: FC<LogCategorizationPageProps> = ({
        </EuiFlexGroup>
      </EuiFlyoutHeader>
      <EuiFlyoutBody data-test-subj="mlJobSelectorFlyoutBody">
+        <FieldValidationCallout validationResults={fieldValidationResult} />


As discussed, looks like the callout isn't getting displayed if no categories are found - for example with field6 from the categorization_functional_test data set. In this case the field is populated so the message here is misleading:

With this example, the field does not produce any warnings because the data is tokenized correctly but it also does not produce any categories. I suspect this is because every doc contains the same data.

walterra · 2023-07-28T07:37:39Z

x-pack/plugins/aiops/public/components/log_categorization/category_validation_callout.tsx

+
+  return (
+    <>
+      {validationResults !== null ? (


Guess this ternary can be removed since the same check is done above and returns early.

Updated in 914c43d

walterra · 2023-07-28T07:54:12Z

x-pack/plugins/aiops/server/routes/error_wrapper.ts

+import { boomify, isBoom } from '@hapi/boom';
+import { ResponseError, CustomHttpResponseOptions } from '@kbn/core/server';
+
+export function wrapError(error: any): CustomHttpResponseOptions<ResponseError> {


The same is already in multiple plugins. We already have @kbn/ml-error-utils and could move it there, but I think for now this package is only used for client side utils, so we need to refactor that to be split into server/client side code to not unnecessarily blow up client side bundle sizes.

walterra · 2023-07-28T07:55:33Z

x-pack/plugins/aiops/tsconfig.json

    "@kbn/ml-date-picker",
    "@kbn/ml-error-utils",
    "@kbn/ml-is-defined",
    "@kbn/ml-is-populated-object",
+    "@kbn/ml-kibana-theme",
+    "@kbn/ml-kibana-theme",


Duplicate entry here.

Updated in 914c43d

…elastic/kibana into pattern-analysis-field-validation

…fix'

peteharverson

Latest changes LGTM

…fix'

qn895 · 2023-07-28T14:59:23Z

LGTM 🎉

kibana-ci · 2023-07-28T15:07:34Z

💚 Build Succeeded

Buildkite Build
Commit: 96a6c88

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`aiops`	427	435	+8

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`aiops`	531.6KB	535.2KB	+3.6KB
`ml`	3.4MB	3.4MB	+15.0B
total			+3.6KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`ml`	73.8KB	73.9KB	+22.0B

Unknown metric groups

API count

id	before	after	diff
`@kbn/ml-category-validator`	32	37	+5

References to deprecated APIs

id	before	after	diff
`aiops`	12	2	-10

History

💔 Build #145511 failed 092a206
💚 Build #145471 succeeded 04f6564
💔 Build #145451 failed 3f10c3f
💔 Build #145430 failed 141b182
💔 Build #145427 failed 914c43d

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @jgowdyelastic

walterra

Great you brought down the bundle size again, latest changes LGTM!

Uses the recently created [category validation package](elastic#161261) to perform validation on the field selected for pattern analysis. If the field is considered unsuitable for categorization, a warning callout is displayed which lists the reasons it is unsuitable. If the field is suitable, no callout is displayed. Other changes: - Adds the selected field to the URL state, so it is remembered on page refresh. - If no field is in the URL, it will look for a field called `message` in the data view and auto select it. - renames the ML route `/jobs/categorization_field_examples` to `/jobs/categorization_field_validation` as it is a more accurate name and it's consistent with the newly added route in AIOPs. **Log Pattern Analysis page in ML** ![image](https://github.com/elastic/kibana/assets/22172091/c0dfda8b-bc34-48b7-9e71-8bae9e65bdf3) **Log Pattern Analysis flyout in Discover** ![image](https://github.com/elastic/kibana/assets/22172091/b4d251f3-bae6-424f-9891-bda57ba1673d) --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

jgowdyelastic and others added 18 commits July 20, 2023 13:23

[ML] Log pattern analysis field validation

9e290aa

Merge branch 'main' into pattern-analysis-field-validation

bacf7d8

conflicts after merge

5963371

removing commented code

db972cf

[CI] Auto-commit changed files from 'node scripts/lint_ts_projects --…

f174123

…fix'

conflicts

6478436

fixing more conflicts

93fecdc

translation id

f401d4c

fixing query and cancel

b1ea5a0

making examples optional in the endpoint response

c9e025a

fixing optional examples

3868966

Merge branch 'main' into pattern-analysis-field-validation

2a8e2d2

text change

f5801ae

Merge branch 'main' into pattern-analysis-field-validation

a42b7bb

adding url state for field selection

d801056

commented code

34066a5

auto select message field

efa072c

Merge branch 'main' into pattern-analysis-field-validation

9a970ff

peteharverson mentioned this pull request Jul 25, 2023

[ML] [META] Log pattern analysis improvements #140715

Open

12 tasks

jgowdyelastic commented Jul 25, 2023

View reviewed changes

variable rename

8000d37

jgowdyelastic requested review from walterra and peteharverson July 25, 2023 11:35

jgowdyelastic self-assigned this Jul 25, 2023

jgowdyelastic requested a review from alvarezmelissa87 July 25, 2023 11:35

jgowdyelastic added release_note:enhancement :ml Feature:ML/AIOps ML AIOps features: Change Point Detection, Log Pattern Analysis, Log Rate Analysis v8.10.0 labels Jul 25, 2023

jgowdyelastic marked this pull request as ready for review July 25, 2023 11:36

jgowdyelastic removed the request for review from a team July 27, 2023 09:37

Merge remote-tracking branch 'origin/main' into pattern-analysis-fiel…

ed77ab5

…d-validation

peteharverson requested review from qn895 and removed request for alvarezmelissa87 July 27, 2023 16:27

qn895 reviewed Jul 27, 2023

View reviewed changes

x-pack/plugins/aiops/public/components/log_categorization/use_categorize_request.ts Outdated Show resolved Hide resolved

peteharverson reviewed Jul 27, 2023

View reviewed changes

walterra reviewed Jul 28, 2023

View reviewed changes

jgowdyelastic and others added 6 commits July 28, 2023 09:35

Merge branch 'main' into pattern-analysis-field-validation

5864f72

Merge branch 'pattern-analysis-field-validation' of github.com:jgowdy…

d02a6cc

…elastic/kibana into pattern-analysis-field-validation

moving runtime mapping schema to package

40fb370

small changes based on review

914c43d

[CI] Auto-commit changed files from 'node scripts/lint_ts_projects --…

141b182

…fix'

Merge branch 'main' into pattern-analysis-field-validation

3f10c3f

peteharverson approved these changes Jul 28, 2023

View reviewed changes

jgowdyelastic and others added 7 commits July 28, 2023 11:26

adding i18n tracking for runtime_field_utils

04f6564

reverting runtimeMappingsSchema schema move

8903063

jsdoc comments

1de953b

[CI] Auto-commit changed files from 'node scripts/lint_ts_projects --…

092a206

…fix'

translation id

8704072

removing runtime_field_utils from translations

922b8da

further clean up after code revert

96a6c88

qn895 approved these changes Jul 28, 2023

View reviewed changes

jgowdyelastic merged commit 716ecb8 into elastic:main Jul 28, 2023
21 checks passed

kibanamachine added the backport:skip This commit does not require backporting label Jul 28, 2023

walterra reviewed Jul 28, 2023

View reviewed changes

peteharverson changed the title ~~[ML] Log pattern analysis field validation~~ [ML] Adds validation of field selected for log pattern analysis Aug 21, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ML] Adds validation of field selected for log pattern analysis #162319

[ML] Adds validation of field selected for log pattern analysis #162319

jgowdyelastic commented Jul 20, 2023 •

edited

jgowdyelastic Jul 25, 2023

walterra Jul 28, 2023

qn895 Jul 27, 2023

jgowdyelastic Jul 28, 2023

jgowdyelastic Jul 28, 2023

peteharverson Jul 27, 2023

jgowdyelastic Jul 28, 2023

walterra Jul 28, 2023

jgowdyelastic Jul 28, 2023

walterra Jul 28, 2023

walterra Jul 28, 2023

jgowdyelastic Jul 28, 2023

peteharverson left a comment

qn895 commented Jul 28, 2023

kibana-ci commented Jul 28, 2023

API count

References to deprecated APIs

walterra left a comment

[ML] Adds validation of field selected for log pattern analysis #162319

[ML] Adds validation of field selected for log pattern analysis #162319

Conversation

jgowdyelastic commented Jul 20, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

peteharverson left a comment

Choose a reason for hiding this comment

qn895 commented Jul 28, 2023

kibana-ci commented Jul 28, 2023

💚 Build Succeeded

Metrics [docs]

Module Count

Async chunks

Page load bundle

API count

References to deprecated APIs

History

walterra left a comment

Choose a reason for hiding this comment

jgowdyelastic commented Jul 20, 2023 •

edited