[ML] Jobs import and export #101037

jgowdyelastic · 2021-06-01T10:12:01Z

Adding export and import of anomaly detection and data frame analytics jobs to the ML section of the Stack Management page.
An exported job file contains a single or array of job configs. In the case of anomaly detector jobs, a wrapping object is used to contain the job and datafeed e.g.

[
  {
    "job: {...}
    "datafeed": {...}
  }
]

Export

Import

When an anomaly detection file has been selected, the indices in the datafeed are checked to ensure that matching index patterns exist. This is based on a requirement of the new job wizards where index patterns need to exist in order to clone a job.

Selecting a file which does not contain valid ML jobs or contains a mixture of anomaly detection and data frame analytic jobs will cause an error:

Also adds a new bulk create endpoint for anomaly detection jobs which takes the same shaped payload as the exported file format.

POST /api/ml/jobs/bulk_create

Checklist

Delete any items that are not applicable to this PR.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios
Any UI touched in this PR is usable by keyboard only (learn more about keyboard accessibility)
Any UI touched in this PR does not create any new axe failures (run axe in browser: FF, Chrome)

For maintainers

This was checked for breaking API changes and was labeled appropriately

jgowdyelastic · 2021-06-01T19:59:46Z

@elasticmachine merge upstream

…na into job-import-and-export

jgowdyelastic · 2021-06-07T13:39:58Z

@elasticmachine merge upstream

…na into job-import-and-export

jgowdyelastic · 2021-06-08T13:35:45Z

@elasticmachine merge upstream

jgowdyelastic · 2021-06-09T07:38:47Z

@elasticmachine merge upstream

jgowdyelastic · 2021-07-07T08:02:00Z

@elasticmachine merge upstream

kibanamachine · 2021-07-07T08:02:02Z

merge conflict between base and head

jgowdyelastic · 2021-07-08T07:54:11Z

@elasticmachine merge upstream

jgowdyelastic · 2021-07-12T09:06:25Z

@elasticmachine merge upstream

jgowdyelastic · 2021-07-21T14:31:00Z

@elasticmachine merge upstream

…na into job-import-and-export

alvarezmelissa87

Still seeing the same 400 error dest index must be empty when trying to start an imported DFA job even when destination index was edited in the import flyout. Looks like the destination index isn't actually getting updated before the job is imported.

What do you think about having the help text in the dest index editor let the user know that the job start will fail if they don't change the destination index?

@alvarezmelissa87

cf29161 fixes the dest index not being saved.

What do you think about having the help text in the dest index editor let the user know that the job start will fail if they don't change the destination index?

This is not what the help text lead me to believe in the DFA wizard. The warning text I'm showing is copied from the DFA wizard and is purposefully not an invalid state in the importer.
Do you think this needs to change and if a destination index already exists it should stop the user from importing?
The exists check could check to see if the index is empty too. This change would need to go into the DFA wizard too.

x-pack/plugins/ml/common/types/job_service.ts

...l/public/application/components/import_export_jobs/export_jobs_flyout/export_jobs_flyout.tsx

darnautov · 2021-07-22T08:02:44Z

...l/public/application/components/import_export_jobs/export_jobs_flyout/jobs_export_service.ts

+    const blob = new Blob([JSON.stringify(configsForExport, null, 2)], {
+      type: 'application/json',
+    });
+    saveAs(blob, 'ml_jobs.json');


nit: I think it might be convenient to have jobs type, stack version, and date in the file name, something like anomaly_detection_ml_jobs_7.14.0_20210808

Yeah this is a good idea to give these files a more specific name.
I've made a change here 330129a so the files are now either anomaly_detection_jobs.json or data_frame_analytics_jobs.json.

There are problems with adding the version to the file name as it may be coming from a mixed version cluster.
For now I think these names are definitely an improvement and we can make changes later on if we decide we should add more information.

x-pack/plugins/ml/server/routes/system.ts

darnautov

Code LGTM. I reckon it's worth adding some telemetry for this feature in a follow-up.

peteharverson

Tested latest changes and LGTM.

jgowdyelastic · 2021-07-22T11:07:44Z

@elasticmachine merge upstream

kibanamachine · 2021-07-22T13:24:52Z

💚 Build Succeeded

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`ml`	1729	1741	+12

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`apm`	39	-	-39
`banners`	9	-	-9
`core`	1080	-	-1080
`maps`	203	-	-203
`metricsEntities`	6	-	-6
`ml`	274	-	-274
`monitoring`	10	-	-10
`observability`	219	-	-219
`osquery`	11	-	-11
`remoteClusters`	4	-	-4
`reporting`	132	-	-132
`rollup`	20	-	-20
`ruleRegistry`	60	-	-60
`runtimeFields`	19	-	-19
`savedObjectsTagging`	50	-	-50
`security`	51	-	-51
`securitySolution`	1130	-	-1130
`snapshotRestore`	22	-	-22
`stackAlerts`	4	-	-4
`taskManager`	25	-	-25
`telemetryCollectionXpack`	1	-	-1
`timelines`	763	-	-763
`triggersActionsUi`	228	-	-228
`uiActionsEnhanced`	147	-	-147
total			-4507

Any counts in public APIs

Total count of every any typed public API. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats any for more detailed information.

id	before	after	diff
`core`	148	-	-148
`maps`	2	-	-2
`ml`	10	-	-10
`reporting`	1	-	-1
`securitySolution`	8	-	-8
`snapshotRestore`	1	-	-1
`timelines`	6	-	-6
`triggersActionsUi`	1	-	-1
`uiActionsEnhanced`	2	-	-2
total			-179

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`ml`	5.9MB	5.9MB	+43.0KB

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id	before	after	diff
`apm`	30	-	-30
`core`	31	-	-31
`maps`	11	-	-11
`metricsEntities`	1	-	-1
`ml`	33	-	-33
`monitoring`	2	-	-2
`observability`	10	-	-10
`reporting`	14	-	-14
`ruleRegistry`	9	-	-9
`runtimeFields`	2	-	-2
`security`	6	-	-6
`securitySolution`	28	-	-28
`snapshotRestore`	1	-	-1
`taskManager`	8	-	-8
`timelines`	25	-	-25
`triggersActionsUi`	19	-	-19
`uiActionsEnhanced`	10	-	-10
total			-240

Unknown metric groups

API count

id	before	after	diff
`apm`	39	-	-39
`banners`	9	-	-9
`core`	2359	-	-2359
`maps`	204	-	-204
`metricsEntities`	9	-	-9
`ml`	278	-	-278
`monitoring`	10	-	-10
`observability`	219	-	-219
`osquery`	11	-	-11
`remoteClusters`	4	-	-4
`reporting`	133	-	-133
`rollup`	20	-	-20
`ruleRegistry`	60	-	-60
`runtimeFields`	24	-	-24
`savedObjectsTagging`	54	-	-54
`security`	112	-	-112
`securitySolution`	1181	-	-1181
`snapshotRestore`	23	-	-23
`spaces`	106	-	-106
`stackAlerts`	4	-	-4
`taskManager`	52	-	-52
`telemetryCollectionXpack`	1	-	-1
`timelines`	882	-	-882
`triggersActionsUi`	237	-	-237
`uiActionsEnhanced`	205	-	-205
total			-6236

References to deprecated APIs

id	before	after	diff
`apm`	7	-	-7
`maps`	592	-	-592
`ml`	140	-	-140
`observability`	34	-	-34
`security`	2	-	-2
`securitySolution`	919	-	-919
`stackAlerts`	104	-	-104
`timelines`	76	-	-76
`transform`	16	-	-16
`uptime`	11	-	-11
`urlDrilldown`	18	-	-18
total			-1919

History

💔 Build #140270 failed 330129a
💔 Build #140184 failed cf29161
💚 Build #140110 succeeded a95b667
💔 Build #140091 failed 62f8068
💔 Build #140067 failed 6276a87

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @jgowdyelastic

alvarezmelissa87

Latest changes LGTM ⚡

* [ML] Jobs import and export * adding dfa job import/export * job id validation * fixing conflicts * adding index pattern validation * updates to job list * validation changes * refactoring job id editing * validating check * file read error * translation ids * duplicate translation id * small fix * updating route and schema * code clean up * catching and displaying job creation errors * useMemo for notifications * small refactor * cleaning up types * adding job import success toast * adding exporting toast * small changes based on review * adding loading spinner * changing util files to services * code changes baased on review * adding missing bind change * file rename * adding confirm for tab changes * adding destination index checks * grouping import errors * fixing translation ids * fixing dest index update * small changes based on review * renaming exported file name Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

kibanamachine · 2021-07-22T15:00:22Z

💚 Backport successful

Status	Branch	Result
✅	7.x

This backport PR will be merged automatically after passing CI.

* [ML] Jobs import and export * adding dfa job import/export * job id validation * fixing conflicts * adding index pattern validation * updates to job list * validation changes * refactoring job id editing * validating check * file read error * translation ids * duplicate translation id * small fix * updating route and schema * code clean up * catching and displaying job creation errors * useMemo for notifications * small refactor * cleaning up types * adding job import success toast * adding exporting toast * small changes based on review * adding loading spinner * changing util files to services * code changes baased on review * adding missing bind change * file rename * adding confirm for tab changes * adding destination index checks * grouping import errors * fixing translation ids * fixing dest index update * small changes based on review * renaming exported file name Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: James Gowdy <jgowdy@elastic.co>

jgowdyelastic added 2 commits June 1, 2021 11:11

[ML] Jobs import and export

9891044

adding dfa job import/export

1f0ae57

kibanamachine and others added 2 commits June 1, 2021 15:59

Merge branch 'master' into job-import-and-export

4fee167

Merge branch 'job-import-and-export' of github.com:jgowdyelastic/kiba…

8f993e2

…na into job-import-and-export

kibanamachine and others added 3 commits June 7, 2021 09:40

Merge branch 'master' into job-import-and-export

945f11c

Merge branch 'job-import-and-export' of github.com:jgowdyelastic/kiba…

66e2d8f

…na into job-import-and-export

job id validation

a89c7b9

Merge branch 'master' into job-import-and-export

805898b

jgowdyelastic changed the title ~~[ML] Jobs import and export~~ [ML] WIP Jobs import and export Jun 8, 2021

Merge branch 'master' into job-import-and-export

84320f2

jgowdyelastic added 3 commits July 7, 2021 09:05

Merge branch 'master' into job-import-and-export

0d401bb

fixing conflicts

d3f715b

adding index pattern validation

dcaab88

kibanamachine and others added 7 commits July 8, 2021 03:54

Merge branch 'master' into job-import-and-export

7a0cc04

updates to job list

3a62555

validation changes

51b9d6a

refactoring job id editing

352b556

validating check

9dbbc4e

file read error

40c0962

translation ids

9cdc49e

kibanamachine and others added 2 commits July 12, 2021 05:06

Merge branch 'master' into job-import-and-export

cda83a8

duplicate translation id

1b1b1c4

jgowdyelastic added 3 commits July 20, 2021 09:32

file rename

195b20f

adding confirm for tab changes

de25728

adding destination index checks

74bee14

kibanamachine and others added 4 commits July 21, 2021 10:31

Merge branch 'master' into job-import-and-export

6276a87

grouping import errors

f45f3cf

Merge branch 'job-import-and-export' of github.com:jgowdyelastic/kiba…

62f8068

…na into job-import-and-export

fixing translation ids

a95b667

alvarezmelissa87 approved these changes Jul 21, 2021

View reviewed changes

alvarezmelissa87 requested changes Jul 21, 2021

View reviewed changes

fixing dest index update

cf29161

darnautov reviewed Jul 22, 2021

View reviewed changes

jgowdyelastic added 2 commits July 22, 2021 09:58

small changes based on review

cfad0ea

renaming exported file name

330129a

darnautov approved these changes Jul 22, 2021

View reviewed changes

peteharverson approved these changes Jul 22, 2021

View reviewed changes

Merge branch 'master' into job-import-and-export

7702291

alvarezmelissa87 approved these changes Jul 22, 2021

View reviewed changes

jgowdyelastic merged commit d9e236b into elastic:master Jul 22, 2021

jgowdyelastic deleted the job-import-and-export branch July 22, 2021 14:57

kibanamachine mentioned this pull request Jul 22, 2021

[7.x] [ML] Jobs import and export (#101037) #106557

Merged

peteharverson mentioned this pull request Jul 28, 2021

[ML] Check for calendars and filters lists during anomaly detection job export / import #106976

Closed

lcawl mentioned this pull request Aug 11, 2021

[ML] Update intro text for ML in Stack Management #108280

Merged

This was referenced Aug 23, 2021

[7.x][DOCS] Import and export jobs elastic/stack-docs#1791

Merged

[DOCS] Import and export jobs elastic/stack-docs#1792

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ML] Jobs import and export #101037

[ML] Jobs import and export #101037

jgowdyelastic commented Jun 1, 2021 •

edited

jgowdyelastic commented Jun 1, 2021

jgowdyelastic commented Jun 7, 2021

jgowdyelastic commented Jun 8, 2021

jgowdyelastic commented Jun 9, 2021

jgowdyelastic commented Jul 7, 2021

kibanamachine commented Jul 7, 2021

jgowdyelastic commented Jul 8, 2021

jgowdyelastic commented Jul 12, 2021

jgowdyelastic commented Jul 21, 2021

alvarezmelissa87 left a comment •

edited by jgowdyelastic

darnautov Jul 22, 2021

jgowdyelastic Jul 22, 2021 •

edited

darnautov left a comment

peteharverson left a comment

jgowdyelastic commented Jul 22, 2021

kibanamachine commented Jul 22, 2021

API count

References to deprecated APIs

alvarezmelissa87 left a comment

kibanamachine commented Jul 22, 2021

[ML] Jobs import and export #101037

[ML] Jobs import and export #101037

Conversation

jgowdyelastic commented Jun 1, 2021 • edited

Checklist

For maintainers

jgowdyelastic commented Jun 1, 2021

jgowdyelastic commented Jun 7, 2021

jgowdyelastic commented Jun 8, 2021

jgowdyelastic commented Jun 9, 2021

jgowdyelastic commented Jul 7, 2021

kibanamachine commented Jul 7, 2021

jgowdyelastic commented Jul 8, 2021

jgowdyelastic commented Jul 12, 2021

jgowdyelastic commented Jul 21, 2021

alvarezmelissa87 left a comment • edited by jgowdyelastic

Choose a reason for hiding this comment

darnautov Jul 22, 2021

Choose a reason for hiding this comment

jgowdyelastic Jul 22, 2021 • edited

Choose a reason for hiding this comment

darnautov left a comment

Choose a reason for hiding this comment

peteharverson left a comment

Choose a reason for hiding this comment

jgowdyelastic commented Jul 22, 2021

kibanamachine commented Jul 22, 2021

💚 Build Succeeded

Metrics [docs]

Module Count

Public APIs missing comments

Any counts in public APIs

Async chunks

Public APIs missing exports

API count

References to deprecated APIs

History

alvarezmelissa87 left a comment

Choose a reason for hiding this comment

kibanamachine commented Jul 22, 2021

💚 Backport successful

jgowdyelastic commented Jun 1, 2021 •

edited

alvarezmelissa87 left a comment •

edited by jgowdyelastic

jgowdyelastic Jul 22, 2021 •

edited