[ml] migrate file_data_visualizer/import route to file_upload plugin #89640

nreese · 2021-01-28T21:22:09Z

This PR starts the process of extracting ml file_data_visualizer into a reusable file_upload plugin. This PR starts small and just extracts import route. PR also extracts all typings, schemas, utility functions, and and telemetry.

elasticmachine · 2021-01-28T21:22:12Z

Pinging @elastic/ml-ui (:ml)

nreese · 2021-01-28T22:33:27Z

@elasticmachine merge upstream

…e_upload

x-pack/plugins/ml/server/routes/file_data_visualizer.ts

x-pack/plugins/file_upload/server/telemetry/mappings.ts

Bamieh · 2021-02-01T08:15:14Z

x-pack/plugins/file_upload/server/telemetry/telemetry.ts

@@ -74,8 +74,8 @@ export async function updateTelemetry(internalRepo?: ISavedObjectsRepository) {

 function incrementCounts(telemetry: Telemetry) {


Any reason not using SavedObjectsRepository.incrementCounter?

Recently we've added support to increment multiple fields with custom increments:

incrementCounter<T = unknown>(type: string, id: string, counterFields: Array<string | SavedObjectsIncrementCounterField>, options?: SavedObjectsIncrementCounterOptions): Promise<SavedObject<T>>;

@Bamieh This PR is migrating code from ML to a separate plugin, trying to change as little as possible. I can create an issue to track using SavedObjectsRepository.incrementCounter but would like to leave any refactoring out of this initial file_upload creation pull request.

Bamieh · 2021-02-01T08:17:04Z

x-pack/plugins/telemetry_collection_xpack/schema/xpack_plugins.json

@@ -1771,10 +1771,14 @@
        }
      }
    },
-    "fileUploadTelemetry": {
+    "fileUpload": {


Can we merge both telemetry types? fileUpload and fileUploadTelemetry

The next step of the process of migrating ML file_upload plugin will be to remove the maps_file_upload plugin. That will remove will fileUploadTelemetry type. For the new file_upload plugin, I wanted to start with a clean namespace to avoid any problems processing telemetry for the old plugin.

Bamieh

Core changes LGTM (including telemetry). I added a couple of questions before approving

peteharverson

LGTM. Tested latest edits, and confirmed the issue around telemetry has been resolved, so I was able to successfully import files from the ML file data visualizer page.

Bamieh

LGTM

jgowdyelastic

LGTM

nreese · 2021-02-02T13:30:50Z

@elasticmachine merge upstream

kibanamachine · 2021-02-02T15:13:28Z

💚 Build Succeeded

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`ml`	1734	1736	+2

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`mapsFileUpload`	825.7KB	825.8KB	+5.0B
`ml`	6.6MB	6.6MB	+2.1KB
total			+2.1KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`ml`	57.2KB	57.1KB	-45.0B

Saved Objects .kibana field count

Every field in each saved object type adds overhead to Elasticsearch. Kibana needs to keep the total field count below Elasticsearch's default limit of 1000 fields. Only specify field mappings for the fields you wish to search on or query. See https://www.elastic.co/guide/en/kibana/master/development-plugin-saved-objects.html#_mappings

id	before	after	diff
`file-upload-usage-collection-telemetry`	-	3	+3
`ml-telemetry`	3	-	-3
total			-0

History

💛 Build #102616 was flaky 6d3bf3a
💔 Build #102601 failed c4d050e
💚 Build #102328 succeeded 0473f5a
💚 Build #102325 succeeded 26e05b2
💚 Build #102255 succeeded 0b31512

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

…lastic#89640) * migrate file_upload plugin to maps_file_upload * update plugins list * migrate ml import endpoint * migrate ml telemetry to file_upload plugin * add fileUpload plugin to ml * add TS project * update ML to use file_upload endpoint * move types to file_upload plugin * ignore error * clean up * i18n clean-up * remove schemas from ml * remove usageCollection from ml * node scripts/build_plugin_list_docs * update telemety collector * revert changes to ingestPipeline schema * change name of TELEMETRY_DOC_ID to unique value * remove ImportFile from ml/server/routes/apidoc.json * fix typo in x=pack/tsconfig.json Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> # Conflicts: # src/plugins/telemetry/schema/oss_plugins.json # x-pack/plugins/telemetry_collection_xpack/schema/xpack_plugins.json

…-ml-jobs * 'master' of github.com:elastic/kibana: (254 commits) [Security Solution] [Detections] Remove allow_no_indices to prevent error being thrown in response of field capabilities (elastic#89927) Skip test for cloud (elastic#89450) [Fleet] Fix duplicate data streams being shown in UI (elastic#89812) Bump package dependencies (elastic#90034) [App Search] DRY helper for encoding/decoding routes that can have special characters in params (elastic#89811) TypeScript project references for Observability plugin (elastic#89320) [SearchSource] Combine sort and parent fields when serializing (elastic#89808) Made imports static (elastic#89935) [ml] migrate file_data_visualizer/import route to file_upload plugin (elastic#89640) [Discover] Adapt default column behavior (elastic#89826) Round start and end values (elastic#89030) Rename getProxyAgents to getCustomAgents (elastic#89813) [Form lib] UseField `onError` listener (elastic#89895) [APM] use latency sum instead of avg for impact (elastic#89990) migrate more core-owned plugins to tsproject ref (elastic#89975) [Logs UI] Load <LogStream> entries via async searches (elastic#86899) [APM] Abort browser requests when appropriate (elastic#89557) [Alerting] Allow user to select existing connector of same type when fixing broken connector (elastic#89062) [Data Table] Use shared CSV export mechanism (elastic#89702) chore(NA): improve logic check when installing Bazel tools (elastic#89634) ...

…89640) (#90075) * migrate file_upload plugin to maps_file_upload * update plugins list * migrate ml import endpoint * migrate ml telemetry to file_upload plugin * add fileUpload plugin to ml * add TS project * update ML to use file_upload endpoint * move types to file_upload plugin * ignore error * clean up * i18n clean-up * remove schemas from ml * remove usageCollection from ml * node scripts/build_plugin_list_docs * update telemety collector * revert changes to ingestPipeline schema * change name of TELEMETRY_DOC_ID to unique value * remove ImportFile from ml/server/routes/apidoc.json * fix typo in x=pack/tsconfig.json Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> # Conflicts: # src/plugins/telemetry/schema/oss_plugins.json # x-pack/plugins/telemetry_collection_xpack/schema/xpack_plugins.json

nreese added 12 commits January 25, 2021 10:07

migrate file_upload plugin to maps_file_upload

f19cece

update plugins list

005170b

migrate ml import endpoint

be1e122

merge with master

1b12e4c

migrate ml telemetry to file_upload plugin

179e7ff

add fileUpload plugin to ml

1da2023

Merge branch 'master' of github.com:elastic/kibana into ml_file_upload

89bbe88

add TS project

4d9680c

Merge branch 'master' of github.com:elastic/kibana into ml_file_upload

d1073bf

update ML to use file_upload endpoint

044ec4f

move types to file_upload plugin

de0ce16

ignore error

8e1c90e

nreese added chore :ml v8.0.0 release_note:skip Skip the PR/issue when compiling release notes Feature:File Upload v7.12.0 labels Jan 28, 2021

nreese requested a review from a team as a code owner January 28, 2021 21:22

nreese requested review from peteharverson and jgowdyelastic January 28, 2021 21:22

nreese added 5 commits January 28, 2021 14:26

clean up

dd57ecd

i18n clean-up

7edbade

remove schemas from ml

1079c41

remove usageCollection from ml

ed0e477

node scripts/build_plugin_list_docs

3bc38e2

kibanamachine and others added 2 commits January 28, 2021 17:33

Merge branch 'master' into ml_file_upload

d7c3811

update telemety collector

3b87cbf

Merge branch 'ml_file_upload' of github.com:nreese/kibana into ml_fil…

baeb6db

…e_upload

nreese requested a review from a team as a code owner January 28, 2021 23:56

peteharverson reviewed Jan 29, 2021

View reviewed changes

x-pack/plugins/ml/server/routes/file_data_visualizer.ts Show resolved Hide resolved

x-pack/plugins/file_upload/server/telemetry/mappings.ts Show resolved Hide resolved

nreese added 3 commits January 29, 2021 10:53

revert changes to ingestPipeline schema

0b31512

change name of TELEMETRY_DOC_ID to unique value

26e05b2

remove ImportFile from ml/server/routes/apidoc.json

0473f5a

nreese requested a review from peteharverson January 29, 2021 20:06

Bamieh reviewed Feb 1, 2021

View reviewed changes

peteharverson approved these changes Feb 1, 2021

View reviewed changes

nreese added 2 commits February 1, 2021 08:16

merge with master

c4d050e

fix typo in x=pack/tsconfig.json

6d3bf3a

Bamieh approved these changes Feb 2, 2021

View reviewed changes

jgowdyelastic approved these changes Feb 2, 2021

View reviewed changes

Merge branch 'master' into ml_file_upload

82b4a1b

nreese merged commit d601090 into elastic:master Feb 2, 2021

This was referenced Feb 2, 2021

[7.x] [ml] migrate file_data_visualizer/import route to file_upload plugin (#89640) #90043

Closed

[7.x] [ml] migrate file_data_visualizer/import route to file_upload plugin (#89640) #90075

Merged

nreese mentioned this pull request Feb 4, 2021

[Maps] remove maps_file_upload plugin and fold public folder into file_upload plugin #90292

Merged

peteharverson mentioned this pull request Mar 3, 2021

[ML] Switch file data visualizer to use new file upload plugin for data import #35947

Closed

4 tasks

nreese mentioned this pull request Mar 4, 2021

File upload missing in homepage #93623

Closed

qn895 mentioned this pull request Jun 4, 2021

[Meta] Move Data Visualizer to its own plugin #101435

Closed

25 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ml] migrate file_data_visualizer/import route to file_upload plugin #89640

[ml] migrate file_data_visualizer/import route to file_upload plugin #89640

nreese commented Jan 28, 2021 •

edited

elasticmachine commented Jan 28, 2021

nreese commented Jan 28, 2021

Bamieh Feb 1, 2021

nreese Feb 1, 2021

Bamieh Feb 1, 2021

nreese Feb 1, 2021

Bamieh left a comment

peteharverson left a comment

Bamieh left a comment

jgowdyelastic left a comment

nreese commented Feb 2, 2021

kibanamachine commented Feb 2, 2021

		@@ -74,8 +74,8 @@ export async function updateTelemetry(internalRepo?: ISavedObjectsRepository) {

		function incrementCounts(telemetry: Telemetry) {

[ml] migrate file_data_visualizer/import route to file_upload plugin #89640

[ml] migrate file_data_visualizer/import route to file_upload plugin #89640

Conversation

nreese commented Jan 28, 2021 • edited

elasticmachine commented Jan 28, 2021

nreese commented Jan 28, 2021

Bamieh Feb 1, 2021

Choose a reason for hiding this comment

nreese Feb 1, 2021

Choose a reason for hiding this comment

Bamieh Feb 1, 2021

Choose a reason for hiding this comment

nreese Feb 1, 2021

Choose a reason for hiding this comment

Bamieh left a comment

Choose a reason for hiding this comment

peteharverson left a comment

Choose a reason for hiding this comment

Bamieh left a comment

Choose a reason for hiding this comment

jgowdyelastic left a comment

Choose a reason for hiding this comment

nreese commented Feb 2, 2021

kibanamachine commented Feb 2, 2021

💚 Build Succeeded

Metrics [docs]

Module Count

Async chunks

Page load bundle

Saved Objects .kibana field count

History

nreese commented Jan 28, 2021 •

edited