Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Maps] [File upload] Geojson upload #36410

Merged
merged 139 commits into from Jun 10, 2019

Conversation

kindsun
Copy link
Contributor

@kindsun kindsun commented May 9, 2019

Replaces #34748

Adds UI for geojson upload on top of the file upload plugin. Specifically, provides the following capabilities:

  • Add file upload source to maps app
  • Add new import file workflow
  • Migrate UI code to file upload plugin and make generic
  • Update file upload plugin API to handle callbacks for fileupload events
  • Update file upload plugin API to handle callbacks for index events
  • Update file upload plugin API to handle callbacks for file parse events
  • Upload geojson files and save in local state
  • Add trigger for later indexing (after file review)
  • Allow specification of index name
  • Modify existing index if provided or create new index if doesn't exist
  • Allow choice between geo_point and geo_shape mapping
  • Allow custom index patterns
  • Permit geojson previewing of file
  • Remove geojson layer and add as es search source on "Add"
  • Determine cause and fix occasional issue indexing geo_points
  • Add relevant file filters to prevent unusable uploads
  • Handle modification of same index vs. new index in layers preview/add
  • Bubble up Elasticsearch ingest errors and pass to client
  • Add new panel between add/create index and layer details for progress indication and failure display
  • Internationalize text

# Conflicts:
#	x-pack/plugins/maps/public/shared/layers/sources/source.js
@kindsun kindsun force-pushed the feature-file-upload-plugin branch from 9ffccc0 to 0526dc4 Compare June 6, 2019 19:18
@kindsun
Copy link
Contributor Author

kindsun commented Jun 6, 2019

After some offline conversation with @nreese, I've rolled back changes to the timing of add/remove layers. This will re-introduce the bug of layers changing colors as the geojson layer is removed and the ES layer is added. This should be potentially fixed by #37904. Additional work could also be done in store_actions.js to clean up async calls (and maybe add others) which could also help with back-to-back remove/add layer calls.

Scratch that. There's a fix in place with this PR. Colors are now filtered so that transient layer colors are not factored in to the next default color option.

@elasticmachine
Copy link
Contributor

💔 Build Failed

@elasticmachine
Copy link
Contributor

💔 Build Failed

@kindsun
Copy link
Contributor Author

kindsun commented Jun 7, 2019

retest

@elasticmachine
Copy link
Contributor

💔 Build Failed

@kindsun
Copy link
Contributor Author

kindsun commented Jun 7, 2019

retest

@elasticmachine
Copy link
Contributor

💔 Build Failed

@nreese
Copy link
Contributor

nreese commented Jun 7, 2019

Need to remove __kbn__feature_id__ from data. Its getting inserted into records

Screen Shot 2019-06-07 at 4 22 25 PM

Uploaded file

{
  "type": "FeatureCollection",
  "features": [
    {
      "type": "Feature",
      "properties": {
        "number_val" : 10
      },
      "geometry": {
        "type": "Point",
        "coordinates": [
          59.765625,
          64.47279382008166
        ]
      }
    },
    {
      "type": "Feature",
      "properties": {
        "number_val" : 12
      },
      "geometry": {
        "type": "Point",
        "coordinates": [
          106.5234375,
          51.17934297928927
        ]
      }
    }
  ]
}

@nreese
Copy link
Contributor

nreese commented Jun 7, 2019

Unable to load MultiPoint as geo_point

Screen Shot 2019-06-07 at 4 34 23 PM

file contents

{
  "type": "FeatureCollection",
  "features": [
    {
      "type": "Feature",
      "properties": {
        "number_val" : 10
      },
      "geometry": {
        "type": "MultiPoint",
        "coordinates": [
          [100.0, 0.0], [101.0, 1.0]
        ]
      }
    }
  ]
}

@kindsun kindsun force-pushed the feature-file-upload-plugin branch from 0526dc4 to aaa5953 Compare June 10, 2019 20:32
@elasticmachine
Copy link
Contributor

💔 Build Failed

@kindsun kindsun merged commit 31955b1 into feature-file-upload-plugin Jun 10, 2019
kindsun pushed a commit that referenced this pull request Jun 19, 2019
* Add file upload x-pack plugin

* Clean up

* Remove unneeded cluster config

* Remove unneeded test

* First pass basic telemetry (not connected).

* Basic telemetry connected

* Review feedback

* Revise telemetry to use savedObjectRepository. Capture metrics on app and file types

* Lots of cleanup, consolidation of logic

* Clean up, reorg

* Update telem tests and telem functions

* Add back import data model

* Clean up and update telemetry tests

* Fix telemetry test issues and update corresponding code

* Up chunk limit to 30 MB

* Add file upload telemetry to saved objects management builder

* Missing space

* Add descriptive comments to dynamic keys in telemetry fields

* [Maps] [File upload] Geojson upload (#36410)

* Client side basics

* File added and default named correctly

* Connect transient layer removal to file import component

* Simplify transient layer removal

* Move file import ui over to new file upload plugin and make more generic

* Add post-processing option to file upload. Make component json-specific

* Add flag for source indexing

* Revise import layer workflow to be separate from add layer workflow

* Differentiate between normal sources and import sources. Add back layer add/next button

* Update indexing boolean in component after file upload & parse

* First pass rough indexing from maps working on specific use cases

* Update parsing logic to handle geojson formats

* Index pattern added following index creation

* Pass onsuccess callback to file_upload to add layer to maps app

* Handle mulltipolygon type in es geo utils

* Add functionality to make es layer permanent and switch to edit panel

* Add to index if exists or create new

* Make plugin API more intuitive. Set up to handle remove action

* Pass transient removal call through to file import

* Clean up layer viewing logic for temp and perm layers

* Remove change source from import screen

* Add option to provide mappings array and pass geo_point and geo_shape to array

* Add support for multiple mappings select and index naming

* Match style of import file button to add layer for now

* Remove duplicate case handling resulting from merge

* Move geo processing logic over to file upload plugin for reuse

* Remove old geo_shape formatting from geo_point code

* Set default index data type. Remove unneeded stringify request logic

* Check for custom processor object which contains function

* Move file picker to separate component

* Some cleaning. Add geojson clean & validate code

* Catch file parsing errors and notify user

* Disable index type if valid file not referenced

* Set error messaging on invalid index name used

* Add index pattern checking logic and error handling

* Dynamically populate geo index options

* Set index data type earlier in the workflow. Don't duplicate requests

* Pass back index ready status from plugin and connect to layer next button

* Increase max bytes to ~50MB

* Don't parse files over max size & warn user. Also, remove toasts and warn similarly to other components

* Uploaded file default label: 'fileToImport' -> 'Import File'

* Expand out feature properties for mapping

* Pass through telemetry data to plugin back-end

* Clean up indexing flow. Separate creation of index patterns. Add new index pattern create callbacks

* Pass back info for indexing failures but don't connect to UI yet

* Fix telemetry test issues and update corresponding code

# Conflicts:
#	x-pack/plugins/file_upload/server/telemetry/telemetry.test.ts

* Add file upload telemetry to saved objects management builder

* Missing space

* Add descriptive comments to dynamic keys in telemetry fields

* Divide up the import layer add workflow on the maps side and tweak to avoid layer color change

* Fix bug affecting file preview replacement not updating coordinate index type

* Remove index pattern. Organize effects. General clean up

* preIndexTransform -> transformDetails

* Update proptypes

* Cleaning, organizing

* Add index name guidelines. Show conditionally

* Add file size, type guidance. Filter on file size, not chunk size. Small tweaks to file/index tips

* Zoom to layer extent on preview of imported data

* Revert "Revise import layer workflow to be separate from add layer workflow"

This reverts commit 3b35f53.

* Handle import file determination to accomodate add layer flow. Update card wording

* Decompose addLayer component into smaller pieces. Add placeholder import progress component

* Dynamically change footer button text for context

* Move import card to top of sources

* Get basic progress tracking in place

* Allow second layer (indexed layer) to get removed. Reconnect file remove. Small UI tweaks

* Add link to management for further index mods

* Fix i18n failures

* Add file parsing progress indicator & text

* Reset importView on source reset

* Add dynamic chunking to handle fluctuating data sizes common to geo features

* Don't duplicate index request if request in flight

* Modify json upload and import to use JS classes & react class component state

* Transition remaining file_upload components over to class/component structure with state

* Move functions into index_settings class

* Review feedback

* Add fetch with timeout

* Split out import editor into separate component and clean up logic

* Clean up add layer panel/button title logic

* More cleanup

* Tweaks to success and error handling flow

* Handle success/error handling on add/view indexed data

* Jump to indexing complete on error. Handle nothing returned to client app

* Update name/location of source select css file

* Update import source card border color and icon

* Suggest name for index based upon file name

* Add validation to auto-suggested index name/pattern

* Use constants for geo_point and geo_shape in mappings

* Update geojson upload card/description

* Catch-all review feedback

* Fix internationalization syntax errors

* Review feedback

* Get index names and patterns only when needed

* Make addLayer async again to fix zoom to extent issue, waits on just the async syncDataForLayer function (which is async)

* Remove panel description as derived state and shift to function

* Remove geojson fit to extent for now

* Remove unused class selector

* Remove unneeded i18 wrapper for what's already passed in as an i18 element

* Revise import state to be handle via redux. Some json upload args changes

* Review feedback. Some cleanup and bug fixes

* Roll back store actions changes and layer_addpanel changes related to color change

* Follow scss file naming conventions for source_select

* Review feedback

* Restore clobbered layer_control view to master state. Add source_select css

* Update import to use plugin local indexPatternService, not maps'

* Review feedback, mostly i18n. Also add index to scss path

* i18n translation updates

* Assign error message to values rather than error object

* Update getMapColors to filter out transient layer

* Wrap Feature as FeatureCollection in Maps

* Add jest tests for geo processing functions. Add fixes for single feature handling

* i18n

* Review feedback. Test cleanup/fixes

* Update layer add panel footer logic to still show when source not selected

* Fix issue of not recognizing MultiPoint type. Remove throw logic for now

* Update telemetry with newly required placeholder function

* Prevent external modification of nested geojson objects

* i18n translation updates

* Revert "Fix issue of not recognizing MultiPoint type. Remove throw logic for now"

This reverts commit d692f91.

* Revert "Prevent external modification of nested geojson objects"

This reverts commit 0ea9fd3.

* yarn.lock update

* [File upload] Remove dynamic fields from mappings, code and telemetry test (#38902)

* Remove dynamic fields from mappings, code and telemetry test

* Add file-upload-telemetry to spaces and es archiver test mappings

* Don't create telemetry saved object if none exists, create on first update instead

* Back out es archiver mappings update
kindsun pushed a commit to kindsun/kibana that referenced this pull request Jun 19, 2019
* Add file upload x-pack plugin

* Clean up

* Remove unneeded cluster config

* Remove unneeded test

* First pass basic telemetry (not connected).

* Basic telemetry connected

* Review feedback

* Revise telemetry to use savedObjectRepository. Capture metrics on app and file types

* Lots of cleanup, consolidation of logic

* Clean up, reorg

* Update telem tests and telem functions

* Add back import data model

* Clean up and update telemetry tests

* Fix telemetry test issues and update corresponding code

* Up chunk limit to 30 MB

* Add file upload telemetry to saved objects management builder

* Missing space

* Add descriptive comments to dynamic keys in telemetry fields

* [Maps] [File upload] Geojson upload (elastic#36410)

* Client side basics

* File added and default named correctly

* Connect transient layer removal to file import component

* Simplify transient layer removal

* Move file import ui over to new file upload plugin and make more generic

* Add post-processing option to file upload. Make component json-specific

* Add flag for source indexing

* Revise import layer workflow to be separate from add layer workflow

* Differentiate between normal sources and import sources. Add back layer add/next button

* Update indexing boolean in component after file upload & parse

* First pass rough indexing from maps working on specific use cases

* Update parsing logic to handle geojson formats

* Index pattern added following index creation

* Pass onsuccess callback to file_upload to add layer to maps app

* Handle mulltipolygon type in es geo utils

* Add functionality to make es layer permanent and switch to edit panel

* Add to index if exists or create new

* Make plugin API more intuitive. Set up to handle remove action

* Pass transient removal call through to file import

* Clean up layer viewing logic for temp and perm layers

* Remove change source from import screen

* Add option to provide mappings array and pass geo_point and geo_shape to array

* Add support for multiple mappings select and index naming

* Match style of import file button to add layer for now

* Remove duplicate case handling resulting from merge

* Move geo processing logic over to file upload plugin for reuse

* Remove old geo_shape formatting from geo_point code

* Set default index data type. Remove unneeded stringify request logic

* Check for custom processor object which contains function

* Move file picker to separate component

* Some cleaning. Add geojson clean & validate code

* Catch file parsing errors and notify user

* Disable index type if valid file not referenced

* Set error messaging on invalid index name used

* Add index pattern checking logic and error handling

* Dynamically populate geo index options

* Set index data type earlier in the workflow. Don't duplicate requests

* Pass back index ready status from plugin and connect to layer next button

* Increase max bytes to ~50MB

* Don't parse files over max size & warn user. Also, remove toasts and warn similarly to other components

* Uploaded file default label: 'fileToImport' -> 'Import File'

* Expand out feature properties for mapping

* Pass through telemetry data to plugin back-end

* Clean up indexing flow. Separate creation of index patterns. Add new index pattern create callbacks

* Pass back info for indexing failures but don't connect to UI yet

* Fix telemetry test issues and update corresponding code

# Conflicts:
#	x-pack/plugins/file_upload/server/telemetry/telemetry.test.ts

* Add file upload telemetry to saved objects management builder

* Missing space

* Add descriptive comments to dynamic keys in telemetry fields

* Divide up the import layer add workflow on the maps side and tweak to avoid layer color change

* Fix bug affecting file preview replacement not updating coordinate index type

* Remove index pattern. Organize effects. General clean up

* preIndexTransform -> transformDetails

* Update proptypes

* Cleaning, organizing

* Add index name guidelines. Show conditionally

* Add file size, type guidance. Filter on file size, not chunk size. Small tweaks to file/index tips

* Zoom to layer extent on preview of imported data

* Revert "Revise import layer workflow to be separate from add layer workflow"

This reverts commit 3b35f53.

* Handle import file determination to accomodate add layer flow. Update card wording

* Decompose addLayer component into smaller pieces. Add placeholder import progress component

* Dynamically change footer button text for context

* Move import card to top of sources

* Get basic progress tracking in place

* Allow second layer (indexed layer) to get removed. Reconnect file remove. Small UI tweaks

* Add link to management for further index mods

* Fix i18n failures

* Add file parsing progress indicator & text

* Reset importView on source reset

* Add dynamic chunking to handle fluctuating data sizes common to geo features

* Don't duplicate index request if request in flight

* Modify json upload and import to use JS classes & react class component state

* Transition remaining file_upload components over to class/component structure with state

* Move functions into index_settings class

* Review feedback

* Add fetch with timeout

* Split out import editor into separate component and clean up logic

* Clean up add layer panel/button title logic

* More cleanup

* Tweaks to success and error handling flow

* Handle success/error handling on add/view indexed data

* Jump to indexing complete on error. Handle nothing returned to client app

* Update name/location of source select css file

* Update import source card border color and icon

* Suggest name for index based upon file name

* Add validation to auto-suggested index name/pattern

* Use constants for geo_point and geo_shape in mappings

* Update geojson upload card/description

* Catch-all review feedback

* Fix internationalization syntax errors

* Review feedback

* Get index names and patterns only when needed

* Make addLayer async again to fix zoom to extent issue, waits on just the async syncDataForLayer function (which is async)

* Remove panel description as derived state and shift to function

* Remove geojson fit to extent for now

* Remove unused class selector

* Remove unneeded i18 wrapper for what's already passed in as an i18 element

* Revise import state to be handle via redux. Some json upload args changes

* Review feedback. Some cleanup and bug fixes

* Roll back store actions changes and layer_addpanel changes related to color change

* Follow scss file naming conventions for source_select

* Review feedback

* Restore clobbered layer_control view to master state. Add source_select css

* Update import to use plugin local indexPatternService, not maps'

* Review feedback, mostly i18n. Also add index to scss path

* i18n translation updates

* Assign error message to values rather than error object

* Update getMapColors to filter out transient layer

* Wrap Feature as FeatureCollection in Maps

* Add jest tests for geo processing functions. Add fixes for single feature handling

* i18n

* Review feedback. Test cleanup/fixes

* Update layer add panel footer logic to still show when source not selected

* Fix issue of not recognizing MultiPoint type. Remove throw logic for now

* Update telemetry with newly required placeholder function

* Prevent external modification of nested geojson objects

* i18n translation updates

* Revert "Fix issue of not recognizing MultiPoint type. Remove throw logic for now"

This reverts commit d692f91.

* Revert "Prevent external modification of nested geojson objects"

This reverts commit 0ea9fd3.

* yarn.lock update

* [File upload] Remove dynamic fields from mappings, code and telemetry test (elastic#38902)

* Remove dynamic fields from mappings, code and telemetry test

* Add file-upload-telemetry to spaces and es archiver test mappings

* Don't create telemetry saved object if none exists, create on first update instead

* Back out es archiver mappings update
kindsun pushed a commit that referenced this pull request Jun 19, 2019
* [File upload] New plugin: file upload (#36404)

* Add file upload x-pack plugin

* Clean up

* Remove unneeded cluster config

* Remove unneeded test

* First pass basic telemetry (not connected).

* Basic telemetry connected

* Review feedback

* Revise telemetry to use savedObjectRepository. Capture metrics on app and file types

* Lots of cleanup, consolidation of logic

* Clean up, reorg

* Update telem tests and telem functions

* Add back import data model

* Clean up and update telemetry tests

* Fix telemetry test issues and update corresponding code

* Up chunk limit to 30 MB

* Add file upload telemetry to saved objects management builder

* Missing space

* Add descriptive comments to dynamic keys in telemetry fields

* [Maps] [File upload] Geojson upload (#36410)

* Client side basics

* File added and default named correctly

* Connect transient layer removal to file import component

* Simplify transient layer removal

* Move file import ui over to new file upload plugin and make more generic

* Add post-processing option to file upload. Make component json-specific

* Add flag for source indexing

* Revise import layer workflow to be separate from add layer workflow

* Differentiate between normal sources and import sources. Add back layer add/next button

* Update indexing boolean in component after file upload & parse

* First pass rough indexing from maps working on specific use cases

* Update parsing logic to handle geojson formats

* Index pattern added following index creation

* Pass onsuccess callback to file_upload to add layer to maps app

* Handle mulltipolygon type in es geo utils

* Add functionality to make es layer permanent and switch to edit panel

* Add to index if exists or create new

* Make plugin API more intuitive. Set up to handle remove action

* Pass transient removal call through to file import

* Clean up layer viewing logic for temp and perm layers

* Remove change source from import screen

* Add option to provide mappings array and pass geo_point and geo_shape to array

* Add support for multiple mappings select and index naming

* Match style of import file button to add layer for now

* Remove duplicate case handling resulting from merge

* Move geo processing logic over to file upload plugin for reuse

* Remove old geo_shape formatting from geo_point code

* Set default index data type. Remove unneeded stringify request logic

* Check for custom processor object which contains function

* Move file picker to separate component

* Some cleaning. Add geojson clean & validate code

* Catch file parsing errors and notify user

* Disable index type if valid file not referenced

* Set error messaging on invalid index name used

* Add index pattern checking logic and error handling

* Dynamically populate geo index options

* Set index data type earlier in the workflow. Don't duplicate requests

* Pass back index ready status from plugin and connect to layer next button

* Increase max bytes to ~50MB

* Don't parse files over max size & warn user. Also, remove toasts and warn similarly to other components

* Uploaded file default label: 'fileToImport' -> 'Import File'

* Expand out feature properties for mapping

* Pass through telemetry data to plugin back-end

* Clean up indexing flow. Separate creation of index patterns. Add new index pattern create callbacks

* Pass back info for indexing failures but don't connect to UI yet

* Fix telemetry test issues and update corresponding code

# Conflicts:
#	x-pack/plugins/file_upload/server/telemetry/telemetry.test.ts

* Add file upload telemetry to saved objects management builder

* Missing space

* Add descriptive comments to dynamic keys in telemetry fields

* Divide up the import layer add workflow on the maps side and tweak to avoid layer color change

* Fix bug affecting file preview replacement not updating coordinate index type

* Remove index pattern. Organize effects. General clean up

* preIndexTransform -> transformDetails

* Update proptypes

* Cleaning, organizing

* Add index name guidelines. Show conditionally

* Add file size, type guidance. Filter on file size, not chunk size. Small tweaks to file/index tips

* Zoom to layer extent on preview of imported data

* Revert "Revise import layer workflow to be separate from add layer workflow"

This reverts commit 3b35f53.

* Handle import file determination to accomodate add layer flow. Update card wording

* Decompose addLayer component into smaller pieces. Add placeholder import progress component

* Dynamically change footer button text for context

* Move import card to top of sources

* Get basic progress tracking in place

* Allow second layer (indexed layer) to get removed. Reconnect file remove. Small UI tweaks

* Add link to management for further index mods

* Fix i18n failures

* Add file parsing progress indicator & text

* Reset importView on source reset

* Add dynamic chunking to handle fluctuating data sizes common to geo features

* Don't duplicate index request if request in flight

* Modify json upload and import to use JS classes & react class component state

* Transition remaining file_upload components over to class/component structure with state

* Move functions into index_settings class

* Review feedback

* Add fetch with timeout

* Split out import editor into separate component and clean up logic

* Clean up add layer panel/button title logic

* More cleanup

* Tweaks to success and error handling flow

* Handle success/error handling on add/view indexed data

* Jump to indexing complete on error. Handle nothing returned to client app

* Update name/location of source select css file

* Update import source card border color and icon

* Suggest name for index based upon file name

* Add validation to auto-suggested index name/pattern

* Use constants for geo_point and geo_shape in mappings

* Update geojson upload card/description

* Catch-all review feedback

* Fix internationalization syntax errors

* Review feedback

* Get index names and patterns only when needed

* Make addLayer async again to fix zoom to extent issue, waits on just the async syncDataForLayer function (which is async)

* Remove panel description as derived state and shift to function

* Remove geojson fit to extent for now

* Remove unused class selector

* Remove unneeded i18 wrapper for what's already passed in as an i18 element

* Revise import state to be handle via redux. Some json upload args changes

* Review feedback. Some cleanup and bug fixes

* Roll back store actions changes and layer_addpanel changes related to color change

* Follow scss file naming conventions for source_select

* Review feedback

* Restore clobbered layer_control view to master state. Add source_select css

* Update import to use plugin local indexPatternService, not maps'

* Review feedback, mostly i18n. Also add index to scss path

* i18n translation updates

* Assign error message to values rather than error object

* Update getMapColors to filter out transient layer

* Wrap Feature as FeatureCollection in Maps

* Add jest tests for geo processing functions. Add fixes for single feature handling

* i18n

* Review feedback. Test cleanup/fixes

* Update layer add panel footer logic to still show when source not selected

* Fix issue of not recognizing MultiPoint type. Remove throw logic for now

* Update telemetry with newly required placeholder function

* Prevent external modification of nested geojson objects

* i18n translation updates

* Revert "Fix issue of not recognizing MultiPoint type. Remove throw logic for now"

This reverts commit d692f91.

* Revert "Prevent external modification of nested geojson objects"

This reverts commit 0ea9fd3.

* yarn.lock update

* [File upload] Remove dynamic fields from mappings, code and telemetry test (#38902)

* Remove dynamic fields from mappings, code and telemetry test

* Add file-upload-telemetry to spaces and es archiver test mappings

* Don't create telemetry saved object if none exists, create on first update instead

* Back out es archiver mappings update

* Update zh-CN translations
@kindsun kindsun added the release_note:skip Skip the PR/issue when compiling release notes label Jul 8, 2019
@thomasneirynck thomasneirynck added release_note:enhancement and removed enhancement New value added to drive a business result release_note:skip Skip the PR/issue when compiling release notes labels Jul 29, 2019
@spalger spalger deleted the feature-maps-file-upload-integration branch November 8, 2019 15:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

7 participants