[Ingest Node Pipelines] Adding create from csv (ecs mapper) functionality #101216
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@elasticmachine merge upstream |
|
@elasticmachine merge upstream |
|
@elasticmachine merge upstream |
gchaps
reviewed
Nov 15, 2021
|
Friendly reminder: Looks like this PR hasn’t been backported yet. |
kibanamachine
added
backport missing
fkanout
pushed a commit
to fkanout/kibana
that referenced
this pull request
Nov 17, 2021
mbondyra
pushed a commit
to mbondyra/kibana
that referenced
this pull request
Nov 19, 2021
dmlemeshko
pushed a commit
that referenced
this pull request
Nov 29, 2021
roeehub
pushed a commit
to build-security/kibana
that referenced
this pull request
Dec 16, 2021
This was referenced Feb 10, 2022
jrodewig
added a commit
to elastic/ecs
that referenced
this pull request
Feb 22, 2022
elastic/kibana#101216 adds a new ECS mapper feature to Kibana's Ingest Pipelines UI. This updates the ECS docs to cover the new feature.
jrodewig
added a commit
to elastic/ecs
that referenced
this pull request
Feb 22, 2022
elastic/kibana#101216 adds a new ECS mapper feature to Kibana's Ingest Pipelines UI. This updates the ECS docs to cover the new feature. (cherry picked from commit c3734cc)
ebeahan
pushed a commit
to ebeahan/ecs
that referenced
this pull request
Feb 22, 2022
elastic/kibana#101216 adds a new ECS mapper feature to Kibana's Ingest Pipelines UI. This updates the ECS docs to cover the new feature. (cherry picked from commit c3734cc)
ebeahan
added a commit
to elastic/ecs
that referenced
this pull request
Feb 22, 2022
elastic/kibana#101216 adds a new ECS mapper feature to Kibana's Ingest Pipelines UI. This updates the ECS docs to cover the new feature. (cherry picked from commit c3734cc) Co-authored-by: James Rodewig <james.rodewig@elastic.co>
jrodewig
added a commit
to elastic/elasticsearch
that referenced
this pull request
Feb 23, 2022
elastic/kibana#101216 adds a new ECS mapper feature to the Ingest Pipelines UI. This updates the ES docs to cover the new feature.
elasticsearchmachine
pushed a commit
to elastic/elasticsearch
that referenced
this pull request
Feb 23, 2022
elastic/kibana#101216 adds a new ECS mapper feature to the Ingest Pipelines UI. This updates the ES docs to cover the new feature. (cherry picked from commit d3d468e)
probakowski
pushed a commit
to probakowski/elasticsearch
that referenced
this pull request
Feb 23, 2022
elastic/kibana#101216 adds a new ECS mapper feature to the Ingest Pipelines UI. This updates the ES docs to cover the new feature.
jrodewig
added a commit
that referenced
this pull request
Feb 24, 2022
#101216 adds a new ECS mapper feature to the Ingest Pipelines UI. This makes some related changes: * Updates the doc link anchor text for the **Create pipeline from CSV** page to "CSV to pipeline docs". * Updates the doc link for the **Create pipeline from CSV** page to link to a related tutorial. * Updates the doc link for the **Create pipeline** page to link to the ingest pipeline docs. Previously, this linked to the create pipeline API docs, which aren't particularly relevant.
jrodewig
added a commit
that referenced
this pull request
Feb 24, 2022
#101216 adds a new ECS mapper feature to the Ingest Pipelines UI. This makes some related changes: * Updates the doc link anchor text for the **Create pipeline from CSV** page to "CSV to pipeline docs". * Updates the doc link for the **Create pipeline from CSV** page to link to a related tutorial. * Updates the doc link for the **Create pipeline** page to link to the ingest pipeline docs. Previously, this linked to the create pipeline API docs, which aren't particularly relevant. (cherry picked from commit b2475b9)
lucasfcosta
pushed a commit
to lucasfcosta/kibana
that referenced
this pull request
Mar 2, 2022
elastic#101216 adds a new ECS mapper feature to the Ingest Pipelines UI. This makes some related changes: * Updates the doc link anchor text for the **Create pipeline from CSV** page to "CSV to pipeline docs". * Updates the doc link for the **Create pipeline from CSV** page to link to a related tutorial. * Updates the doc link for the **Create pipeline** page to link to the ingest pipeline docs. Previously, this linked to the create pipeline API docs, which aren't particularly relevant.
This pull request was closed.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Resolves #99823
Fixes #113439
Summary
ECS team wants to add the functionality of the current standalone tool ECS Mapper into the Ingest Node Pipelines UI.
Ecs-mapper is an experimental CLI tool that turns CSV field mappings into data pipeline configurations. Users create a CSV, and ecs-mapper generates pipelines to map the original fields onto ECS fields. The tool is frequently recommended to users who want to map their custom data sources to ECS. See more here from the current ecs-mapper repo.
What's left
Note to reviewers
This is my first Kibana contribution, and I am not a front-end engineer, so please don't hesitate to call attention to anything.
Release note
Users can now create ingest pipelines from a CSV upload that enables mapping custom data source into ECS.
Demo
Here you will find a cloud instance of the PR for test. Sample input files can be found here.
mapping.csvis the success example,mapping_noheadersis an error example.Wireframes
https://www.figma.com/file/lgoyCDNZjVjd15qigMZQZO/ECS-Mapper?node-id=0%3A1
Screenshots
Navigating to Ingest Pipeline UI (with existing pipelines)
Navigating to Ingest Pipeline UI (without existing pipelines)
When the user arrives:
After uploading a good file:
Continue to create:
After uploading a bad file: