[ML] Use CSV ingest processor in find_file_structure ingest pipeline #51492
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Changes the find_file_structure response to include a CSV ingest processor in the ingest pipeline it suggests. Previously the Kibana file upload functionality parsed CSV in the browser, but by parsing CSV in the ingest pipeline it makes the Kibana file upload functionality more easily interchangable with Filebeat such that the configurations it creates can more easily be used to import data with the same structure repeatedly in production.
|
Pinging @elastic/ml-core (:ml) |
2 tasks
droberts195
deleted the
use_csv_processor_in_file_structure_finder_ingest
branch
droberts195
added a commit
to elastic/kibana
that referenced
this pull request
Jan 28, 2020
Changes the file upload functionality to process delimited files by splitting them into to messages, then sending these to the ingest pipeline as a single field for further processing in Elasticsearch. The csv_importer has been removed and the old sst_importer replaced with a similar message_importer that has been enhanced to cover the edge cases required by delimited file processing. Previously the file upload functionality parsed CSV in the browser, but by parsing CSV in the ingest pipeline it makes the Kibana file upload functionality more easily interchangable with Filebeat such that the configurations it creates can more easily be used to import data with the same structure repeatedly in production. Companion to elastic/elasticsearch#51492
droberts195
added a commit
that referenced
this pull request
Jan 28, 2020
…51492) Changes the find_file_structure response to include a CSV ingest processor in the ingest pipeline it suggests. Previously the Kibana file upload functionality parsed CSV in the browser, but by parsing CSV in the ingest pipeline it makes the Kibana file upload functionality more easily interchangable with Filebeat such that the configurations it creates can more easily be used to import data with the same structure repeatedly in production.
albertzaharovits
pushed a commit
to albertzaharovits/elasticsearch
that referenced
this pull request
Jan 28, 2020
…lastic#51492) Changes the find_file_structure response to include a CSV ingest processor in the ingest pipeline it suggests. Previously the Kibana file upload functionality parsed CSV in the browser, but by parsing CSV in the ingest pipeline it makes the Kibana file upload functionality more easily interchangable with Filebeat such that the configurations it creates can more easily be used to import data with the same structure repeatedly in production.
droberts195
added a commit
to elastic/kibana
that referenced
this pull request
Jan 28, 2020
Changes the file upload functionality to process delimited files by splitting them into to messages, then sending these to the ingest pipeline as a single field for further processing in Elasticsearch. The csv_importer has been removed and the old sst_importer replaced with a similar message_importer that has been enhanced to cover the edge cases required by delimited file processing. Previously the file upload functionality parsed CSV in the browser, but by parsing CSV in the ingest pipeline it makes the Kibana file upload functionality more easily interchangable with Filebeat such that the configurations it creates can more easily be used to import data with the same structure repeatedly in production. Companion to elastic/elasticsearch#51492
albertzaharovits
added a commit
that referenced
this pull request
Jan 28, 2020
* Done * Update docs/reference/settings/security-settings.asciidoc Co-Authored-By: Ioannis Kakavas <ikakavas@protonmail.com> * Update docs/reference/settings/security-settings.asciidoc Co-Authored-By: Ioannis Kakavas <ikakavas@protonmail.com> * refactored ldap_search explanation * Tim's review! * [ML] Use CSV ingest processor in find_file_structure ingest pipeline (#51492) Changes the find_file_structure response to include a CSV ingest processor in the ingest pipeline it suggests. Previously the Kibana file upload functionality parsed CSV in the browser, but by parsing CSV in the ingest pipeline it makes the Kibana file upload functionality more easily interchangable with Filebeat such that the configurations it creates can more easily be used to import data with the same structure repeatedly in production. * Add test verify replica allocator with sync_id (#51512) We no longer issue new sync_ids in 8.0, but we still need to make sure that the replica allocator prefers copies with matching sync_id. This commit adds tests for that. Relates #50776 * Formatting: keep simple if / else on the same line (#51526) Previous the formatter was breaking simple if/else statements (i.e. without braces) onto separate lines, which could be fragile because the formatter cannot also introduce braces. Instead, keep such expressions on the same line. * Nits Co-authored-by: Ioannis Kakavas <ikakavas@protonmail.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: David Roberts <dave.roberts@elastic.co> Co-authored-by: Nhat Nguyen <nhat.nguyen@elastic.co> Co-authored-by: Rory Hunter <pugnascotia@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes the find_file_structure response to include a CSV
ingest processor in the ingest pipeline it suggests.
Previously the Kibana file upload functionality parsed CSV
in the browser, but by parsing CSV in the ingest pipeline
it makes the Kibana file upload functionality more easily
interchangable with Filebeat such that the configurations
it creates can more easily be used to import data with the
same structure repeatedly in production.
Companion to elastic/kibana#56038