Skip to content
Ingest CSV processor parses CSV data and stores it as individual fields
Java
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.github/workflows Release automation on GitHub (#8) Dec 12, 2019
gradle/wrapper Upgrade to 7.4.0 Oct 2, 2019
licenses First commit Mar 9, 2017
src Add maxchars property (#9) Dec 12, 2019
.gitignore First commit Mar 9, 2017
LICENSE.txt
NOTICE.txt Update Elasticsearch version to 6.0.0-rc1 Oct 7, 2017
README.md Upgrade to 7.5.2 Jan 22, 2020
build.gradle
gradle.properties
gradlew
gradlew.bat Upgrade 6.6 and gradle 5 Jan 30, 2019
settings.gradle Upgrade Elasticsearch 7.3.1 and csv parser library 2.8.3 Aug 29, 2019
uploadArchives.gradle

README.md

Elasticsearch csv Ingest Processor

This processor can parse CSV data and stores it as individual fields. This filter can also parse data with any separator, not just commas.

Caution

Official CSV ingest processor will release on 7.6.0. This plugin will deprecate 7.6.0.

Installation

ES version Command
7.5.2 ./bin/elasticsearch-plugin install https://github.com/johtani/elasticsearch-ingest-csv/releases/download/7.5.2/ingest-csv-7.5.2.0.zip
7.5.1 ./bin/elasticsearch-plugin install https://github.com/johtani/elasticsearch-ingest-csv/releases/download/7.5.1/ingest-csv-7.5.1.0.zip
7.5.0 ./bin/elasticsearch-plugin install https://github.com/johtani/elasticsearch-ingest-csv/releases/download/7.5.0/ingest-csv-7.5.0.0.zip
7.4.2 ./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/7.4.2.0/ingest-csv-7.4.2.0.zip
7.4.1 ./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/7.4.1.0/ingest-csv-7.4.1.0.zip
7.4.0 ./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/7.4.0.0/ingest-csv-7.4.0.0.zip
7.3.2 ./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/7.3.2.0/ingest-csv-7.3.2.0.zip
7.3.1 ./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/7.3.1.0/ingest-csv-7.3.1.0.zip
7.3.0 ./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/7.3.0.0/ingest-csv-7.3.0.0.zip
7.2.0 ./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/7.2.0.0/ingest-csv-7.2.0.0.zip
7.0.0 ./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/7.0.0.0/ingest-csv-7.0.0.0.zip
Older versions (6.8.5 - 5.5.0)
ES version Command
6.8.5 ./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/6.8.5.0/ingest-csv-6.8.5.0.zip
6.8.4 ./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/6.8.4.0/ingest-csv-6.8.4.0.zip
6.7.1 ./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/6.7.1.0/ingest-csv-6.7.1.0.zip
6.7.0 ./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/6.7.0.0/ingest-csv-6.7.0.0.zip
6.6.2 ./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/6.6.2.0/ingest-csv-6.6.2.0.zip
6.6.1 ./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/6.6.1.0/ingest-csv-6.6.1.0.zip
6.6.0 ./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/6.6.0.0/ingest-csv-6.6.0.0.zip
6.5.4 ./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/6.5.4.0/ingest-csv-6.5.4.0.zip
6.5.3 ./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/6.5.3.0/ingest-csv-6.5.3.0.zip
6.5.2 ./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/6.5.2.0/ingest-csv-6.5.2.0.zip
6.5.1 ./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/6.5.1.0/ingest-csv-6.5.1.0.zip
6.5.0 ./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/6.5.0.0/ingest-csv-6.5.0.0.zip
6.4.3 ./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/6.4.3.0/ingest-csv-6.4.3.0.zip
6.4.2 ./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/6.4.2.0/ingest-csv-6.4.2.0.zip
6.4.1 ./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/6.4.1.0/ingest-csv-6.4.1.0.zip
6.4.0 ./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/6.4.0.0/ingest-csv-6.4.0.0.zip
6.3.2 ./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/6.3.2.0/ingest-csv-6.3.2.0.zip
6.3.1 ./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/6.3.1.0/ingest-csv-6.3.1.0.zip
6.3.0 ./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/6.3.0.0/ingest-csv-6.3.0.0.zip
6.2.4 ./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/6.2.4.0/ingest-csv-6.2.4.0.zip
6.2.4 ./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/6.2.4.0/ingest-csv-6.2.4.0.zip
6.2.3 ./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/6.2.3.0/ingest-csv-6.2.3.0.zip
6.2.2 ./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/6.2.2.0/ingest-csv-6.2.2.0.zip
6.2.1 ./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/6.2.1.0/ingest-csv-6.2.1.0.zip
6.2.0 ./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/6.2.0.0/ingest-csv-6.2.0.0.zip
6.1.3 ./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/6.1.3.0/ingest-csv-6.1.3.0.zip
6.1.2 ./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/6.1.2.0/ingest-csv-6.1.2.0.zip
6.1.1 ./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/6.1.1.0/ingest-csv-6.1.1.0.zip
6.0.0 ./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/6.0.0.0/ingest-csv-6.0.0.0.zip
5.6.3 ./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/5.6.3.0/ingest-csv-5.6.3.0.zip
5.5.0 ./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/5.5.0.0/ingest-csv-5.5.0.0.zip

Usage

PUT _ingest/pipeline/csv-pipeline
{
  "description": "A pipeline to do whatever",
  "processors": [
    {
      "csv" : {
        "field" : "my_field",
        "columns" : ["a", "b"]
      }
    }
  ]
}

PUT /my-index/_doc/1?pipeline=csv-pipeline
{
  "my_field" : "a_value,b_value"
}

GET /my-index/_doc/1

then, the doc has 3 fields like this.
{
  "my_field" : "a_value,b_value",
  "a" : "a_value",
  "b" : "b_value"
}

Configuration

Parameter Use Required
field Field name of where to read the content from Yes
columns Define a list of column names. Yes
quote_char Define the character used to quote CSV fields. If this is not specified the default is a double quote ". No
separator Define the column separator value. If this is not specified, the default is a comma ,. No
max_chars_per_column Define the maximum amount of chars a column can have in it, before the plugin gives up. This is set to limit the amount of memory consumed. Valid values are 2 - 64000. Default is 4096 No

Setup

Install Java 12 + compiler

wget https://download.java.net/java/GA/jdk12/GPL/openjdk-12_linux-x64_bin.tar.gz tar -xzf openjdk-12_linux-x64_bin.tar.gz export JAVA_HOME=$(pwd)/jdk-12 export PATH=$(pwd)/jdk-12/bin:$PATH

In order to install this plugin, you need to create a zip distribution first by running

./gradlew clean check

This will produce a zip file in build/distributions.

After building the zip file, you can install it like this

bin/plugin install file:///path/to/ingest-csv/build/distribution/ingest-csv-0.0.1-SNAPSHOT.zip

Bugs & TODO

  • Need more test, like using separator and quote_char
  • and todos...

Thanks & More information about making Ingest Processor Plugin

Thanks for making great cookiecutter template @spinscale ! If you are interested in making Ingest Processor Plugin, you should use https://github.com/spinscale/cookiecutter-elasticsearch-ingest-processor

You can’t perform that action at this time.