Skip to content
master
Switch branches/tags
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 
 
 
 
 
 
 

npm npm npm GitHub license

duckimport

Elasticsearch cli Data Importer

see faq

Features

  • Can Index(import) very large files.
  • Runs on any platform (Windows, Mac, Linux)
  • Easy to use
  • Ability to provide a command-line encoded config. No need to create a local config file

Install

Install Nodejs if you haven't already.
Node.js version 11.10 or higher is required.

Then install the package globally:
npm i -g duckimport
or
yarn global add duckimport

Demo

Gif

Usage

duckimport <command>
You can see available options with duckimport --help

Usage: duckimport [options]

Options:
  -c, --config <path>          config file path
  -i, --inline <configString>  base64 encoded config object
  -h, --help                   output usage information

Examples:
  $ duckimport -c ./config.json
  $ duckimport -i NDJjNGVx........GZzZGY=

Examples

  • duckimport -c ./config.json
  • duckimport -i ewogICAgIm.....KfQ==

You will need a proper json config in order to run duckimport

An example config file:

{
    "client": {
        "node": "http://localhost:9200"
    },
    "file": "bigFile.csv",
    "separator": ",",
    "columns": [
        "firstname",
        "lastname"
    ],
    "lines": 10000,
    "createNewIndex": true,
    "index": {
        "index": "peopleIndex",
        "body": {
            "settings": {
                "number_of_replicas": 0,
                "auto_expand_replicas": false
            },
            "mappings": {
                "properties": {
                    "firstname": {
                        "type": "keyword"
                    },
                    "lastname": {
                        "type": "keyword"
                    }
                }
            }
        }
    }
}

config

You can pass a config file using duckimport -c <config file path>
or duckimport -i <base64 encoded config object>

  • client

    • Type: Object
    • Elasticsearch client configuration. Reference
  • file

    • Type: String
    • The file you want to import(aka. indexing) into Elasticsearch. Can be .txt .csv .tsv etc.
  • separator

    • Type: String
    • The separator between your data's columns
    • Exp: ";", ":", "," etc.
  • columns

    • Type: Array
    • Array of column headers or field names. If there is a non-specified headers they won't be imported.
  • lines

    • Type: Number
    • Number of lines included in every chunk sent to the Elasticsearch
  • createNewIndex

    • Type: Boolean
  • index

    • Type: Object
    • Index configuration. Reference
    • index field is represents your Index name and it's mandatory. You don't need to include a body field or so if you set createNewIndex: false

F.A.Q

How does it work?

duckimport process your files in any size line by line and send them into Elasticsearch as chunks. Thanks to nexline

What is lps mean?

Lines Per Seconds. Represents how many lines of your file is processing in a second,

How can I use without a config file?

You can use inline base64 encoded config string using -i flag. All you need is prepare your config object(json or js object and encode it using base64. duckimport will decode the encoded string and process it.

Milestones

  • duckimport GUI - 100+ Github stars

Duck icon made by Freepik from http://www.flaticon.com/

About

Elasticsearch Data Import Tool

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published