Skip to content

ubleipzig/solrbulk

master
Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?
Code
This branch is 4 commits behind miku:master.

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

solrbulk

Motivation:

Sometimes you need to index a bunch of documents really, really fast. Even with Solr 4.0 and soft commits, if you send one document at a time you will be limited by the network. The solution is two-fold: batching and multi-threading. http://lucidworks.com/blog/high-throughput-indexing-in-solr/

solrbulk expects as input a file with line-delimited JSON. Each line represents a single document. solrbulk takes care of reformatting the documents into the bulk JSON format, that SOLR understands.

solrbulk will send documents in batches and in parallel. The number of documents per batch can be set via -size, the number of workers with -w.

Project Status: Active – The project has reached a stable, usable state and is being actively developed. GitHub All Releases

This tool has been developed for project finc at Leipzig University Library.

Installation

Installation via Go tools.

$ go get github.com/miku/solrbulk/cmd/...

There are also DEB, RPM and arch packages available at https://github.com/miku/solrbulk/releases/.

Usage

Flags.

$ solrbulk
Usage of solrbulk:
  -commit int
        commit after this many docs (default 1000000)
  -cpuprofile string
        write cpu profile to file
  -memprofile string
        write heap profile to file
  -no-final-commit
        omit final commit
  -optimize
        optimize index
  -purge
        remove documents from index before indexing (use purge-query to selectively clean)
  -purge-pause duration
        insert a short pause after purge (default 2s)
  -purge-query string
        query to use, when purging (default "*:*")
  -server string
        url to SOLR server, including host, port and path to collection,
        e.g. http://localhost:8983/solr/biblio
  -size int
        bulk batch size (default 1000)
  -update-request-handler-name string
        where solr.UpdateRequestHandler is mounted on the server,
        https://is.gd/s0eirv (default "/update")
  -v    prints current program version
  -verbose
        output basic progress
  -w int
        number of workers to use (default 4)
  -z    unzip gz'd file on the fly

Example

Given a newline delimited JSON file:

$ cat file.ldj
{"id": "1", "state": "Alaska"}
{"id": "2", "state": "California"}
{"id": "3", "state": "Oregon"}
...

$ solrbulk -verbose -server https://192.168.1.222:8085/collection1 file.ldj

The server parameter contains host, port and path up to, but excluding the default update route for search (since 0.3.4, this can be adjusted via -update-request-handler-name flag).

For example, if you usually update via https://192.168.1.222:8085/solr/biblio/update the server parameter would be:

$ solrbulk -server https://192.168.1.222:8085/solr/biblio file.ldj

Some performance observations

  • Having as many workers as core is generally a good idea. However the returns seem to diminish fast with more cores.
  • Disable autoCommit, autoSoftCommit and the transaction log in solrconfig.xml.
  • Use some high number for -commit. solrbulk will issue a final commit request at the end of the processing anyway.
  • For some use cases, the bulk indexing approach is about twice as fast as a standard request to /solr/update.
  • On machines with more cores, try to increase maxIndexingThreads.

Elasticsearch?

Try esbulk.

About

SOLR bulk indexing utility for the command line.

Topics

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages

  • Go 74.7%
  • Shell 13.7%
  • Makefile 11.6%