light log stash for Elasticsearch
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
configuration
dispatcher
document
elastic
errors
indexer
log
server
template
transport
uuid
vendor/github.com
.gitignore
Dockerfile
Gopkg.lock
Gopkg.toml
LICENSE
README.md
config.json
entrypoint.sh
espipe.png
main.go

README.md

What is espipe?

espipe is a service indexing documents in bulk requests to Elasticsearch.

Run

Docker

docker run -p 5000:5000 -v /etc/espipe:/etc/espipe khezen/espipe:6

Supported tags and respective Dockerfile links

API

Request Response Description
POST /espipe/v1/{templateName}/{documentType} JSON body 200 OK indexes JSON body as {documentType} in Elasticsearch {template.name}-yyyy.MM.dd
GET /espipe/v1/health 200 OK healthcheck

Config

{
  "redis": {},
  "elasticsearch": {}
}

Redis

Redis is disabled by default in which case buffer is stored in memory.

  "redis": {
    "enabled": true,
    "address":"http://localhost:6379",
    "password": "changeme",
    "parition":0
  }

Elasticsearch

  "elasticsearch": {
    "address":"http://localhost:9200",
    "templates": [{
      "name": "logs",
      "flushPeriodMS": 5000,
      "body": {...}
    }]
  }

templates

  • espipe creates index templates in elasticsearch if they do not exist yet
  • for each template, espipe creates indices on daily basis
    • template=logs-*,
    • indices=logs-2017.01.05, logs-2017.01.06, etc..
  • For each index, espipe triggers bulk requests every flushPeriodMS

template.body

template.body takes the template settings and mappings with types definition. See the Create Template API documentation.

The mapping itself is flexible and is schema-free. New fields are automatically added to the type mapping definition when espipe indexes a new document. Check out the mapping section for more information on mapping definitions.

AWS Sign

espipe supports AWS authentication for Amazon Elasticsearch Service

"elasticsearch": {
  "address":"https://host.eu-west-1.es.amazonaws.com",
  "AWSAuth": {
    "accessKeyId": "changeme",
    "secretAccessKey": "changeme",
    "region": "eu-west-1"
  },
  "templates": [...]
}

Basic Auth

espipe supports basic authentication for Elasticsearch (shield, search-guard)

"elasticsearch": {
  "address": "http://localhost:9200",
  "basicAuth":{ 
    "username": "elastic",
    "password": "changeme"
  },
  "templates": [...]
}

Default config.json

Request Response Description
POST /espipe/v1/logs/log JSON body 200 OK indexes JSON body as log in Elasticsearch logs-yyyy.MM.dd
POST /espipe/v1/web/trace JSON body 200 OK indexes JSON body as trace in Elasticsearch web-yyyy.MM.dd
GET /espipe/v1/health 200 OK healthcheck

User Feedback

Issues

If you have any problems or questions, please ask for help through a GitHub issue.