Elasticsearch Dumpdata for AWS Lambda

About

Runtime

Python 2.7

Lambda Hander

lambda_function.lambda_handler

Input event

Example: Input event:

{
  "source_host": "http://<your_elasticsearch_server:9200>/",
  "source_index": "blog",
  "scroll": "5m",
  "scan_options": {
    "size": 500
  },
  "s3_bucket": "myBucket",
  "s3_prefix": "dumps"
}

• source_host: index to read documents from.
• source_index: index to read documents from.
• scroll: (Optional) keep the scroll open for another minute. default to 5m
• scan_options.size: (Optional) you will get back a maximum of size * number_of_primary_shards documents in each batch. default to 500
• s3_bucket: s3 bucket name.
• s3_prefix: (optional) s3 object prefix.

Execution result

Execution result sample:

{
  "acknowledged": true
}

Setup on local machine

# 1. Clone this repository with lambda function name
git clone https://github.com/KunihikoKido/aws-lambda-es-dumpdata.git es-daumpdata

# 2. Create and Activate a virtualenv
cd es-daumpdata
virtualenv env
source env/bin/activate

# 3. Install Python modules for virtualenv
pip install -r requirements/local.txt

# 4. Install Python modules for lambda function
fab setup

Run lambda function on local machine

fab invoke

Run lambda function with custom event

fab invoke:custom-event.json

Make zip file

fab makezip

Update function code on AWS Lambda

fab aws-updatecode

Get function configuration on AWS Lambda

fab aws-getconfig

Invoke function on AWS Lambda

fab aws-invoke

Show fabric Available commands

fab -l

Amazon API Gateway

Example settings

Methods and Resources:

POST /{indexname}/_dumpdata

Request mapping template:

{
  "source_host": "http://<your_elasticsearch_server>/",
  "source_index": "$input.params('indexname')",
  "scroll": "5m",
  "scan_options": {
    "size": 500
  },
  "s3_bucket": "myBucket",
  "s3_prefix": "dumps"
}

Example Request:

POST /blog/_dumpdata