scrapy-elasticsearch-bulk-item-exporter provides an exporter for Scrapy items that writes Elasticsearch Bulk format for easy further use with elasticsearch.
pip install scrapy-elasticsearch-bulk-item-exporter
scrapy crawl -o my.bulk -t elasticsearchbulk
Elasticsearch has an upper limit of bulk document size. 100mb is standard, it can be pushed up to 2GB (not advisable). This splitting can be done using split(1):
scrapy crawl -o - -t elasticsearchbulk
FEED_EXPORTERS = { 'elasticsearchbulk': 'scrapyelasticsearch.ElasticSearchBulkItemExporter' }
0.1: Initial release
Thanks to Julien Duponchelle, I used his scrapy-elasticsearch for inspriration.
Scrapys License: BSD. See LICENSE for details.