Bulk indexing instead of single item indexing

[jenkin]

Working in a pipeline, every item is indexed separately with many requests to ES (one per item). In addition, in some cases you want to break the pipeline concept, applying a global transformation to items before indexing (ie by overloading open_spider and close_spider methods in a pipeline class).

Using ES bulk api you can temporarily add items to an item buffer (with a length controlled by a setting) and then index them sometimes and not for every single item.

[jayzeng]

Here you go:
https://github.com/knockrentals/scrapy-elasticsearch/pull/22/files

[jayzeng]

@jenkin will be great if you can code review or do some testings

[jenkin]

Yep, thanks! Maybe it can be useful to add a setting for buffer length, so every some items it indexes them to ES. In my application now I have dozens of items every run, but for thousands of them a performance problem can raise. I added two line notes to code... :)

[jayzeng]

Now available at https://pypi.python.org/pypi/ScrapyElasticSearch/0.7