Provided dedicated support for Spark Streaming

[costin]

In Spark 1.6 streaming support is implemented as micro-batching meaning each data window triggers a Spark job. While this works, for very small time windows (under 1 minute) this can lead to resource exhaustion (such as consuming all the HTTP connections).
The Spark documentation mentions the usage of a connection pool to get around this.
Spark 2.0 might revise this pattern and provide a richer/better scenario.
Either way, ES-Hadoop should provide a work-around for both Spark 1.x and 2.x.

[jbaiera]

Spark 2.0 has provided improvements to the processing capabilities for streams, but not much in the ways of how the streams are executed. The microbatching still exists in Spark 2.0 Structured Streaming, and as such, we'll definitely need this for both Streaming and Structure Streaming.