Bulk load performance and timeout, HTTPoison options, HTTP request timeout option

[brettrp]

When bulk loading a large number of documents, in can be tricky to get the bulk_page_size, bulk_wait_interval correct, without significantly slowing down an index build.

Also, in some cases the HTTP request takes longer than the default of 8000ms resulting in a timeout.

Digging into the code, there is a default_opts: configuration option to pass options to HTTPoison.request, but this doesn't seem to be documented. Setting this to [recv_timeout: 20000] (or something large) and setting bulk_wait_interval to zero gives much better performance, since it's waiting at most 20 seconds but only as long as the request needs. For me at least this seemed to eliminate the need to wait a fixed time between requests.

It would be good to at least document the default_opts: option, or maybe even make configuration of the HTTP request timeout a first-class configuration option.

[ghost]

There is also one more option, the switching to something better than httpoison, e.g. Tesla.