Support for compression (GZIP?) on PUT/POST

We are indexing millions of documents per day for which we are using the bulk API.
Unfortunately the requests we are generating result in JSON data of more than 100MB per request and a total daily data transfer of multiple GB's.

Is there any way to PUT/POST compressed JSON data in order to reduce the total transferred data?

[gmarz]

Hi @jplusje

Try enabling HTTP compression in your ES settings http.compression: true: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-http.html

and enabling compressed responses on the client:

new ConnectionSettings(...).EnableCompressedResponses();

which will add Accept-Encoding: gzip,deflated to the request.

We are using HTTP compression on the ES responses (server to client) exactly as you described but we would like to also use compression with requests from client to server.

[gmarz]

Gotcha. At the moment, the client doesn't support this, but it should be possible by adding Content-Encoding: gzip to the request headers and writing the request stream using GZipStream.

We'll look into adding this. In the meantime, you can test this out by subclassing HttpConnection and overriding the CreateHttpWebRequest and DoSynchronousRequestmethods.

[Mpdreamz]

Hi @jplusje

When I first implemented this feature I found some contradicting information online whether elasticsearch supported it or not and also while testing found that it didn't.

Rather embarrassing I have to admit I think simply forgot to send the Http Header Content-Encoding: gzip back then. You can't imagine how hard I'm face-palming right now.

I just submitted a PR that will fix this in the upcoming release albeit with a new toggle .EnableHttpCompression() for backward compatible reasons.

Thank you for reporting and forcing us to reevaluate our current implementation

😫 🌴

[typedefstructer]

@Mpdreamz is this support only done in the server, I'm using java rest high level client, can I just enable some setting to achieve compression? or should I implement it for myself? also the server and client I'm using is 6.8.12