Compression support for bulk method?

[ashwinaggarwal]

Is there a way to send compressed gzipped data for the bulk API?

import zlib from 'zlib';
import elasticsearch from 'elasticsearch';
import { promisify } from 'util';

const { ELASTICSEARCH_URL, ELASTICSEARCH_INDEX, ELASTICSEARCH_TYPE } = process.env;

const es = new elasticsearch.Client({
  host: ELASTICSEARCH_URL,
  suggestCompression: true
});

const bulkP = promisify(es.bulk.bind(es));
const gzipP = promisify(zlib.gzip.bind(zlib));

const mapRecordsToElasticSearchBulk = records => records.reduce(
  (mappedRecords, record) => mappedRecords.concat([
    {
      index: {
        _index: ELASTICSEARCH_INDEX,
        _type: ELASTICSEARCH_TYPE
      }
    },
    record
  ]), []
);

export const bulk = async (records) => {
  const buffer = Buffer.from(mapRecordsToElasticSearchBulk(records));
  const zippedRecords = await gzipP(buffer);
  await bulkP({ body: zippedRecords });
};

Passing compressed data as above throws the following error.

Bulk body should either be an Array of commands/string, or a String.

There is nothing in the documentation related to compressed data.

[spalger]

Hmm, not right now I don't think. I think we'd need to support buffers, and include the appropriate content-type headers, but I'm not sure.

[ashwinaggarwal]

Ah okay, I have created this example by allowing gzipped Buffer to be sent to Elasticsearch cluster.

Made this change to the elasticsearch-js/src/lib/serializers/json.js to allow Buffers.

Doesn't work with my Amazon ES though. Works fine with local ES.

[stale]

We understand that this might be important for you, but this issue has been automatically marked as stale because it has not had recent activity either from our end or yours.
It will be closed if no further activity occurs, please write a comment if you would like to keep this going.

Note: in the past months we have built a new client, that has just landed in master. If you want to open an issue or a pr for the legacy client, you should do that in https://github.com/elastic/elasticsearch-js-legacy