Make librdkafka message buffer size configurable to handle large messages

[thatsdone]

DESCRIPTION

ngx_kafka_module currently cannot handle large messages.

REPRODUCE PROCEDURE

Anyway, POST a large message using ngx_kafka_module.
In case of me, I used a 6MB bynary message that contains a movie file.

ANALYSYS and SUGGESTED RESOLUTION

At the moment, there is a consideration to handle a large message storing as files at:

https://github.com/brg-liuwei/ngx_kafka_module/blob/master/ngx_http_kafka_module.c#L413

But this is not implemented yet.

Here, it's also possible to workaround the large message handling
by configuring message buffer size by specifying 'client_body_buffer_size' like below:

        location = /api/test-topic {
             kafka_topic test-topic;
             client_max_body_size 1g;
             client_body_buffer_size 64m;
        }

Here, the problem is that we need to increase librdkafka layer buffer size
before initializing librdkafka by calling 'rd_kafka_new()' at:

https://github.com/brg-liuwei/ngx_kafka_module/blob/master/ngx_http_kafka_module.c#L500

Koutaro (@tsukada-ascade) has already written a patch for this enhancement.

https://github.com/tsukada-ascade/ngx_kafka_module/tree/feature/configurable-message-max-bytes

His patch introduces a new configuration parameter 'kafka_message_max_bytes' and
enables adjusting the desired buffer size so that we can reduce memory consumption.

I asked him to create a PR, and I hope this is useful for the community.

[tsukada-ascade]

Hi,
Thank you for make this issue.

https://github.com/tsukada-ascade/ngx_kafka_module/tree/feature/configurable-message-max-bytes

Now, I pushed two new branches.

• configurable-message-max-bytes-v2
• configurable-message-max-bytes-v2-test

In the v2 branch, that only has 1-commit, we have improved the parser part of nginx.conf to make specifications like 'k', 'm', 'g' are available(e.g. kafka_message_max_bytes 16m).
In the *-test branch, that has 2-commits, I've refreshed .travis.yml to allow it to pass the tests. And I also added testcase(t/005-huge_body.t) for the feature I newly added.

Since the all tests have passed, so I would like to make a PR later.