Beaver transport truncate messages to 32KB

[gregsterin]

This code appears to be truncating the event to 32K. Not sure if it is intended or why it's done.
https://github.com/python-beaver/python-beaver/blob/master/beaver/transports/base_transport.py#L119

The affect is that lines larger than 32KB aren't handled.
While this may be an unusual case, I do have messages that are larger than this sometimes (raw request logs), this should at least be configurable.

[gregsterin]

I'll submit a PR for this soon... unless there was a reason to truncate?

[josegonzalez]

You really shouldn't be writing out request logs that are that large.

05eb40c

[CherryDT]

And why is that? We log a huge amount of data for some requests because they are afterwards directly used for generating statistics data. All other parts in the pipeline (Redis, Logstash, Elasticsearch) have no issue handling them.

I see the commit you linked, but it doesn't explain why it is like that...

EDIT: In case this was the reasoning: It's true that Lucene has a limit of 32766 characters per term for indexing, but this shouldn't silently truncate the whole message in Beaver which is several layers before and hence make it invalid JSON (causing Logstash to drop the whole request later on), especially because it may not even be intended as single indexed field later.

[rwunderer]

I too see no reason why the truncate here should be needed and took the liberty to create this PR: https://github.com/python-beaver/python-beaver/pull/422

[CherryDT]

@rwunderer Your link's URL is broken. For simplicity, here again as working link: PR