-
Notifications
You must be signed in to change notification settings - Fork 173
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Beaver transport truncate messages to 32KB #415
Comments
I'll submit a PR for this soon... unless there was a reason to truncate? |
You really shouldn't be writing out request logs that are that large. |
And why is that? We log a huge amount of data for some requests because they are afterwards directly used for generating statistics data. All other parts in the pipeline (Redis, Logstash, Elasticsearch) have no issue handling them. I see the commit you linked, but it doesn't explain why it is like that... EDIT: In case this was the reasoning: It's true that Lucene has a limit of 32766 characters per term for indexing, but this shouldn't silently truncate the whole message in Beaver which is several layers before and hence make it invalid JSON (causing Logstash to drop the whole request later on), especially because it may not even be intended as single indexed field later. |
I too see no reason why the truncate here should be needed and took the liberty to create this PR: https://github.com/python-beaver/python-beaver/pull/422 |
@rwunderer Your link's URL is broken. For simplicity, here again as working link: PR |
This code appears to be truncating the event to 32K. Not sure if it is intended or why it's done.
https://github.com/python-beaver/python-beaver/blob/master/beaver/transports/base_transport.py#L119
The affect is that lines larger than 32KB aren't handled.
While this may be an unusual case, I do have messages that are larger than this sometimes (raw request logs), this should at least be configurable.
The text was updated successfully, but these errors were encountered: