Join GitHub today
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.
Sign upGitHub is where the world builds software
Millions of developers and companies build, ship, and maintain their software on GitHub — the largest and most advanced development platform in the world.
LB-872 Improve Node with retry logic and queue #5
Conversation
-make buffer size configurable -handle 403 error(bad token) -retries to send buffered logs to loggly in every 5 sec -queue support for bulk and input mode
Shwetajain148
force-pushed the
Shwetajain148:buffer-mode-support
branch
from
2b37753
to
fdba23a
| // function to truncate message over 1 MB | ||
| // | ||
| function truncateLargeMessage(message) { | ||
| var MaximumBytesAllowedToLoggly = 1000 * 1000; |
mchaudhary
Jan 10, 2017
Ideally good not to hard code the MaximumBytesAllowedToLoggly. It will be good to have the size variable and can be passed as config. For now I am approving this. @Shwetajain148 please log JIRA to make the value define in config.
Ideally good not to hard code the MaximumBytesAllowedToLoggly. It will be good to have the size variable and can be passed as config. For now I am approving this. @Shwetajain148 please log JIRA to make the value define in config.
Shwetajain148
Jan 11, 2017
Author
@mchaudhary, Actually I hard coded value of MaximumBytesAllowedToLoggly because we can send 1 MB per event i.e. maximum 1048576 bytes and in different encoding techniques like utf-8, utf-16 etc, individual character can take 1 to 6 bytes of memory space that increases the log message size each time. Due to this reason I took a safe side value less than actually accepted value so that if log message size increases due to different encoding techniques our final log message size would be less than 1 MB after truncation and will also reach to loggly successfully. I have also tested with different payload size and this logic was successfully truncating log message greater than 1 MB and logs were also reaching to loggly of less than 1 MB event size.
In that case we can not make MaximumBytesAllowedToLoggly configurable/dynamic. Anyways this logic is only applicable for truncating log message greater than 1 MB and not affecting any other code in library.
I have updated code to set value of MaximumBytesAllowedToLoggly using a constant variable.
@mchaudhary, Actually I hard coded value of MaximumBytesAllowedToLoggly because we can send 1 MB per event i.e. maximum 1048576 bytes and in different encoding techniques like utf-8, utf-16 etc, individual character can take 1 to 6 bytes of memory space that increases the log message size each time. Due to this reason I took a safe side value less than actually accepted value so that if log message size increases due to different encoding techniques our final log message size would be less than 1 MB after truncation and will also reach to loggly successfully. I have also tested with different payload size and this logic was successfully truncating log message greater than 1 MB and logs were also reaching to loggly of less than 1 MB event size.
In that case we can not make MaximumBytesAllowedToLoggly configurable/dynamic. Anyways this logic is only applicable for truncating log message greater than 1 MB and not affecting any other code in library.
I have updated code to set value of MaximumBytesAllowedToLoggly using a constant variable.
|
@Shwetajain148 once you create JIRA for above I can merge the PR |
Shwetajain148
force-pushed the
Shwetajain148:buffer-mode-support
branch
from
0e8f4a3
to
da058a3
In this PR, I have covered below checked tasks