Removes an invisible control character from our log output #104
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Oh goodness, that is wild. Awesome job! 🎉 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Ho ho! What is this? It looks as though nothing has changed!
Turns out, there was an invisible control character in our logging output:
@EliAndrewC this is definitely a change you'll want to pull upstream.
Not sure why the logs were coming out clean in Python 3.4 but were being garbled in Python 3.6. This was breaking our elastalert/logstash/kibana slack notifications. Tracking down this invisible control character comprised about 4 hours of work. I suppose this is why it's so hard for software engineers to accurately estimate the amount of time a task will take. Who could possibly foresee that an invisible control character inserted into the source code over two years ago would suddenly break our log notifications?
The logging for the current deployment follows this chain:
stdout->syslog->/var/log/*.log->filebeat->tcp->elk stack->slack. It was... tedious trying to track down which part of that chain was failing. Especially when I could watch the logs being properly shipped from one place to the next.The logs made it all the way to
elastalertand it would just say, "no matching rules". I compared the logs being shipped from our old servers to the logs being shipped from our new servers. They were visually identical. Well, almost identical.I happened to notice that all of the tracebacks had "#36" at the end of every line. Weird... I also happened to remember that we wrote our own log handler to indent multiline messages. Okay, I figured maybe we weren't handling unicode properly, and something changed between Python 3.4 and 3.6.
I opened up
sideboard/internal/logging.py. Everything looked fine. Except... wait... that's when I noticed the control character in the screenshot above!I wanna give Sublime Text a big shoutout for doing the right thing by displaying invisible characters in the text editor! Best text editor I've ever used. Worth every penny. I don't think I would have ever figured this out if I hadn't been using Sublime Text.