chip is a local and remote log parsing and monitoring tool for system admins and developers.
It wraps the features of
One line install
Want to jump right in? This will work on most *nix systems. You may have to execute it with
perl -e "$(curl -fsSL https://gist.githubusercontent.com/katzgrau/940608/raw/4fe8b9ae09ea12e31979406f28b8a6e3f06e86df/install-chip)"
Now read all about what you've just installed :)
What chip can do for you, really quickly
- Tail multiple logs at once, even remote logs
- Search multiple logs (even remote ones) for patterns. Like grep across n hosts/files
- Monitor multiple logs for occurrences of some match. Ie, log a certain line, send an email, start a process, more.
chip starts up and tries to open log file(s), either local or remote. If one isn't specified, it reads from standard input.
It goes through each line looking for patterns that you have specified as arguments. When it finds a match, chip does
whatever you told it to do with that match.
Maybe you'd want to:
- Show the match(es) on the screen (like grep)
- Colorize matches (like a colorizer)
- Follow the file (like tail)
- Log it to a file (like output redirection)
- Have the line emailed to you (maybe for really important stuff)
- Kick off another program
- Any combination of the above, with any number of matches, all at once
Split matches into different files
You have a long-running process that prints out its progress. Some items might be errors (maybe prefixed with 'ERROR'), others are warnings.
$ ./lame-process | chip --match0=ERROR --log0=errors.txt --match1=WARN --log1=warnings.txt
View some matches in your terminal, stick others in a file
You're about to run a script that does something nasty, like importing 1,000s of email addresses into an email service that rhymes with 'mailpimp'. A whole bunch of status lines get output, along with occasional errors.
You want to run the script, see only the errors on the screen, and at the same time, send all output to one file and all errors to another.
$ ./email-import | chip --match0='.*' --log0=output.txt --match1='ERROR' --log1=errors.txt --screen1
Highlight matches in a terminal, and have errors emailed to you, and logged
Want to get notified whenever a fatal (like an HTTP 500) error occurs and is logged to some file?
Want to log them separately too? Want to view all output as it goes by (like
tail), but highlight
fatal errors in red? No prob. This time, in shorthand:
$ chip -f -m0='HTTP 500' -s0='red' -firstname.lastname@example.org -L0=fatal.txt -m1='.*' -s1 error_log.txt
The kicker: Do all of the above on multiple, remote files
chip can tail multiple files too. It will also tail files over ssh! Ever want to monitor multiple logs and separate boxes behind a load balancer? Let's do the last example again but with two remote files:
$ chip -f -m0='HTTP 500' -s0='red' -email@example.com -L0=fatal.txt -m1='.*' -s1 firstname.lastname@example.org:/var/log/errors.log email@example.com:/var/log/errors.log
Here are some tips for making chip super-useful:
When dealing with remote servers, set up ssh keys (authorized_keys)
Your chip commands will run quickly without prompting you for your password every time you run a chip command
alias chip for an easy remote grep
In .bashrc, put something like:
alias grepprod='chip -s0 firstname.lastname@example.org:/var/log/errors.log email@example.com:/var/log/errors.log'
Then from the prompt, all you need to grep through all of your prod logs is:
Command not working? Try -d
Type out an entire
chip command, and tack on -d. Chip will tell you exactly what it was about to do.
./chip -f -i0='(error|fatal)' -s0=red -firstname.lastname@example.org,email@example.com -firstname.lastname@example.org log.txt example.com:/tmp/log.txt -d
chip is planning to open ... ------------------------------------------------- File: log.txt SSH : [none] @ example.com => /tmp/log.txt with these patterns and associated handlers ------------------------------------------------- Pattern #0 => /(error|fatal)/i - screen => red - email => email@example.com,firstname.lastname@example.org with these options ------------------------------------------------- follow => 1 debug => 1 complete => email@example.com
More Fun Stuff
How about kicking off some process every time something specific shows up in the log?
$ chip --follow --match0='out of space' --exec0='rm -rf tmp/*' maillog
How about being notified of when a script ends too?
$ ./alter-table | chip --firstname.lastname@example.org --match0=ERROR --log0=errors.txt
How about color-coding your output?
$ chip --follow --match0=ERROR --screen0='bold red on_black' --match1=WARN --screen1=yellow --match2=INFO --screen2=blue ap_log.txt
- Input by STDIN or file
- Open local or remote files (piggybacks on
tail-ing of input files
- Pattern-based matching of lines
- Email notification on matches all-at-once or per-line
- Color-coded screen output for matches
- Logging of matches in separate files
- Arbitrary command execution for matches
- Adding as many command per match as needed
- Email notification on script completion
- Very portable, fast (perl)
To view the usage:
$ chip --help
chip used to be named
stalk. Before that, it was named
divvy. I found out
there was a window manager named 'Divvy'. This fact ate at my sanity for weeks, and I finally gave in.
stalk was harder to pronounce and much creepier, but I appreciated the double-meaning.
stalk monitored multiple remote files, the new name reflected the expanded
scope of the project.
... One hour later, I realized I would now be receiving emails from
stalk. This bothered
me to the point that I knew another name change was iminent.
stalk was just too creepy.
The utility is now named
chip. Chip is everybody's innocent pal, right? He's no
stalker, that's for sure.
chip has a similar play on words (log chipper).
Couldn't you trust
chip to keep an eye on things while you're away?
Written in Perl. Inspired by my work at Yahoo!. First presented at Hack and Tell in NYC.
Special thanks to Andrew Gwozdziewycz (co-organizer of H & T) for the contribution of cowsay.
Copyright (c) 2011 Kenny Katzgrau
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.