Skip to content
Command-line processor for structured logs
Go JavaScript Other
Branch: master
Clone or download

Latest commit

Files

Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.github/workflows github action workflow for windows testing (#444) Mar 20, 2020
ast Fix logical not in expressions (#486) Mar 31, 2020
cmd make space deletion cancel and await other operations (#451) Apr 3, 2020
driver Cache min and max times (#388) Mar 7, 2020
emitter adjust mccanne/zq to brimsec/zq (#302) Feb 7, 2020
expr Add "in" operator (#495) Apr 2, 2020
filter Move zng.Unsafe* to pkg/byteconv (#450) Mar 23, 2020
pcap make space deletion cancel and await other operations (#451) Apr 3, 2020
performance Perf updates & current run (#352) Mar 4, 2020
pkg Graceful http shutdown (#500) Apr 3, 2020
proc Put warnings (#477) Mar 28, 2020
reducer Zx cleanup (#447) Mar 23, 2020
reglob adjust mccanne/zq to brimsec/zq (#302) Feb 7, 2020
scanner Improve scanner.Combiner error messages (#470) Mar 27, 2020
scripts Perf updates & current run (#352) Mar 4, 2020
streamfn adjust mccanne/zq to brimsec/zq (#302) Feb 7, 2020
tests Bzng write test (#507) Apr 3, 2020
zbuf normalize BZNG sets (#282) Feb 18, 2020
zcode normalize BZNG sets (#282) Feb 18, 2020
zeek types.json: change rules field name to rules (#476) Mar 27, 2020
zio Typed ndjson: accept Ts-less points (#479) Mar 30, 2020
zng Bzng write test (#507) Apr 3, 2020
zngnative Rename NativeValue -> Value (#456) Mar 24, 2020
zqd make space deletion cancel and await other operations (#451) Apr 3, 2020
zql Add "in" operator (#495) Apr 2, 2020
ztest Bzng write test (#507) Apr 3, 2020
.gitignore More testing for packet post endpoint (#399) Mar 9, 2020
CHANGELOG.md changelog for v0.9.0 (#512) Apr 3, 2020
CONTRIBUTING.md adjust mccanne/zq to brimsec/zq (#302) Feb 7, 2020
LICENSE update license to use Brim Security, Inc. (#413) Mar 13, 2020
Makefile Moving zeek process into own package (#432) Mar 20, 2020
README.md added binaries link (#457) Mar 24, 2020
code_of_conduct.md add code of conduct (#262) Jan 30, 2020
conn.log Improve README for new users (#169) Dec 25, 2019
go.mod Add waterfall logger (#492) Apr 3, 2020
go.sum go api client (#458) Mar 30, 2020
package.json import github.com/looky-cloud/lookytalk (#3) Nov 12, 2019

README.md

zq CI GoDoc

zq is a command-line tool for searching and analyzing logs, particularly Zeek logs. If you are familiar with zeek-cut, you can think of zq as zeek-cut on steroids.

zq is comprised of:

  • an execution engine for log pattern search and analytics,
  • a query language that compiles into a program that runs on the execution engine, and
  • an open specification for structured logs, called ZNG.
    (Note: The ZNG format is in Alpha and subject to change.)

zq takes Zeek/ZNG logs as input and filters, transforms, and performs analytics using the zq query language, producing a log stream as its output.

Install

To install zq, you can clone the repo and compile the source. For Windows, MacOS, and Linux there are pre-compiled binary releases.

If you don't have Go installed, download and install it from the Go downloads page.

If you're new to Go, remember to set GOPATH. A common convention is to create ~/go and point GOPATH at $HOME/go.

To install the binaries in $GOPATH/bin, clone this repo and execute make install:

git clone https://github.com/brimsec/zq
cd zq
make install

Usage

For zq command usage, see the built-in help by running:

zq help

zq program syntax and semantics are documented in the query language README.

Examples

Here are a few examples based on a very simple "conn" log from Zeek (conn.log), located in this directory. See the zq-sample-data repo for more test data, which is used in the examples in the query language documentation.

To cut the columns of a Zeek "conn" log like zeek-cut does, run:

zq "* | cut ts,id.orig_h,id.orig_p" conn.log

The "*" tells zq to match every line, which is sent to the cut processor using the UNIX-like pipe syntax.

When looking over everything like this, you can omit the search pattern as a shorthand and simply type:

zq "cut ts,id.orig_h,id.orig_p" conn.log

The default output is a ZNG file. If you want just the tab-separated lines like zeek-cut, you can specify text output:

zq -f text "cut ts,id.orig_h,id.orig_p" conn.log

If you want the old-style Zeek ASCII TSV log format, run the command with the -f flag specifying zeek for the output format:

zq -f zeek "cut ts,id.orig_h,id.orig_p" conn.log

You can use an aggregate function to summarize data over one or more fields, e.g., summing field values, counting, or computing an average.

zq "sum(orig_bytes)" conn.log
zq "orig_bytes > 10000 | count()" conn.log
zq "avg(orig_bytes)" conn.log

The ZNG specification describes the significance of the _path field. By leveraging this, diverse Zeek logs can be combined into a single file.

zq *.log > all.zng

Comparisons

Revisiting the cut example shown above:

zq -f text "cut ts,id.orig_h,id.orig_p" conn.log

This is functionally equivalent to the zeek-cut command-line:

zeek-cut ts id.orig_h id.orig_p < conn.log

If your Zeek events are stored as JSON and you are accustomed to querying with jq, the equivalent would be:

jq -c '. | { ts, "id.orig_h", "id.orig_p" }' conn.ndjson

Comparisons of other simple operations and their relative performance are described at the performance page.

Contributing

See the contributing guide on how you can help improve zq!

Join the Community

Join our Public Slack workspace for announcements, Q&A, and to trade tips!

You can’t perform that action at this time.