Higher level alternative to jq for common use cases
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
jqwim
testdata
.gitignore
LICENSE
README.md
README.template
make-readme.py
setup.py

README.md

jqwim -- jq what I mean

A high-level command-line tool for filtering json data. Simpler but less powerful than jq.

Installing

pip install git+https://github.com/talwrii/jqwim#egg=jqwim

Examples / Cheat sheet

# Search for entries whose field value matches a regular expression)
jsonsource | jqwim --field-name regular-expression

# Show the command line flags we can use to filter daya
jsonsource | jqwim --help

# Show general help
jqwim --help < /dev/null

Usage

usage: make-readme.py [--help] [--or] [--spec-file SPEC_FILE]
                      [--args-file ARGS_FILE]

A limited alternative to jq with a simpler interface.The command line options
depend upon your data source (or --spec-file). Provide sample data when
running --help to see source specific options.

optional arguments:
  --help                Show help output
  --or                  Read specification of json from this file rather than
                        guessing. If this file does not exist create it. This
                        protects one against format changes causing crashes.
  --spec-file SPEC_FILE
                        Read specification of json from this file rather than
                        guessing. If this file does not exist create it. This
                        protects one against format changes causing crashes.
  --args-file ARGS_FILE
                        In addition to the command line arguments, read
                        arguments from this this file. Dynamically update
                        filtering as this file is updated

WARNING: Non-homogenous data

The approach used here may be a little surprising. For consistency, unix-style command line flags are used for filtering. Now new language needs to be learned. For convenience and to prevent typing errors, the acceptable flags are derived from the first record. This, however, can cause unexpected behaviour when records have different keys, or suddenly change.

For settings where strict correctness is important, for example unsupervised running in automated scripts (like cronjobs) the --spec-file is provided. This options specifies a json file that contains a list of acceptable keys. When use the first time this file is built from the first key.

Prior work

jq is a highly recommeded tool. It provides a domain specific language for filtering and transforming JSON. For easy use cases it can be rather verbose, however.

recordstream is a similar tool to jq. For both learning and turning-completeness (I imagine), it uses perl fragments for filtering. This has some disadvantages.

There are a number of similar young projects on github. None of these fit my use cases. Understandably, lots of these tools are "code dumps" without documentation which can make quickly working out what they do difficult. There may be other tools that provide very similar functionaly. Some examples include jsongrep, jsongrep, jqwim, jku.

Dynamic update of filters

The --args-file (analogous to the -f option for grep) allows you to load your filters from a file. This file contains a list of command line flags. In addition, these options will be reloaded if this file changes. This is useful if you need to change your filters regularly. Care should be taken to avoid sync-update race conditions however.