wgrep is a command line utility for running selector queries against URLs
JavaScript
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
.gitignore
README.md
package.json
wgrep.js

README.md

wgrep

wgrep is a command line utility for running selector queries against HTTP-based resources.

Examples

wgrep -q "a" http://google.com

This example will return a tab-delimited list of link text and resources they are linked to.

If you want only the a tags that have an href:

wgrep -q "a[href]" http://google.com

The q flag accepts a css selector for text/html (and XML/XHTML) MIME'd resources.

But what if you've got a JSON url?

wgrep -miq "$.feed.entry[*].title[*].$t" "https://gdata.youtube.com/feeds/api/videos?q=cats&v=2&alt=json"

The above will show you an ordered list of the title of the top 25 search results on YouTube for "cats".

We use JSONPath to handle JSON query operations.

Usage

wgrep is your standard command line utility. As such, you run it basically with:

wgrep [options] <url>

An HTTP URL is required (although we are definitely thinking about adding file and stdin inputs as well).

Install

Install using npm - we recommend a global install:

npm install -g wgrep
...
which wgrep
> /usr/local/bin/wgrep

Now you should have a working wgrep command.

Options

  • -h, --help: Prints out the help message.

  • -V, --version: Prints out the version number.

  • -u, --user <username>: Allows you to provide a username for basic auth.

  • -p, --password [password]: Allows you to provide a password for basic auth. Will ask for one if it's not provided.

  • -q, --query <query>: Allows you to provide a query selector string to run against the resource.

  • -m, --markdown: Returns results in a (best-guess) Markdown format.

    This works really nicely with links and images:

    wgrep -mq "a[href]" http://google.com
    wgrep -mq "img" http://google.com
  • -i, --ordered: With -m, returns results as an ordered list.

  • -l, --list: With -m, returns results as an unordered list.

  • -j, --json: Returns results as a JSON string.

  • -n, --inspect: With -j, inspects the JSON results, with a default depth of 2 and colorized output.

Credits

Written by Paul DeLeeuw for debugging URLs and other HTTP-based shenanigans.

Written in Javascript, runs in node. Makes use of several awesome open source projects, like

Thanks to all the contributors of these fine projects.