wgrep is a command line utility for running selector queries against HTTP-based
wgrep -q "a" http://google.com
This example will return a tab-delimited list of link text and resources they are linked to.
If you want only the
a tags that have an href:
wgrep -q "a[href]" http://google.com
q flag accepts a css selector for
text/html (and XML/XHTML) MIME'd
But what if you've got a JSON url?
wgrep -miq "$.feed.entry[*].title[*].$t" "https://gdata.youtube.com/feeds/api/videos?q=cats&v=2&alt=json"
The above will show you an ordered list of the title of the top 25 search results on YouTube for "cats".
We use JSONPath to handle JSON query operations.
wgrep is your standard command line utility. As such, you run it basically
wgrep [options] <url>
An HTTP URL is required (although we are definitely thinking about adding file and stdin inputs as well).
Install using npm - we recommend a global install:
npm install -g wgrep ... which wgrep > /usr/local/bin/wgrep
Now you should have a working wgrep command.
-h, --help: Prints out the help message.
-V, --version: Prints out the version number.
-u, --user <username>: Allows you to provide a username for basic auth.
-p, --password [password]: Allows you to provide a password for basic auth. Will ask for one if it's not provided.
-q, --query <query>: Allows you to provide a query selector string to run against the resource.
-m, --markdown: Returns results in a (best-guess) Markdown format.
This works really nicely with links and images:
wgrep -mq "a[href]" http://google.com wgrep -mq "img" http://google.com
-i, --ordered: With
-m, returns results as an ordered list.
-l, --list: With
-m, returns results as an unordered list.
-j, --json: Returns results as a JSON string.
-n, --inspect: With -j, inspects the JSON results, with a default depth of 2 and colorized output.
Written by Paul DeLeeuw for debugging URLs and other HTTP-based shenanigans.
Thanks to all the contributors of these fine projects.