xurl is a command-line interface (CLI) utility to pull out bits of URLs.
Visit the releases page and find the appropriate archive for your operating system and architecture. Download the archive from your browser or copy its URL and retrieve it with wget or curl:
-
...with
wget:wget https://github.com/hueristiq/xurl/releases/download/v<version>/xurl-<version>-linux-amd64.tar.gz
-
...or, with
curl:curl -OL https://github.com/hueristiq/xurl/releases/download/v<version>/xurl-<version>-linux-amd64.tar.gz
...then, extract the binary:
tar xf xurl-<version>-linux-amd64.tar.gzTIP: The above steps, download and extract, can be combined into a single step with this onliner
curl -sL https://github.com/hueristiq/xurl/releases/download/v<version>/xurl-<version>-linux-amd64.tar.gz | tar -xzv
NOTE: On Windows systems, you should be able to double-click the zip archive to extract the xurl executable.
...move the xurl binary to somewhere in your PATH. For example, on GNU/Linux and OS X systems:
sudo mv xurl /usr/local/bin/NOTE: Windows users can follow How to: Add Tool Locations to the PATH Environment Variable in order to add xurl to their PATH.
Before you install from source, you need to make sure that Go is installed on your system. You can install Go by following the official instructions for your operating system. For this, we will assume that Go is already installed.
go install -v github.com/hueristiq/xurl/cmd/xurl@latest-
Clone the repository
git clone https://github.com/hueristiq/xurl.git
-
Build the utility
cd xurl/cmd/xurl && \ go build .
-
Move the
xurlbinary to somewhere in yourPATH. For example, on GNU/Linux and OS X systems:sudo mv xurl /usr/local/bin/
NOTE: Windows users can follow How to: Add Tool Locations to the PATH Environment Variable in order to add
xurlto theirPATH.
NOTE: While the development version is a good way to take a peek at xurl's latest features before they get released, be aware that it may have bugs. Officially released versions will generally be more stable.
To display help message for xurl use the -h flag:
$ xurl -hhelp message:
_
__ ___ _ _ __| |
\ \/ / | | | '__| |
> <| |_| | | | |
/_/\_\\__,_|_| |_| v0.0.0
A CLI utility to pull out bits of URLs.
USAGE:
xurl [MODE] [FORMATSTRING] [OPTIONS]
INPUT:
-i, --input input file (use `-` to get from stdin)
OUTPUT:
-m, --monochrome disable output content coloring
-u, --unique output unique values
-v, --verbosity debug, info, warning, error, fatal or silent (default: info)
MODE:
domains the hostname (e.g. sub.example.com)
apexes the apex domain (e.g. example.com from sub.example.com)
paths the request path (e.g. /users)
query `key=value` pairs from the query string (one per line)
params keys from the query string (one per line)
values values from the query string (one per line)
format custom format (see below)
FORMAT DIRECTIVES:
%% a literal percent character
%s the request scheme (e.g. https)
%u the user info (e.g. user:pass)
%d the domain (e.g. sub.example.com)
%S the subdomain (e.g. sub)
%r the root of domain (e.g. example)
%t the TLD (e.g. com)
%P the port (e.g. 8080)
%p the path (e.g. /users)
%e the path's file extension (e.g. jpg, html)
%q the raw query string (e.g. a=1&b=2)
%f the page fragment (e.g. page-section)
%@ inserts an @ if user info is specified
%: inserts a colon if a port is specified
%? inserts a question mark if a query string exists
%# inserts a hash if a fragment exists
%a authority (alias for %u%@%d%:%P)
EXAMPLES:
cat urls.txt | xurl params -i -
cat urls.txt | xurl format %s://%h%p?%q -i -
$ cat urls.txt
https://sub.example.com/users?id=123&name=Sam
https://sub.example.com/orgs?org=ExCo#about
http://example.net/about#contact
You can extract the domains from the URLs with the domains mode:
$ cat urls.txt | xurl domains -i -
sub.example.com
sub.example.com
example.net
If you don't want to output duplicate values you can use the -u or --unique flag:
```
$ cat urls.txt | xurl domains -i - --unique
sub.example.com
example.net
```
The -u/--unique flag works for all modes.
You can extract the apex part of the domain (e.g. the example.com in http://sub.example.com) using the apexes mode:
$ cat urls.txt | unfurl apexes -i - -u
example.com
example.net
$ cat urls.txt | xurl paths -i -
/users
/orgs
/about
$ cat urls.txt | xurl query -i -
id=123
name=Sam
org=ExCo
$ cat urls.txt | xurl params -i -
id
name
org
$ cat urls.txt | xurl values -i -
123
Sam
ExCo
You can use the format mode to specify a custom output format:
$ cat urls.txt | xurl format %d%p -i -
sub.example.com/users
sub.example.com/orgs
example.net/about
The available format directives are:
%% A literal percent character
%s The request scheme (e.g. https)
%u The user info (e.g. user:pass)
%d The domain (e.g. sub.example.com)
%S The subdomain (e.g. sub)
%r The root of domain (e.g. example)
%t The TLD (e.g. com)
%P The port (e.g. 8080)
%p The path (e.g. /users)
%e The path's file extension (e.g. jpg, html)
%q The raw query string (e.g. a=1&b=2)
%f The page fragment (e.g. page-section)
%@ Inserts an @ if user info is specified
%: Inserts a colon if a port is specified
%? Inserts a question mark if a query string exists
%# Inserts a hash if a fragment exists
%a Authority (alias for %u%@%d%:%P)
For more format directives, checkout the help message
xurl -hunderFormat Directives.
Any characters that don't match a format directive remain untouched:
$ cat urls.txt | xurl format "%d (%s)" -i - -u
sub.example.com (https)
example.net (http)
Note that if a URL does not include the data requested, there will be no output for that URL:
$ echo http://example.com | xurl format "%P" -i -
$ echo http://example.com:8080 | xurl format "%P" -i -
8080
- Tom Hudson, we took the initial code from his unfurl.
Issues and Pull Requests are welcome! Check out the contribution guidelines.
This utility is distributed under the MIT license