Permalink
Browse files

Streamed terminal output

`--stream` can be used to enable streaming also with `--pretty` and to ensure
a more frequent output flushing.
  • Loading branch information...
jakubroztocil committed Aug 2, 2012
1 parent 4615011 commit c7657e3c4b51d86be9d62c2b9d05f5371a2b02fc
Showing with 618 additions and 388 deletions.
  1. +3 −0 .gitignore
  2. +35 −8 README.rst
  3. +0 −3 httpie/__init__.py
  4. +20 −5 httpie/cli.py
  5. +64 −36 httpie/core.py
  6. +57 −39 httpie/models.py
  7. +214 −89 httpie/output.py
  8. +225 −208 tests/tests.py
View
@@ -4,3 +4,6 @@ build
*.pyc
.tox
README.html
+.coverage
+htmlcov
+
View
@@ -79,7 +79,7 @@ There are five different types of key/value pair ``items`` available:
| | nested ``Object``, or an ``Array``. It's because |
| | simple data items are always serialized as a |
| | ``String``. E.g., ``pies:=[1,2,3]``, or |
-| | ``'meals:=["ham","spam"]'`` (note the quotes). |
+| | ``meals:='["ham","spam"]'`` (note the quotes). |
| | It may be more convenient to pass the whole JSON |
| | body via ``stdin`` when it's more complex |
| | (see examples bellow). |
@@ -221,18 +221,31 @@ respectively:
esac
fi
+**The output is always streamed** unless ``--pretty`` is set or implied. You
+can use ``--stream`` / ``-S`` to enable streaming even with ``--pretty``, in
+which case every line of the output will processed and flushed as soon as it's
+avaialbe (as opossed to buffering the whole response which wouldn't work for
+long-lived requests). You can test it with the Twitter streaming API:
+
+.. code-block:: shell
+
+ http -Sfa <your-twitter-username> https://stream.twitter.com/1/statuses/filter.json track='Justin Bieber'
+ # \/
+ # The short options for --stream, --form and --auth.
+
+``--stream`` can also be used regardless of ``--pretty`` to ensure a more
+frequent output flushing (sort of like ``tail -f``).
Flags
-----
``$ http --help``::
- usage: http [--help] [--version] [--json | --form] [--traceback]
- [--pretty | --ugly]
+ usage: http [--help] [--version] [--json | --form] [--pretty | --ugly]
[--print OUTPUT_OPTIONS | --verbose | --headers | --body]
- [--style STYLE] [--check-status] [--auth AUTH]
+ [--style STYLE] [--stream] [--check-status] [--auth AUTH]
[--auth-type {basic,digest}] [--verify VERIFY] [--proxy PROXY]
- [--allow-redirects] [--timeout TIMEOUT]
+ [--allow-redirects] [--timeout TIMEOUT] [--debug]
[METHOD] URL [ITEM [ITEM ...]]
HTTPie - cURL for humans. <http://httpie.org>
@@ -266,7 +279,6 @@ Flags
-www-form-urlencoded (if not specified). The presence
of any file fields results into a multipart/form-data
request.
- --traceback Print exception traceback should one occur.
--pretty If stdout is a terminal, the response is prettified by
default (colorized and indented if it is JSON). This
flag ensures prettifying even when stdout is
@@ -282,7 +294,7 @@ Flags
piped to another program or to a file, then only the
body is printed by default.
--verbose, -v Print the whole request as well as the response.
- Shortcut for --print=HBhb.
+ Shortcut for --print=HBbh.
--headers, -h Print only the response headers. Shortcut for
--print=h.
--body, -b Print only the response body. Shortcut for --print=b.
@@ -291,10 +303,19 @@ Flags
colorful, default, emacs, friendly, fruity, manni,
monokai, murphy, native, pastie, perldoc, rrt,
solarized, tango, trac, vim, vs. Defaults to
- solarized. For this option to work properly, please
+ "solarized". For this option to work properly, please
make sure that the $TERM environment variable is set
to "xterm-256color" or similar (e.g., via `export TERM
=xterm-256color' in your ~/.bashrc).
+ --stream, -S Always stream the output by line, i.e., behave like
+ `tail -f'. Without --stream and with --pretty (either
+ set or implied), HTTPie fetches the whole response
+ before it outputs the processed data. Set this option
+ when you want to continuously display a prettified
+ long-lived response, such as one from the Twitter
+ streaming API. It is useful also without --pretty: It
+ ensures that the output is flushed more often and in
+ smaller chunks.
--check-status By default, HTTPie exits with 0 when no network or
other fatal errors occur. This flag instructs HTTPie
to also check the HTTP status code and exit with an
@@ -321,6 +342,9 @@ Flags
POST-ing of data at new ``Location``)
--timeout TIMEOUT Float describes the timeout of the request (Use
socket.setdefaulttimeout() as fallback).
+ --debug Prints exception traceback should one occur and other
+ information useful for debugging HTTPie itself.
+
Contribute
@@ -373,6 +397,9 @@ Changelog
=========
* `0.2.7dev`_
+ * Streamed terminal output. ``--stream`` / ``-S`` can be used to enable
+ streaming also with ``--pretty`` and to ensure a more frequent output
+ flushing.
* Support for efficient large file downloads.
* Response body is fetched only when needed (e.g., not with ``--headers``).
* Improved content type matching.
View
@@ -5,6 +5,3 @@
__author__ = 'Jakub Roztocil'
__version__ = '0.2.7dev'
__licence__ = 'BSD'
-
-
-CONTENT_TYPE = 'Content-Type'
View
@@ -9,7 +9,7 @@
from . import __doc__
from . import __version__
-from .output import AVAILABLE_STYLES
+from .output import AVAILABLE_STYLES, DEFAULT_STYLE
from .input import (Parser, AuthCredentialsArgType, KeyValueArgType,
PRETTIFY_STDOUT_TTY_ONLY,
SEP_PROXY, SEP_CREDENTIALS, SEP_GROUP_ITEMS,
@@ -56,7 +56,7 @@ def _(text):
parser.add_argument(
- '--output', '-o', type=argparse.FileType('wb'),
+ '--output', '-o', type=argparse.FileType('w+b'),
metavar='FILE',
help= argparse.SUPPRESS if not is_windows else _(
'''
@@ -131,16 +131,31 @@ def _(text):
)
parser.add_argument(
- '--style', '-s', dest='style', default='solarized', metavar='STYLE',
+ '--style', '-s', dest='style', default=DEFAULT_STYLE, metavar='STYLE',
choices=AVAILABLE_STYLES,
help=_('''
- Output coloring style, one of %s. Defaults to solarized.
+ Output coloring style, one of %s. Defaults to "%s".
For this option to work properly, please make sure that the
$TERM environment variable is set to "xterm-256color" or similar
(e.g., via `export TERM=xterm-256color' in your ~/.bashrc).
- ''') % ', '.join(sorted(AVAILABLE_STYLES))
+ ''') % (', '.join(sorted(AVAILABLE_STYLES)), DEFAULT_STYLE)
)
+parser.add_argument('--stream', '-S', action='store_true', default=False, help=_(
+ '''
+ Always stream the output by line, i.e., behave like `tail -f'.
+
+ Without --stream and with --pretty (either set or implied),
+ HTTPie fetches the whole response before it outputs the processed data.
+
+ Set this option when you want to continuously display a prettified
+ long-lived response, such as one from the Twitter streaming API.
+
+ It is useful also without --pretty: It ensures that the output is flushed
+ more often and in smaller chunks.
+
+ '''
+))
parser.add_argument(
'--check-status', default=False, action='store_true',
help=_('''
View
@@ -3,20 +3,27 @@
Invocation flow:
1. Read, validate and process the input (args, `stdin`).
- 2. Create a request and send it, get the response.
- 3. Process and format the requested parts of the request-response exchange.
- 4. Write to `stdout` and exit.
+ 2. Create and send a request.
+ 3. Stream, and possibly process and format, the requested parts
+ of the request-response exchange.
+ 4. Simultaneously write to `stdout`
+ 5. Exit.
"""
import sys
import json
+import errno
+from itertools import chain
+from functools import partial
import requests
import requests.auth
from requests.compat import str
from .models import HTTPRequest, HTTPResponse, Environment
-from .output import OutputProcessor, formatted_stream
+from .output import (OutputProcessor, RawStream, PrettyStream,
+ BufferedPrettyStream, EncodedStream)
+
from .input import (OUT_REQ_BODY, OUT_REQ_HEAD,
OUT_RESP_HEAD, OUT_RESP_BODY)
from .cli import parser
@@ -85,41 +92,50 @@ def output_stream(args, env, request, response):
"""
- prettifier = (OutputProcessor(env, pygments_style=args.style)
- if args.prettify else None)
+ # Pick the right stream type for this exchange based on `env` and `args`.
+ if not env.stdout_isatty and not args.prettify:
+ Stream = partial(
+ RawStream,
+ chunk_size=RawStream.CHUNK_SIZE_BY_LINE
+ if args.stream
+ else RawStream.CHUNK_SIZE)
+ elif args.prettify:
+ Stream = partial(
+ PrettyStream if args.stream else BufferedPrettyStream,
+ processor=OutputProcessor(env, pygments_style=args.style),
+ env=env)
+ else:
+ Stream = partial(EncodedStream, env=env)
- with_request = (OUT_REQ_HEAD in args.output_options
- or OUT_REQ_BODY in args.output_options)
- with_response = (OUT_RESP_HEAD in args.output_options
- or OUT_RESP_BODY in args.output_options)
+ req_h = OUT_REQ_HEAD in args.output_options
+ req_b = OUT_REQ_BODY in args.output_options
+ resp_h = OUT_RESP_HEAD in args.output_options
+ resp_b = OUT_RESP_BODY in args.output_options
- if with_request:
- request_iter = formatted_stream(
- msg=HTTPRequest(request),
- env=env,
- prettifier=prettifier,
- with_headers=OUT_REQ_HEAD in args.output_options,
- with_body=OUT_REQ_BODY in args.output_options)
+ req = req_h or req_b
+ resp = resp_h or resp_b
- for chunk in request_iter:
- yield chunk
+ output = []
- if with_request and with_response:
- yield b'\n\n\n'
+ if req:
+ output.append(Stream(
+ msg=HTTPRequest(request),
+ with_headers=req_h,
+ with_body=req_b))
- if with_response:
- response_iter = formatted_stream(
- msg=HTTPResponse(response),
- env=env,
- prettifier=prettifier,
- with_headers=OUT_RESP_HEAD in args.output_options,
- with_body=OUT_RESP_BODY in args.output_options)
+ if req and resp:
+ output.append([b'\n\n\n'])
- for chunk in response_iter:
- yield chunk
+ if resp:
+ output.append(Stream(
+ msg=HTTPResponse(response),
+ with_headers=resp_h,
+ with_body=resp_b))
if env.stdout_isatty:
- yield b'\n\n'
+ output.append([b'\n\n'])
+
+ return chain(*output)
def get_exist_status(code, allow_redirects=False):
@@ -170,18 +186,30 @@ def main(args=sys.argv[1:], env=Environment()):
except AttributeError:
buffer = env.stdout
- for chunk in output_stream(args, env, response.request, response):
- buffer.write(chunk)
- if env.stdout_isatty:
- env.stdout.flush()
+ try:
+ for chunk in output_stream(args, env, response.request, response):
+ buffer.write(chunk)
+ if env.stdout_isatty or args.stream:
+ env.stdout.flush()
+
+ except IOError as e:
+ if debug:
+ raise
+ if e.errno == errno.EPIPE:
+ env.stderr.write('\n')
+ else:
+ env.stderr.write(str(e) + '\n')
+ return 1
except (KeyboardInterrupt, SystemExit):
+ if debug:
+ raise
env.stderr.write('\n')
return 1
except Exception as e:
if debug:
raise
- env.stderr.write(str(e.message) + '\n')
+ env.stderr.write(str(e) + '\n')
return 1
return status
Oops, something went wrong.

0 comments on commit c7657e3

Please sign in to comment.