If you just want to use it already.
Declare dependency in your project.clj:
(defproject your-project "1.0.0-SNAPSHOT"
:description "Your project description"
:dependencies [[org.clojure/clojure "1.2.1"]
[org.clojure/clojure-contrib "1.2.0"]
[http.async.client "0.3.1"]])Make sure that your project depends on at least 1.2.0 Clojure as http.async.client will not run in earlier versions.
Require it from your code:
(ns your.ns (:require [http.async.client :as client]))To get HTTP resource:
(with-open [client (client/create-client)] ;; Create client
;; request http resource
(let [response (client/GET client "http://github.com/neotyk/http.async.client/")]
;; wait for response to be received
(client/await response)
;; read body of response as string
(client/string response)))When you do:
(client/GET <client> url)Result will be a map of *http.async.client.util/promise*s, and represents response. Do not fear, this is clojure.core/promise that was extended to be able to check if it was delivered.
Following HTTP methods have been covered so far:
For detailed description see HTTP methods.
You can submit options to HTTP methods as keyworded arguments, like this:
(client/GET <client> url :query {:key "value"})Following options are supported:
- :query
- query parameters
- :headers
- custom headers to be sent out
- :body
- body to be sent, allowed only with PUT/POST
- :cookies
- cookies to be sent
- :proxy
- proxy to be used
For detailed usage of options see Request options.
Response map contains following keys:
- :status
- promise of lazy map of status fields
- :code
- response code
- :msg
- response message
- :protocol
- protocol with version
- :major
- major version of protocol
- :minor
- minor version of protocol
- :headers
- promise of lazy map of headers where header names are keyworded, like :server for example
- :body
- promise of response body, this is ByteArrayOutputStream, but you
have convenience functions to convert it for example to string:
(client/string (client/GET <client> <url>))
- :done
- promise that is delivered once response receiving is done
- :error
- promise, if there was an error you will find Throwable here
For consuming HTTP streams use:
(client/stream-seq <client> :get url)Response here is same as in Asynchronous operations but :body will be lazy sequence of ByteArrayOutputStreams.
You can still use convenience functions like client/string for body, but remember that you are dealing now with seq.
For more details please see Lazy sequence.
This allows you to provide callbacks that will get triggered on HTTP response events like:
- received status line,
- received headers,
- received body part,
- completed request,
- handle error.
All callbacks are expected to return tuple with first element been a value to be delivered for given response processing phase, second element is controlling execution and if you make it :abort than processing response is going to be terminated.
For detailed information on how to use this mode please see Low level.
HTTP methods and convenience functions to request them.
Most basic invocation of *http.async.client/GET* is only with url you want to get. Extended invocation includes options that can be any options accepted by *http.async.client.request/prepare-request* [:headers :query ..].
Simple invocation:
(with-open [client (client/create-client)] ;; Create client
(let [resp (client/GET client "<your url>")
status (client/status resp)
headers (client/headers resp)]
(println (:code status))
(client/await resp)
(println (client/string resp))))Invocation with query parameters:
(with-open [client (client/create-client)] ;; Create client
(let [resp (client/GET client "<your url>" :query {:param-name "some-value"})
status (client/status resp)
headers (client/headers resp)]
(println (:code status))
(client/await resp)
(println (client/string resp))))Invocation with proxy:
(with-open [client (client/create-client)] ;; Create client
(let [resp (client/GET client "<your url>"
:query {:param-name "some-value"}
:proxy {:host host :port port})
status (client/status resp)]
(println (:code status))
(client/await resp)
(println (client/string resp))))Invocation with cookies:
(with-open [client (client/create-client)] ;; Create client
(let [resp (client/GET client "http://localhost:8123/cookie"
;; Send cookie
:cookies #{{:domain "http://localhost:8123/"
:name "sample-name"
:value "sample-value"
:path "/cookie"
:max-age 10
:secure false}})]
(doseq [cookie (client/cookies resp)] ;; Read cookies from server response
(println "name:" (:name cookie) ", value:" (:value cookie)))))Notice *http.async.client/cookies* function extracts cookies from response headers, so to start processing it you don’t need to wait for whole response to arrive.
*http.async.client/PUT*/*http.async.client/POST* work the same way as *GET* but they also accept :body argument.
:body can be:
- String
- map, for easy form data submissions
- InputStream for any content
- File for zero byte copy
You can send String as body with PUT/POST:
(with-open [client (client/create-client)] ;; Create client
(let [resp (client/POST client "<your url>" :body "SampleBody")]
; do something with resp
))Submitting parameters via body map:
(with-open [client (client/create-client)] ;; Create client
(let [resp (client/POST client "<your url>" :body {:u "user" :p "s3cr3t"})]
; do something with resp
))Another method to provide body is via InputStream:
(use '[clojure.java.io :only [input-stream]])
(with-open [client (client/create-client)] ;; Create client
(let [resp (client/PUT client "<your url>" :body (input-stream (.getBytes "SampleContent" "UTF-8")))]
; do something with resp
))To use zero byte copy future, provide a File as :body
(import '(java.io File))
(with-open [client (client/create-client)] ;; Create client
(let [resp (client/PUT "<your url>" :body (File. "<path to file>"))]
;; do something with resp
))To call *http.async.client/DELETE* on a resource:
(with-open [client (client/create-client)] ;; Create client
(let [resp (client/DELETE "<your url>")]
; do something with resp
))To call *http.async.client/HEAD* on a resource:
(with-open [client (client/create-client)] ;; Create client
(let [resp (client/HEAD "<your url>")]
; do something with resp
))To call *http.async.client/OPTIONS* on a resource:
(with-open [client (client/create-client)] ;; Create client
(let [resp (client/OPTIONS "<your url>")]
; do something with resp
))Following options can be provided to requests and are defined by *http.async.client.request/prepare-request*:
- :query
- query parameters
- :headers
- custom headers to be sent out
- :body
- body to be sent, allowed only with PUT/POST
- :cookies
- cookies to be sent
- :proxy
- proxy to be used
- :auth
- authentication map
- :timeout
- timeout configuration
Query parameters is a map of keywords and their values. You use it like so:
(with-open [client (client/create-client)] ;; Create client
(let [resp (client/GET client url :query {:key1 "value1" :key2 "value2"})]
(client/await resp)
(client/string resp)))Custom headers can be submitted same way as :query:
(with-open [client (client/create-client)] ;; Create client
(let [resp (client/GET client url :headers {:header-name1 "value1"
:header-name2 "value2"})]
(client/await resp)
(client/string resp)))Body can be provided with a message only with PUT/POST, it doesn’t make sense to have body with other HTTP methods.
(with-open [client (client/create-client)] ;; Create client
(let [resp (client/PUT client url :body "sample body")]
(client/await resp)
(client/string resp))):body can be String, form parameters (that is map), input stream or java.io.File, please see PUT/POST for more documentation.
Cookies can be provided to request as follows:
(with-open [client (client/create-client)] ;; Create client
(let [resp (client/GET client "http://localhost:8123/cookie"
:cookies #{{:domain "http://localhost:8123/"
:name "sample-name"
:value "sample-value"
:path "/cookie"
:max-age 10
:secure false}})]
(client/await resp)
(client/string resp))):cookies option takes sequence of cookie maps, in this example a hash set. Cookie map consist of:
- :domain
- Domain that cookie has been installed
- :name
- Cookie name
- :value
- Cookie value, note that there is no additional processing so you should encode it yourself if needed.
- :path
- Path on with cookie has been installed
- :max-age
- Max age that cookie was configured to live
- :secure
- If cookie is secure cookie
Cookie reading is described in Reading cookies.
Proxy can be configured per request basis as follows:
(with-open [client (client/create-client)] ;; Create client
(let [resp (client/GET client url :proxy {:host h :port p})]
(client/await resp)
(client/string resp)))Proxy expects a map with following keys:
- :host
- proxy host
- :port
- proxy port
- :protocol
- optional protocol to communicate with proxy. Can be :http (default) or :https.
- :user
- optional user name to use for proxy authentication. Must be provided with :password.
- :password
- optional password to use for proxy authentication. Must be provided with :user.
Authentication can be configured per request basis. For now BASIC and DIGEST methods are supported.
Basic method is default, so you don’t have to specify it:
(with-open [client (client/create-client)] ;; Create client
(let [resp (client/GET client url :auth {:user u :password p})]
;; Check if response is not 401 or so and process response
))Though you can:
(with-open [client (client/create-client)] ;; Create client
(let [resp (client/GET client url :auth {:type :basic :user u :password p})]
;; Check if response is not 401 or so and process response
))And for digest method you will need realm as well:
(with-open [client (client/create-client)] ;; Create client
(let [resp (client/GET client url
:auth {:type :digest :user u :password p :realm r})]
;; Check if response is not 401 or so and process response
))Controlling preemptive authentication behavior is also possible:
(with-open [client (client/create-client)]
(let [resp (client/GET client url :auth {:user u :password p :preemptive true})]
;; process response
))Response timeout can be configured per request as well. Timeout value is time in milliseconds in which response has to be received. There is special value -1 that indicates infinite timeout.
(with-open [client (client/create-client)] ;; Create client
(let [resp (client/GET client url :timeout -1)]
(client/await resp)
;; process response
))Sample above will wait until response is fully received, as long as it takes (-1 timeout).
(with-open [client (client/create-client)] ;; Create client
(let [resp (client/GET client url :timeout 100)]
(client/await resp)
(if (client/failed? resp)
;; did not get response in configured timeout
;; process response
)))Example above configures timeout to 100ms, so await will only wait for 100ms, after that response is done. Which doesn’t necessarily mean that it was delivered to client successfully, because it was restricted by timeout, that is why example contains check if response has failed.
HTTP Stream is response with chunked content encoding. Those streams might not be meant to ever finish, see twitter.com streams, so collecting those responses as a whole is impossible, they should be processed by response parts (chunks) as they are been received.
Two ways of consuming a HTTP Stream are supported:
You can get HTTP Stream as lazy sequence of it’s body. This is very convenient method as seq is native type of Clojure so you can apply all mapping, filtering and any other standard function that you like to it.
(with-open [client (client/create-client)] ;; Create client
(let [resp (client/stream-seq client :get url)]
(doseq [s (s/string resp)]
(println s))))stream-seq arguments:
- http method
- any of supported methods can be used, though it makes sense only to use :get, :put and :post
- url
- URL of HTTP resource
- options
- same as normal Request options.
It is important to understand that seqs returned by body or string (which in turn calls body) are backed by queue. One of consequences of it is that once you consumed some body parts they will not be available anymore. Let’s see code speak for itself.
(let [resp (client/stream-seq :get url)]
(println "1: " (first (client/string resp)))
(println "2: " (first (client/string resp))))This code will print following:
1: part1 2: part2
Assuming that part1 is first chunk and part2 is second.
Second consequence of been directly backed by queue is that you can have multiple consumers of same response and non of them will get same body part.
And finally this implementation is not holding to it’s head.
Consuming HTTP Stream with call-back is quite straight forward with http.async.client. You will need to know what HTTP Method you will call, what URL and provide a call back function to handle body parts been received.
(with-open [client (client/create-client)] ;; Create client
(let [parts (ref #{})
resp (client/request-stream client :get url
(fn [state body]
(dosync (alter parts conj (string body)))
[body :continue]))]
;; do something to @parts
))Few notes on implementing body part callback:
- state is a map with :status and :headers as promises, at stage when you get called for body part, both of them should be in place already, though it is advised to use convenience methods to read them, see Reading status line and Reading headers,
- call-back has to follow guidelines described in Body part,
- some streams are not meant to be finish, in that case don’t collect body parts, as for sure you will run out of available resources,
- try not to do any heavy lifting in this callback, better send it to agent.
http.async.client exposes some convenience functions for response handling.
If you call any of Asynchronous operations, Streaming or Raw mode you actually asynchronously execute HTTP request. Some times you might need to wait for response processing to be done before proceeding, in order to do so you call *http.async.client/await*. It takes only one argument, that is response and returns once receiving has finished.
(with-open [client (client/create-client)] ;; Create client
(let [resp (client/GET client url)]
(client/await resp)))Sample above will behave like synchronous HTTP operation. For convenience it returns same response so you can use it further, for example like that:
(with-open [client (client/create-client)] ;; Create client
(let [resp (client/GET client url)]
(client/string (client/await resp))))*http.async.client/status* returns status lazy map of response. It will wait until HTTP Status has been received.
(with-open [client (client/create-client)] ;; Create client
(let [resp (client/GET client url)
status (client/status resp)]
(:code status)))Sample above will return HTTP response status code, notice that after this returns headers and body, might not been delivered yet.
*http.async.client/headers* returns headers lazy map of response. It will wait until HTTP Headers are received.
(with-open [client (client/create-client)] ;; Create client
(let [resp (client/GET client url)
headers (client/headers resp)]
(:server headers)))Again, like in case of status, body might not have been delivered yet after this returns.
*http.async.client/cookies* returns seq of maps representing cookies. It will wait until HTTP Headers are received.
(with-open [client (client/create-client)] ;; Create client
(let [resp (client/GET client url)
cookies (client/cookies resp)]
(map :name cookies)))Sample above will return sequence of cookie names that server has set.
*http.async.client/body* returns either ByteArrayOutputStream or seq of it, depending if you used Asynchronous operations or Streaming respectively. It will not wait for response to be finished, it will return as soon as first chunk of HTTP response body is received.
*http.async.client/string* returns either string or seq of strings, again depending if you used Asynchronous operations or Streaming respectively. It will not wait for response to be finished, it will return as soon as first chunk of HTTP response body is received.
(with-open [client (client/create-client)] ;; Create client
(let [resp (client/GET client url)]
(client/string (client/await resp))))Sample above will return string of response body. *http.async.client/string* is lazy so you can use it in case of streams as well.
(with-open [client (client/create-client)] ;; Create client
(let [resp (client/stream-seq client :get url)
strings (client/string resp)]
(doseq [part strings]
(println part))))Sample above will print parts as they are received, and will return once response receiving is finished.
*http.async.client/error* will return Throwable that was cause of request failure iff request failed, else nil.
(with-open [client (client/create-client)] ;; Create client
(let [resp (client/GET client url)]
(client/await resp)
(when-let [err (client/error resp)]
(println "failed processing request: " err))))At any given time of processing HTTP Response you can cancel it by calling *http.async.client/cancel*.
(with-open [client (client/create-client)] ;; Create client
(let [resp (client/GET client url)]
(client/cancel resp)))Please see canceling-request test.
You can also check status of request.
*http.async.client/done?* will tell you if response processing has finished:
(with-open [client (client/create-client)] ;; Create client
(let [resp (client/GET client url)]
(when-not (client/done? resp)
(client/await resp)
(client/done? resp))))Sample above will check if response was finished, if not - will wait for it and return true as a result of call to done?.
*http.async.client/failed?* will return true iff request has failed. If this return true you can read error.
*http.async.client/canceled?* will return true iff request has been canceled, else false is return.
http.async.client can be configured with User-Agent. To do so you can use *http.async.client/create-client* and remember to close created client yourself, best is to use it within macro like with-open, though make sure that body of it will wait for whore response to finish.
(with-open [client (client/create-client {:user-agent "Your User Agent/1.0"})]
(let [resp (client/GET client url)]
;; do stuff with resp
))http.async.client can be configured to allow, or not, HTTP compression.
(with-open [client (client/create-client {:compression-enabled true})]
(let [resp (client/GET client url)]
;; do stuff with resp
))Enabling HTTP redirects following.
(with-open [client (client/create-client {:follow-redirects true})]
(let [resp (client/GET client url)]
;; do stuff with resp
))Keep Alive is enabled by default. This implies using pool for connections.
(with-open [client (client/create-client {:keep-alive true})]
(let [resp (client/GET client url)]
;; do stuff with resp
))Maximum number of connections to be cached per host. Above this number connections will still be created but will not be kept alive.
(with-open [client (client/create-client {:max-conns-per-host 10})]
(let [resp (client/GET client url)]
;; do stuff with resp
))Maximum number of total connections opened, submitting new request while all allowed connections are active, will result in rejection.
(with-open [client (client/create-client {:max-conns-total 100})]
(let [resp (client/GET client url)]
;; do stuff with resp
))Maximum number of redirects to follow.
(with-open [client (client/create-client {:max-redirects 3})]
(let [resp (client/GET client url)]
;; do stuff with resp
))With http.async.client apart from per connection :timeout you can globally configure connection, request and idle timeouts. All timeout values are in milliseconds and magic value -1 is interpreted as infinite wait. idle connection in pool timeout works only on connections in pool, connections idle, for configured time, in pool will be closed.
(with-open [client (client/create-client {:connection-timeout 10
:request-timeout 1000
:idle-in-pool-timeout 100})]
(let [resp (client/client client/GET url)]
;; request processing
))Example above will timeout connection if it was not established in 10ms, request if it was not received in 1sec, or connection when it was idling in pool for more than 100ms.
Client can be also configured with global HTTP Proxy settings.
(with-open [client (client/create-client {:proxy {:host h :port p}})]
(let [resp (client/GET client url)]
;; do stuff with resp
))Proxy expects a map with following keys:
- :host
- proxy host
- :port
- proxy port
- :protocol
- optional protocol to communicate with proxy. Can be :http (default) or :https.
- :user
- optional user name to use for proxy authentication. Must be provided with :password.
- :password
- optional password to use for proxy authentication. Must be provided with :user.
Default authentication realm to be used globally can be configured. For now BASIC and DIGEST methods are supported.
Basic method is default, so you don’t have to specify it:
(with-open [client (client/create-client {:auth {:user u :password p}})]
(let [resp (client/GET client url)]
;; Check if response is not 401 or so and process response
))Though you can:
(with-open [client (client/create-client :auth {:type :basic :user u :password p})]
(let [resp (client/GET client url)]
;; Check if response is not 401 or so and process response
))And for digest method you will need realm as well:
(with-open [client (client/create-client :auth {:type :digest :user u :password p :realm r})]
(let [resp (client/GET client url)]
;; Check if response is not 401 or so and process response
))Preemptive authentication can be enabled or disabled globally per client:
(with-open [client (client/create-client :auth {:type :basic :user u :passowrd p :preemptive true})]
(let [resp (client/GET client url)]
;; process response
))Whenever you’ve created http.async.client via *http.async.client/create-client* you will need to close it. To do so you call *http.async.client/close*, http.async.client relies on client been bound to **client**.
(let [client (client/create-client)]
(try
(let [resp (client/GET client url)]
;; process response
)
(finally
(client/close *client*))))This is lowest level access to http.async.client. Mechanics here is based on asynchronous call-backs. It provides default set of callbacks and functions to create and execute requests.
*http.async.client.request/prepare-request* is responsible for request preparation, like name suggests. It takes following arguments:
- HTTP Method like :get :head
- url that you want to call
- and options, a keyworded map described already in Request options.
Sample:
(with-open [client (client/create-client)] (let [req (prepare-request client :get "http://google.com" :headers {:my-header "value"})] ;; now you have request, next thing to do would be to execute it ))
*http.async.client/execute-request* returns same map of promises as Asynchronous operations. Its arguments are: request to be executed (result of Preparing request) and options as keyworded map consisting of call-backs. Following options are recognized:
- :status Status line
- :headers Headers
- :part Body part
- :completed Body completed
- :error Error
All callbacks take response map as first argument and callback specific argument if any. Callbacks are expected to return tuple of result and action:
- result
- will be delivered to respective promise in response map
- action
- if its value is :abort than response processing will be aborted, anything else here will result in continuation.
Status line callback gets called after status line has been received with arguments:
- response map
- Status map has following keys:
- :code status code (200, 404, ..)
- :msg status message (“OK”, ..)
- :protocol protocol with version (“HTTP/1.1”)
- :major major protocol version (1)
- :minor minor protocol version (0, 1)
Headers callback gets called after headers have been received with arguments:
- response map
- lazy map of headers. Keys in that map are (keyword (.toLowerCase <header name>)), so “Server” headers is :server and so on.
Body part callback gets called after each part of body has been received with arguments:
- response map
- ByteArrayOutputStream that contains body part received.
This callback gets called when receiving of response body has finished with only one argument, i.e. response map.
Error callback gets called when error while processing has been encountered with arguments
- response map
- Throwable that was a cause of failure
*http.async.client.request/*default-callbacks** is a map of default callbacks. This fill allow you to easy change only few callbacks and reuse default for the rest.
Please look at source of *http.async.client/stream-seq* to see how to do it.