💎HTTP Requests Client with a chainable REST API, built-in sessions and middlewares.
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
.circleci
benchmarks
spec fix(option): parse json data with sub hash/array/namedtuple etc (#41) Oct 15, 2018
src release: dump v0.7.3 Oct 18, 2018
.editorconfig
.gitignore
CHANGELOG.md
CODE_OF_CONDUCT.md
LICENSE
README.md
halite-logo-small.png
halite-logo.png
shard.yml

README.md

halite-logo

Halite

Language Tag Source Document Build Status

HTTP Requests with a chainable REST API, built-in sessions and middlewares written by Crystal. Inspired from the awesome Ruby's HTTP/RESTClient and Python's requests.

Build in Crystal version >= v0.25.0, this document valid with latest commit.

Index

Installation

Add this to your application's shard.yml:

dependencies:
  halite:
    github: icyleaf/halite

Usage

require "halite"

Making Requests

Make a GET request:

# Direct get url
Halite.get("http://httpbin.org/get")

# Support NamedTuple as query params
Halite.get("http://httpbin.org/get", params: {
  language: "crystal",
  shard: "halite"
})

# Also support Array as query params
Halite.get("http://httpbin.org/get", headers: {
    "Private-Token" => "T0k3n"
  }, params: {
    "language" => "crystal",
    "shard" => "halite"
  })

# And support chainable
Halite.header(private_token: "T0k3n")
      .get("http://httpbin.org/get", params: {
        "language" => "crystal",
        "shard" => "halite"
      })

See also all chainable methods.

Many other HTTP methods are avaiabled as well:

  • get
  • head
  • post
  • put
  • delete
  • patch
  • options

Passing Parameters

Query string parameters

Use the params argument to add query string parameters to requests:

Halite.get("http://httpbin.org/get", params: { "firstname" => "Olen", "lastname" => "Rosenbaum" })

Form data

Use the form argument to pass data serialized as form encoded:

Halite.post("http://httpbin.org/post", form: { "firstname" => "Olen", "lastname" => "Rosenbaum" })

File uploads (via form data)

To upload files as if form data, construct the form as follows:

Halite.post("http://httpbin.org/post", form: {
  "username" => "Quincy",
  "avatar" => File.open("/Users/icyleaf/quincy_avatar.png")
})

It is possible to upload multiple files:

Halite.post("http://httpbin.org/post", form: {
  photos: [
    File.open("/Users/icyleaf/photo1.png"),
    File.open("/Users/icyleaf/photo2.png")
  ],
  album_name: "samples"
})

Or pass the name with []:

Halite.post("http://httpbin.org/post", form: {
  "photos[]" => [
    File.open("/Users/icyleaf/photo1.png"),
    File.open("/Users/icyleaf/photo2.png")
  ],
  "album_name" => "samples"
})

Multiple files aslo can be uploaded using both ways above, it depend on web server.

JSON data

Use the json argument to pass data serialized as body encoded:

Halite.post("http://httpbin.org/post", json: { "firstname" => "Olen", "lastname" => "Rosenbaum" })

Raw String

Use the raw argument to pass raw string as body and set the Content-Type manually:

# Set content-type to "text/plain" by default
Halite.post("http://httpbin.org/post", raw: "name=Peter+Lee&address=%23123+Happy+Ave&language=C%2B%2B")

# Set content-type manually
Halite.post("http://httpbin.org/post",
  headers: {
    "content-type" => "application/json"
  },
  raw: %Q{{"name":"Peter Lee","address":"23123 Happy Ave","language":"C++"}}
)

Passing advanced options

Headers

Here are two way to passing headers data:

1. Use the #headers method
Halite.headers(private_token: "T0k3n").get("http://httpbin.org/get")

# Also support Hash or NamedTuple
Halite.headers({ "private_token" => "T0k3n" }).get("http://httpbin.org/get")

# Or
Halite.headers({ private_token: "T0k3n" }).get("http://httpbin.org/get")
2. Use the headers argument in availabled request method:
Halite.get("http://httpbin.org/anything" , headers: { private_token: "T0k3n" })

Halite.post("http://httpbin.org/anything" , headers: { private_token: "T0k3n" })

Auth

Use the #basic_auth method to perform HTTP Basic Authentication using a username and password:

Halite.basic_auth(user: "user", password: "p@ss").get("http://httpbin.org/get")

# We can pass a raw authorization header using the auth method:
Halite.auth("Bearer dXNlcjpwQHNz").get("http://httpbin.org/get")

Cookies

Passing cookies in requests

The Halite.cookies option can be used to configure cookies for a given request:

Halite.cookies(session_cookie: "6abaef100b77808ceb7fe26a3bcff1d0")
      .get("http://httpbin.org/headers")
Get cookies in requests

To obtain the cookies(cookie jar) for a given response, call the #cookies method:

r = Halite.get("http://httpbin.org/cookies?set?session_cookie=6abaef100b77808ceb7fe26a3bcff1d0")
pp r.cookies
# => #<HTTP::Cookies:0x10dbed980 @cookies={"session_cookie" =>#<HTTP::Cookie:0x10ec20f00 @domain=nil, @expires=nil, @extension=nil, @http_only=false, @name="session_cookie", @path="/", @secure=false, @value="6abaef100b77808ceb7fe26a3bcff1d0">}>

Redirects and History

Automatically following redirects

The Halite.follow method can be used for automatically following redirects(Max up to 5 times):

# Set the cookie and redirect to http://httpbin.org/cookies
Halite.follow
      .get("http://httpbin.org/cookies/set/name/foo")
Limiting number of redirects

As above, set over 5 times, it will raise a Halite::TooManyRedirectsError, but you can change less if you can:

Halite.follow(2)
      .get("http://httpbin.org/relative-redirect/5")
Disabling unsafe redirects

It only redirects with GET, HEAD request and returns a 300, 301, 302 by default, otherwise it will raise a Halite::StateError. We can diasble it to set :strict to false if we want any method(verb) requests, in which case the GET method(verb) will be used for that redirect:

Halite.follow(strict: false)
      .post("http://httpbin.org/relative-redirect/5")
History

Response#history property list contains the Response objects that were created in order to complete the request. The list is orderd from the orderst to the most recent response.

r = Halite.follow
          .get("http://httpbin.org/redirect/3")

r.uri
# => http://httpbin.org/get

r.status_code
# => 200

r.history
# => [
#      #<Halite::Response HTTP/1.1 302 FOUND {"Location" => "/relative-redirect/2" ...>,
#      #<Halite::Response HTTP/1.1 302 FOUND {"Location" => "/relative-redirect/1" ...>,
#      #<Halite::Response HTTP/1.1 302 FOUND {"Location" => "/get" ...>,
#      #<Halite::Response HTTP/1.1 200 OK    {"Content-Type" => "application/json" ...>
#    ]

NOTE: It contains the Response object if you use history and HTTP was not a 30x, For example:

r = Halite.get("http://httpbin.org/get")
r.history.size # => 0

r = Halite.follow
          .get("http://httpbin.org/get")
r.history.size # => 1

Timeout

By default, the Halite does not enforce timeout on a request. We can enable per operation timeouts by configuring them through the chaining API.

The connect timeout is the number of seconds Halite will wait for our client to establish a connection to a remote server call on the socket.

Once our client has connected to the server and sent the HTTP request, the read timeout is the number of seconds the client will wait for the server to send a response.

# Separate set connect and read timeout
Halite.timeout(connect: 3.0, read: 2.minutes)
      .get("http://httpbin.org/anything")

# Boath set connect and read timeout
# The timeout value will be applied to both the connect and the read timeouts.
Halite.timeout(5)
      .get("http://httpbin.org/anything")

HTTPS

The Halite supports HTTPS via Crystal's built-in OpenSSL module. All you have to do in order to use HTTPS is pass in an https://-prefixed URL.

To use client certificates, you can pass in a custom OpenSSL::SSL::Context::Client object containing the certificates you wish to use:

ssl = OpenSSL::SSL::Context::Client.new
ssl.ca_certificates = File.expand_path("~/client.crt")
ssl.private_key = File.expand_path("~/client.key")

Halite.get("https://httpbin.org/anything", ssl: ssl)

Response Handling

After an HTTP request, Halite::Response object have several useful methods. (Also see the API documentation).

  • #body: The response body.
  • #body_io: The response body io.
  • #status_code: The HTTP status code.
  • #content_type: The content type of the response.
  • #content_length: The content length of the response.
  • #cookies: A HTTP::Cookies set by server.
  • #headers: A HTTP::Headers of the response.
  • #links: A list of Halite::HeaderLink set from headers.
  • #parse: (return value depends on MIME type) parse the body using a parser defined for the #content_type.
  • #to_a: Return a Hash of status code, response headers and body as a string.
  • #to_raw: Return a raw of response as a string.
  • #to_s: Return response body as a string.
  • #version: The HTTP version.

Response Content

We can read the content of the server's response by call #body:

r = Halite.get("http://httpbin.org/user-agent")
r.body
# => {"user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36"}

The gzip and deflate transfer-encodings are automatically decoded for you. And requests will automatically decode content from the server. Most unicode charsets are seamlessly decoded.

JSON Content

There’s also a built-in a JSON adapter, in case you’re dealing with JSON data:

r = Halite.get("http://httpbin.org/user-agent")
r.parse("json")
r.parse # simplily by default
# => {
# =>   "user-agent" => "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36"
# => }

Parsing Content

Halite::Response has a MIME type adapter system that you can use a decoder to parse the content, we can inherit Halite::MimeTypes::Adapter make our adapter:

# Define a MIME type adapter
class YAMLAdapter < Halite::MimeType::Adapter
  def decode(string)
    YAML.parse(string)
  end

  def encode(obj)
    obj.to_yaml
  end
end

# Register to Halite to invoke
Halite::MimeType.register YAMLAdapter.new, "application/x-yaml", "yaml", "yml"

# Test it!
r = Halite.get "https://raw.githubusercontent.com/icyleaf/halite/master/shard.yml"
r.parse("yaml") # or "yml"
# => {"name" => "halite", "version" => "0.4.0", "authors" => ["icyleaf <icyleaf.cn@gmail.com>"], "crystal" => "0.25.0", "license" => "MIT"}

Binary Data

Store binary data(application/octet-stream) to file, you can do this:

r = Halite.get("http://example.com/foo/bar.zip")
filename = r.headers["Content-Disposition"].split("filename=")[1]
File.open(filename, "w") do |f|
  while byte = r.body.read_byte
    f.write_byte byte
  end
end

Error Handling

  • For any status code, a Halite::Response will be returned.
  • If request timeout, a Halite::TimeoutError will be raised.
  • If a request exceeds the configured number of maximum redirections, a Halite::TooManyRedirectsError will raised.
  • If request uri is http and configured ssl context, a Halite::RequestError will raised.
  • If request uri is invalid, a Halite::ConnectionError/Halite::UnsupportedMethodError/Halite::UnsupportedSchemeError will raised.

Raise for status code

If we made a bad request(a 4xx client error or a 5xx server error response), we can raise with Halite::Response.raise_for_status.

But, since our status_code was not 4xx or 5xx, it returns nil when we call it:

urls = [
  "https://httpbin.org/status/404",
  "https://httpbin.org/status/500?foo=bar",
  "https://httpbin.org/status/200",
]

urls.each do |url|
  r = Halite.get url
  begin
    r.raise_for_status
    p r.body
  rescue ex : Halite::ClientError | Halite::ServerError
    p "[#{ex.status_code}] #{ex.status_message} (#{ex.class})"
  end
end

# => "[404] not found error with url: https://httpbin.org/status/404  (Halite::ClientError)"
# => "[500] internal server error error with url: https://httpbin.org/status/500?foo=bar  (Halite::ServerError)"
# => ""

Advanced Usage

Configuring

Halite provides a traditional way to instance client, and you can configure any chainable methods with block:

client = Halite::Client.new do
  # Set basic auth
  basic_auth "username", "password"

  # Enable logging
  logging true

  # Set timeout
  timeout 10.seconds

  # Set user agent
  headers user_agent: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36"
end

# You also can configure in this way
client.accept("application/json")

r = client.get("http://httpbin.org/get")

Sessions

As like requests.Session(), Halite built-in session by default.

Let's persist some cookies across requests:

client = Halite::Client.new
client.get("http://httpbin.org/cookies/set?private_token=6abaef100b77808ceb7fe26a3bcff1d0")
client.get("http://httpbin.org/cookies")
# => 2018-06-25 18:41:05 +08:00 | request | GET    | http://httpbin.org/cookies/set?private_token=6abaef100b77808ceb7fe26a3bcff1d0
# => 2018-06-25 18:41:06 +08:00 | response | 302    | http://httpbin.org/cookies/set?private_token=6abaef100b77808ceb7fe26a3bcff1d0 | text/html
# => <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
# => <title>Redirecting...</title>
# => <h1>Redirecting...</h1>
# => <p>You should be redirected automatically to target URL: <a href="/cookies">/cookies</a>.  If not click the link.
# => 2018-06-25 18:41:06 +08:00 | request | GET    | http://httpbin.org/cookies
# => 2018-06-25 18:41:07 +08:00 | response | 200    | http://httpbin.org/cookies | application/json
# => {"cookies":{"private_token":"6abaef100b77808ceb7fe26a3bcff1d0"}}

All it support with chainable methods in the other examples list in requests.Session.

Logging

Halite does not enable logging on each request and response too. We can enable per operation logging by configuring them through the chaining API.

By default, Halite will logging all outgoing HTTP requests and their responses(without binary stream) to STDOUT on DEBUG level. You can configuring the following options:

  • logger: Instance your Halite::Logger::Abstract, check Use the custom logger.
  • format: Outputing format, built-in common and json, you can write your own.
  • file: Write to file with path, works with format.
  • filemode: Write file mode, works with format, by default is a. (append to bottom, create it if file is not exist)
  • skip_request_body: By default is false.
  • skip_response_body: By default is false.
  • skip_benchmark: Display elapsed time, by default is false.
  • colorize: Enable colorize in terminal, only apply in common format, by default is true.

NOTE: format (file and filemode) and logger are conflict, you can not use both.

Let's try with it:

# Logging json request
Halite.logger
      .get("http://httpbin.org/get", params: {name: "foobar"})

# => 2018-06-25 18:33:14 +08:00 | request  | GET    | http://httpbin.org/get?name=foobar
# => 2018-06-25 18:33:15 +08:00 | response | 200    | http://httpbin.org/get?name=foobar | 381.32ms | application/json
# => {"args":{"name":"foobar"},"headers":{"Accept":"*/*","Accept-Encoding":"gzip, deflate","Connection":"close","Host":"httpbin.org","User-Agent":"Halite/0.3.2"},"origin":"60.206.194.34","url":"http://httpbin.org/get?name=foobar"}

# Logging image request
Halite.logger
      .get("http://httpbin.org/image/png")

# => 2018-06-25 18:34:15 +08:00 | request  | GET    | http://httpbin.org/image/png
# => 2018-06-25 18:34:15 +08:00 | response | 200    | http://httpbin.org/image/png | image/png

# Logging with options
Halite.logger(skip_request_body: true, skip_response_body: true)
      .post("http://httpbin.org/get", form: {image: File.open("halite-logo.png")})

# => 2018-08-28 14:33:19 +08:00 | request  | POST   | http://httpbin.org/post
# => 2018-08-28 14:33:21 +08:00 | response | 200    | http://httpbin.org/post | 1.61s | application/json

JSON-formatted logging

It has JSON formatted for developer friendly logger.

Halite.logger(format: "json")
      .get("http://httpbin.org/get", params: {name: "foobar"})

Write to a log file

# Write plain text to a log file
Halite.logger(file: "logs/halite.log", skip_benchmark: true, colorize: false)
      .get("http://httpbin.org/get", params: {name: "foobar"})

# Write json data to a log file
Halite.logger(format: "json", file: "logs/halite.log")
      .get("http://httpbin.org/get", params: {name: "foobar"})

Use the custom logger

Creating the custom logger by integration Halite::Logger::Abstract abstract class. Here has two methods must be implement: #request and #response.

class CustomLogger < Halite::Logging::Abstract
  def request(request)
    @logger.info "| >> | %s | %s %s" % [request.verb, request.uri, request.body]
  end

  def response(response)
    @logger.info "| << | %s | %s %s" % [response.status_code, response.uri, response.content_type]
  end
end

# Add to adapter list (optional)
Halite::Logging.register "custom", CustomLogger.new

Halite.logger(logger: CustomLogger.new)
      .get("http://httpbin.org/get", params: {name: "foobar"})

# We can also call it use format name if you added it.
Halite.logger(format: "custom")
      .get("http://httpbin.org/get", params: {name: "foobar"})

# => 2017-12-13 16:40:13 +08:00 | >> | GET | http://httpbin.org/get?name=foobar
# => 2017-12-13 16:40:15 +08:00 | << | 200 | http://httpbin.org/get?name=foobar application/json

Middlewares

Halite now has middlewares (a.k.a features) support providing a simple way to plug in intermediate custom logic in your HTTP client and allowing you to monitor outgoing requests, and incoming responses, even use interceptor.

Avaiabled features:

  • logging (Yes, logging is based on feature, cool, aha!)
  • cache (local storage, speed up in development)

Write a simple feature

Let's implement simple middleware that prints each request:

class RequestMonister < Halite::Feature
  @label : String
  def initialize(**options)
    @label = options.fetch(:label, "")
  end

  def request(request) : Halite::Request
    puts @label
    puts request.verb
    puts request.uri
    puts request.body

    request
  end

  Halite.register_feature "request_monster", self
end

Then use it in Halite:

Halite.use("request_monster", label: "testing")
      .post("http://httpbin.org/post", form: {name: "foo"})

# Or configure to client
client = Halite::Client.new do
  use "request_monster", label: "testing"
end

client.post("http://httpbin.org/post", form: {name: "foo"})

# => testing
# => POST
# => http://httpbin.org/post
# => name=foo

Write a interceptor

Halite features has a killer feature is the *interceptor, Use Halite::Feature::Chain to process with two result:

  • next: perform and run next interceptor
  • return: perform and return

So, you can intercept and turn to the following registered features.

class AlwaysNotFound < Halite::Feature
  def intercept(chain)
    response = chain.perform
    response = Halite::Response.new(chain.request.uri, 404, response.body, response.headers)
    chain.next(response)
  end

  Halite.register_feature "404", self
end

class PoweredBy < Halite::Feature
  def intercept(chain)
    if response = chain.response
      response.headers["X-Powered-By"] = "Halite"
      chain.return(response)
    else
      chain
    end
  end

  Halite.register_feature "powered_by", self
end

r = Halite.use("404").use("powered_by").get("http://httpbin.org/user-agent")
r.status_code               # => 404
r.headers["X-Powered-By"]   # => Halite
r.body                      # => {"user-agent":"Halite/0.6.0"}

For more implementation details about the feature layer, see the feature class and examples and specs.

Link Headers

Many HTTP APIs feature Link headers. Github uses these for pagination in their API, for example:

r = Halite.get "https://api.github.com/users/icyleaf/repos?page=1&per_page=2"
r.links
# => {"next" =>
# =>   Halite::HeaderLink(
# =>    @params={},
# =>    @rel="next",
# =>    @target="https://api.github.com/user/17814/repos?page=2&per_page=2"),
# =>  "last" =>
# =>   Halite::HeaderLink(
# =>    @params={},
# =>    @rel="last",
# =>    @target="https://api.github.com/user/17814/repos?page=41&per_page=2")}

r.links["next"]
# => "https://api.github.com/user/17814/repos?page=2&per_page=2"

r.links["next"].params
# => {}

Help and Discussion

You can browse the API documents:

https://icyleaf.github.io/halite/

You can browse the all chainable methods:

https://icyleaf.github.io/halite/Halite/Chainable.html

You can browse the Changelog:

https://github.com/icyleaf/halite/blob/master/CHANGELOG.md

If you have found a bug, please create a issue here:

https://github.com/icyleaf/halite/issues/new

Donate

Halite is a open source, collaboratively funded project. If you run a business and are using Halite in a revenue-generating product, it would make business sense to sponsor Halite development. Individual users are also welcome to make a one time donation if Halite has helped you in your work or personal projects.

You can donate via Paypal.

How to Contribute

Your contributions are always welcome! Please submit a pull request or create an issue to add a new question, bug or feature to the list.

Here is a throughput graph of the repository for the last few weeks:

Throughput Graph

All Contributors are on the wall.

You may also like

  • totem - Load and parse a configuration file or string in JSON, YAML, dotenv formats.
  • markd - Yet another markdown parser built for speed, Compliant to CommonMark specification.
  • poncho - A .env parser/loader improved for performance.
  • popcorn - Easy and Safe casting from one type to another.
  • fast-crystal - 💨 Writing Fast Crystal 😍 -- Collect Common Crystal idioms.

License

MIT License © icyleaf