Skip to content
This repository

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

An experimental non-blocking HTTP client with throttling and retrying, based on AsyncHttpClient.

branch: master

Fetching latest commit…

Octocat-spinner-32-eaf2f5

Cannot retrieve the latest commit at this time

Octocat-spinner-32 project
Octocat-spinner-32 src
Octocat-spinner-32 .gitignore
Octocat-spinner-32 README.md
README.md

A simple asynchronous HTTP client for Scala

TGV is an experimental HTTP client for Scala that offers:

  • configurable retrying in case of errors;
  • strict throttling of requests, guaranteeing that the server receiving the requests will not see more than n concurrent requests per t seconds; and
  • custom transformations.

TGV is based on Ning's Async HTTP Client (currently version 1.8-SNAPSHOT).

TGV is experimental code, a playground to explore non-blocking request/response IO in Scala. Currently, TGV reads the whole response from a HTTP request into memory before returning it. To make this more (resource-) efficient, there are plans to introduce a streaming API. For this, I am currently looking into Iteratees and similar techniques, like Pipes, Machines, or Conduits.

Even though TGV does come with unit tests, not all edge cases have been tested well enough yet. Use TGV in production with care.

Example, please!

Here is a short sample applications that downloads the body of some Wikipedia pages using throttling:

object SampleApp extends App {

  // Create the underlying HTTP client
  // (Note: currently, only Ning's AsyncHttpClient is supported.)
  val httpClient = new AsyncHttpClient(new Builder().
    setAllowPoolingConnection(true).
    setFollowRedirects(true).
    setCompressionEnabled(true).
    build)

  // Create threads to handle the request processing; all processing will be done
  // asynchronously, so the number of requests we can handle is not limited by the number
  // of threads configured here
  val numberOfThreads = Runtime.getRuntime().availableProcessors() * 2
  val executionService = new ExecutionContextService(numberOfThreads).start()
  implicit val executionContext = executionService.context()

  // Create a basic client (no retrying, no throttling, no transforms)
  val transport = AsyncHttpTransport(new StreamingAsyncHttpClient(httpClient))

  // Add throttling on top
  val throttled = transport.withThrottling(3 per (1 second))

  // Add retrying on top
  // Note: any requests on `retrying` will be throttled and, if needed, retried
  val retrying = throttled.withRetryStrategy(backoffStrategy = exponentialBackoffStrategy(maxRetries = 10))

  // Execute requests in parallel
  val words = List("Zurich", "Basel", "Paris", "London", "Washington", "Rome", "Palermo", "Nice", "Barcelona", "Vienna", "Berlin")
  val futures = words.map(w => retrying.body(retrying.getBuilder("http://en.wikipedia.org/wiki/" + w).build))

  // Wait until all responses have come in
  result(sequence(futures))(30 seconds)
  println("Done.")

  executionService.stop()
  httpClient.close()
  println("Shut down.")
}

Deploy to your own Maven repository

If you want sbt publish to deploy the library jar, its sources and API documentation to deploy to your in-house Maven repository, follow these steps:

  1. Create, inside the project directory (and thus next to this README.md file) a file called local.sbt with content:

    publishTo := Some("My Repository" at "http://repo.my.com/some/path/")
    
    credentials += Credentials(Path.userHome / ".ivy2" / ".my-credentials")
    

    (Adapt the URL and file name .my-credentials.)

  2. Create a file ~/.ivy2/.my-credentials and store the login and password to your repository in there. For a Sonatype Nexus repository, this should look something like this:

    realm=Sonatype Nexus Repository Manager
    host=repo.my.com
    user=myname
    password=mypassword
    

After this, you can do a sbt publish to deploy the library.

Something went wrong with that request. Please try again.