Easy Google App Engine’s URLFetch service jruby library

Overview

Easy access to Google’s App Engine URLFetch Service. You need something like this if you want to use NET:HTTP things, cause NET:HTTP is NOT supported by GAE.

  response = URLFetch.get(url)
  response = URLFetch.head(url)
  response = URLFetch.post(url,payload)
  response = URLFetch.put(url,payload) #Doesn't seem to work
  response = URLFetch.delete(url) #Doesn't seem to work

  response.to_s #or response.content
  => Response body

  response.code
  => 200  #or 404 or whatever

  response.header
  => [['Set-Cookie','ses=13213'],['Set-Cookie','ses2=53222'],etc...]

  response.header_hash
  => {'Set-Cookie' => 'ses2=53222' }
  #Doesn't support double headers
  #That is infact a problem plaging many http and rest libs.
  #Cause sometimes a site gives two Set-Cookie headers

You can also provide your own headers for a request.

  URLFetch.get(url,:header => {'Cookie' => 'ses=12143' })

Options

You can specify google’s FetchOptions: allowTruncate and allowRedirect

  URLFetch.get(url,:redirect => false, :truncate => false)
  #Default is :redirect => true and :truncate => true

The truncate option means truncate your response if it is bigger then 1 megabyte. If you have truncate set to false and your response is bigger then 1 megabyte it will throw an error.

Setting redirect to false means that URLFetchService will no longer automaticly redirect when an 3xx response code returns. This automatic redirecting is limited to 5 redirects so if you have a page that redirects more then 5 times you can turn redirecting off and do it yourself.

Handle Errors

URLFetch throws a few kinds of errors

    java.net.MalformedURLException - If the provided URL is malformed.
    java.io.IOException - If the remote service could not be contacted or the URL could not be fetched.
    ResponseTooLargeException - If :truncate => false on request and the response is too large. Some responses are too large to even retrieve from the remote server, and in these cases the exception is thrown even if response truncation is enabled.

The most common error you will have to capture is java.io.IOException. Here is a quick example:

    begin
      return URLFetch.get("http://"+params[:site]).to_s
    rescue java.io.IOException
      return "The remote service could not be contacted or the URL could not be fetched."
    end

Limitations

Due to limitations set by google not everything is possible. code.google.com/intl/nl/appengine/docs/java/urlfetch/

• http:// is always port 80
• https:// is always port 443
• Only 5 automatic redirects, You can however turn it off and do it yourself. (:redirect => false)
• Max request size: 1 megabyte
• Max response size: 1 megabyte

For security reasons, the following headers cannot be modified by the application:

• Content-Length
• Host
• Referer
• User-Agent
• Vary
• Via
• X-Forwarded-For

Todo

• Providing a drop in replacement for NET:HTTP
• Maybe some built-in cookie support

Meta

Made by: Bas Wilbers

Released under the MIT License: www.opensource.org/licenses/mit-license.php