Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
A streaming JSON parsing and encoding library for Ruby (C bindings to yajl)

Fetching latest commit…

Cannot retrieve the latest commit at this time

Failed to load latest commit information.
benchmark
ext
lib
spec
.gitignore
CHANGELOG.rdoc
MIT-LICENSE
README.rdoc
Rakefile
VERSION.yml
yajl-ruby.gemspec

README.rdoc

YAJL C Bindings for Ruby

This gem (although not in gem form just yet) is a C binding to the excellent YAJL JSON parsing and generation library.

You can read more info at the projects website lloydforge.org/projects/yajl or check out it's codes at github.com/lloyd/yajl.

Example of use

First, you're probably gonna want to include it:

include 'yajl'

Parsing

Then maybe parse some JSON from:

a File IO

json_contents = File.new('test.json', 'r')
hash = Yajl::Stream.parse(json)

or maybe a StringIO

json_contents = StringIO.new
hash = Yajl::Stream.parse(json)

or maybe STDIN

cat someJsonFile.json | ruby -ryajl -e "puts Yajl::Stream.parse(STDIN).inspect"

Or lets say you didn't have access to the IO object that contained JSON data, but instead only had access to chunks of it at a time. No problem!

(Assume we're in an EventMachine::Connection instance)

def object_parsed(obj)
   puts "Sometimes one pays most for the things one gets for nothing. - Albert Einstein"
   puts obj.inspect
 end

def connection_completed
  # once a full JSON object has been parsed from the stream
  # object_parsed will be called, and passed the constructed object
  Yajl::Chunked.on_parse_complete = method(:object_parsed)
end

def receive_data(data)
  # continue passing chunks
  Yajl::Chunked.parse_some(data)

  # Or as an alias, you could have done:
  # Yajl::Chunked << data
end

Or how about a JSON API HTTP request? This actually makes a request using a raw TCPSocket, then parses the JSON body right off the socket. While it's being received over the wire!

require 'uri'
require 'yajl/http_stream'

url = URI.parse("http://search.twitter.com/search.json?q=engineyard")
results = Yajl::HttpStream.get(url)

Or do the same request, with Gzip and Deflate output compression support (also supports Bzip2, if loaded): (this does the same raw socket Request, but transparently parses the compressed response body)

require 'uri'
require 'yajl/gzip'
require 'yajl/deflate'
require 'yajl/http_stream'

url = URI.parse("http://search.twitter.com/search.json?q=engineyard")
results = Yajl::HttpStream.get(url)

Or how about parsing directly from a compressed file?

require 'yajl/bzip2'

file = File.new('some.json.bz2', 'r')
result = Yajl::Bzip2::StreamReader.parse(file)

Encoding

Since yajl-ruby does everything using streams, you simply need to pass the object to encode, and the IO to write the stream to (this happens in chunks).

This allows you to encode JSON as a stream, writing directly to a socket

socket = TCPSocket.new(192.168.1.101, 9000)
hash = {:foo => 12425125, :bar => "some string", ... }
Yajl::Stream.encode(hash, socket)

Or what if you wanted to compress the stream over the wire?

require 'yajl/gzip'
socket = TCPSocket.new(192.168.1.101, 9000)
hash = {:foo => 12425125, :bar => "some string", ... }
Yajl::Gzip::StreamWriter.encode(hash, socket)

You can also use Yajl::Bzip2::StreamWriter and Yajl::Deflate::StreamWriter. So you can pick whichever fits your CPU/bandwidth sweet-spot.

There are a lot more possibilities, some of which I'm going to write other gems/plugins for.

Some ideas are:

How to install

First, Yajl uses CMake to build itself (yes, the author realizes this isn't the norm for open source and is willing and ready to accept patches, fork away kids!) so you'll need to grab it first from www.cmake.org.

After you've got that, grab the latest version (I suggest at least 1.0.4 as it contains fixes for Unicode parsing) of Yajl itself from the Githubs at github.com/lloyd/yajl.

After you have that installed, you should be able to install it like any other gem hosted here like so:

(more instructions here: gems.github.com)

sudo gem install brianmario-yajl-ruby

Benchmarks

After I finished implementation - this library performs close to the same as the current JSON.parse (C gem) does on small/medium files.

But on larger files, and higher amounts of iteration, this library was around 2x faster than JSON.parse.

The main benefit of this library is in it's memory usage. Since it's able to parse the stream in chunks, it's memory requirements are very, very low.

Here's what parsing a 2.43MB JSON file off the filesystem 20 times looks like:

Memory Usage

Average

  • Yajl::Stream.parse: 32MB

  • JSON.parse: 54MB

  • ActiveSupport::JSON.decode: 63MB

Peak

  • Yajl::Stream.parse: 32MB

  • JSON.parse: 57MB

  • ActiveSupport::JSON.decode: 67MB

Parse Time

  • Yajl::Stream.parse: 4.54s

  • JSON.parse: 5.47s

  • ActiveSupport::JSON.decode: 64.42s

Encode Time

  • Yajl::Stream.encode: 3.59s

  • JSON#to_json: 6.2s

  • ActiveSupport::JSON.encode: 45.58s

Compared to YAML

NOTE: I converted the 2.4MB JSON file to YAML for this test.

Parse Time (from their respective formats)

  • Yajl::Stream.parse: 4.33s

  • JSON.parse: 5.37s

  • YAML.load_stream: 19.47s

Encode Time (to their respective formats)

  • Yajl::Stream.encode: 3.47s

  • JSON#to_json: 6.6s

  • YAML.dump(obj, io): 1309.93s

Special Thanks

I've had a lot of inspiration, and a lot of help. Thanks to everyone who's been a part of this and those to come!

  • Lloyd Hilaiel (github.com/lloyd) - for writing Yajl!!

  • Josh Ferguson (github.com/besquared) - for peer-pressuring me into getting back into C; it worked ;) Also tons of support over IM

  • Jonathan Novak (github.com/cypriss) - pointer-hacking help

  • Tom Smith (github.com/rtomsmith) - pointer-hacking help

  • Rick (github.com/technoweenie) - for making an ActiveSupport patch with support for this library and teasing me that it might go into Rails 3. You sure lit a fire under my ass and I got a ton of work done because of it! :)

  • The entire Github Crew - my inspiration, time spent writing this, finding Yajl, So many-MANY other things wouldn't have been possible without this awesome service. I owe you guys some whiskey at Kilowatt.

Something went wrong with that request. Please try again.