updated benchmarks for Marshal.load/dump to fix multiple iteration pr…

…oblems

added C documentation for the JSON compatibility API functions
bumped the internal version, updated changelog, readme and gemspec for 0.5.5 release

CHANGELOG.md

@@ -1,5 +1,14 @@
 # Changelog
 
+## 0.5.5 (June 17th, 2009)
+* Introduction of the JSON gem API compatibility layer
+** NOTE: this isn't a 1:1 compatibility API, the goal was to be compatible with as many of the projects using the JSON gem as possible - not the JSON gem API itself
+** the compatibility API must be explicitly enabled by requiring 'yajl/json_gem' in your project
+** JSON.parse, JSON.generate, and the #to_json instance method extension to ruby's primitive classes are all included
+* Fix Yajl::Encoder to ensure map keys are strings
+* Encoding multiple JSON objects to a single stream doesn't separate by a newline character anymore
+* Yajl::Encoder now checks for the existence of, and will call #to_json on any non-primitive object
+
 ## 0.5.4 (June 16th, 2009)
 * Yajl::Parser's :symbolize_keys option now defaults to false
 * remove use of sprintf for a little speed improvement while parsing

README.rdoc

@@ -4,6 +4,19 @@ This gem (although not in gem form just yet) is a C binding to the excellent YAJ
 
 You can read more info at the projects website http://lloydforge.org/projects/yajl or check out it's codes at http://github.com/lloyd/yajl.
 
+== Features
+
+* JSON parsing and encoding directly to and from an IO stream (file, socket, etc) or String. Compressed stream parsing and encoding supported for Bzip2, Gzip and Deflate.
+* Parse and encode *multiple* JSON objects to and from streams or strings continuously.
+* JSON gem compatibility API - allows yajl-ruby to be used as a drop-in replacement for the JSON gem
+* Basic HTTP client (only GET requests supported for now) which parses JSON directly off the response body *as it's being received*
+* ~3.5x faster than JSON.generate
+* ~1.9x faster than JSON.parse
+* ~4.5x faster than YAML.load
+* ~377.5x faster than YAML.dump
+* ~1.5x faster than Marshal.load
+* ~2x faster than Marshal.dump
+
 == How to install
 
 Install it like any other gem hosted at the Githubs like so:
@@ -114,7 +127,7 @@ This allows you to encode JSON as a stream, writing directly to a socket
  socket = TCPSocket.new(192.168.1.101, 9000)
  hash = {:foo => 12425125, :bar => "some string", ... }
  encoder = Yajl::Encoder.new
- Yajl::encoder.encode(hash, socket)
+ Yajl::Encoder.encode(hash, socket)
 
 Or what if you wanted to compress the stream over the wire?
 
@@ -135,7 +148,24 @@ This example will encode and send 50 JSON objects over the same stream, continuo
 
 You can also use Yajl::Bzip2::StreamWriter and Yajl::Deflate::StreamWriter. So you can pick whichever fits your CPU/bandwidth sweet-spot.
 
-There are a lot more possibilities, some of which I'm going to write other gems/plugins for.
+== JSON gem Compatibility API
+
+The JSON gem compatibility API isn't enabled by default. You have to explicitly require it like so:
+
+ require 'yajl/json_gem'
+
+That's right, you can just replace "require 'json'" with the line above and you're done!
+
+This will require yajl-ruby itself, as well as enable it's JSON gem compatibility API.
+
+This includes the following API:
+
+ JSON.parse, JSON.generate, JSON.pretty_generate, JSON.load, JSON.dump
+ and all of the #to_json instance method overrides for Ruby's primitive objects
+
+Once the compatibility API is enabled, you're existing or new project should work as if the JSON gem itself were being used. Only you'll be using Yajl ;)
+
+There are a lot more possibilities that I'd love to see other gems/plugins for someday.
 
 Some ideas are:
 * parsing logs in JSON format
@@ -192,14 +222,30 @@ NOTE: I converted the 2.4MB JSON file to YAML for this test.
 
 * Yajl::Parser#parse: 4.33s
 * JSON.parse: 5.37s
-* YAML.load_stream: 19.47s
+* YAML.load: 19.47s
 
 ==== Encode Time (to their respective formats)
 
 * Yajl::Encoder#encode: 3.47s
 * JSON#to_json: 6.6s
 * YAML.dump(obj, io): 1309.93s
 
+=== Compared to Marshal.load/Marshal.dump
+
+NOTE: I converted the 2.4MB JSON file to a Hash and a dump file from Marshal.dump for this test.
+
+==== Parse Time (from their respective formats)
+
+* Yajl::Parser#parse: 4.54s
+* JSON.parse: 7.40s
+* Marshal.load: 7s
+
+==== Encode Time (to their respective formats)
+
+* Yajl::Encoder#encode: 2.39s
+* JSON#to_json: 8.37s
+* Marshal.dump: 4.66s
+
 == Third Party Sources Bundled
 
 This project includes code from the BSD licensed yajl project, copyright 2007-2009 Lloyd Hilaiel

benchmark/encode_json_and_marshal.rb

@@ -5,12 +5,12 @@
 require 'stringio'
 require 'json'
 
-filename = ARGV[0] || 'benchmark/subjects/contacts.json'
+times = ARGV[0] ? ARGV[0].to_i : 1
+filename = 'benchmark/subjects/contacts.json'
 json = File.new(filename, 'r')
 hash = Yajl::Parser.new.parse(json)
 json.close
 
-times = ARGV[1] ? ARGV[1].to_i : 1
 puts "Starting benchmark encoding #{filename} #{times} times\n\n"
 Benchmark.bm { |x|
   encoder = Yajl::Encoder.new

benchmark/parse_json_and_marshal.rb

@@ -21,12 +21,11 @@
 times = ARGV[0] ? ARGV[0].to_i : 1
 puts "Starting benchmark parsing #{File.size(filename)} bytes of JSON data #{times} times\n\n"
 Benchmark.bm { |x|
-  parser = Yajl::Parser.new
   x.report {
     puts "Yajl::Parser#parse"
     times.times {
       json.rewind
-      hash = parser.parse(json)
+      hash = Yajl::Parser.new.parse(json)
     }
   }
   x.report {
@@ -39,6 +38,7 @@
   x.report {
     puts "Marshal.load"
     times.times {
+      marshal_file.rewind
       Marshal.load(marshal_file)
     }
   }

ext/yajl_ext.c

@@ -523,6 +523,19 @@ static VALUE rb_yajl_encoder_encode(int argc, VALUE * argv, VALUE self) {
 
 
 // JSON Gem compatibility
+
+/*
+ * Document-class: Object
+ */
+/*
+ * Document-method: to_json
+ *
+ * call-seq: to_json(encoder=Yajl::Encoder.new)
+ *
+ * +encoder+ is an existing Yajl::Encoder used to encode JSON
+ *
+ * Encodes an instance of Object to JSON
+ */
 static VALUE rb_yajl_json_ext_object_to_json(int argc, VALUE * argv, VALUE self) {
     VALUE rb_encoder;
     rb_scan_args(argc, argv, "01", &rb_encoder);
@@ -532,6 +545,18 @@ static VALUE rb_yajl_json_ext_object_to_json(int argc, VALUE * argv, VALUE self)
     return rb_yajl_encoder_encode(1, &self, rb_encoder);
 }
 
+/*
+ * Document-class: Hash
+ */
+/*
+ * Document-method: to_json
+ *
+ * call-seq: to_json(encoder=Yajl::Encoder.new)
+ *
+ * +encoder+ is an existing Yajl::Encoder used to encode JSON
+ *
+ * Encodes an instance of Hash to JSON
+ */
 static VALUE rb_yajl_json_ext_hash_to_json(int argc, VALUE * argv, VALUE self) {
     VALUE rb_encoder;
     rb_scan_args(argc, argv, "01", &rb_encoder);
@@ -541,6 +566,18 @@ static VALUE rb_yajl_json_ext_hash_to_json(int argc, VALUE * argv, VALUE self) {
     return rb_yajl_encoder_encode(1, &self, rb_encoder);
 }
 
+/*
+ * Document-class: Array
+ */
+/*
+ * Document-method: to_json
+ *
+ * call-seq: to_json(encoder=Yajl::Encoder.new)
+ *
+ * +encoder+ is an existing Yajl::Encoder used to encode JSON
+ *
+ * Encodes an instance of Array to JSON
+ */
 static VALUE rb_yajl_json_ext_array_to_json(int argc, VALUE * argv, VALUE self) {
     VALUE rb_encoder;
     rb_scan_args(argc, argv, "01", &rb_encoder);
@@ -550,6 +587,18 @@ static VALUE rb_yajl_json_ext_array_to_json(int argc, VALUE * argv, VALUE self)
     return rb_yajl_encoder_encode(1, &self, rb_encoder);
 }
 
+/*
+ * Document-class: Fixnum
+ */
+/*
+ * Document-method: to_json
+ *
+ * call-seq: to_json(encoder=Yajl::Encoder.new)
+ *
+ * +encoder+ is an existing Yajl::Encoder used to encode JSON
+ *
+ * Encodes an instance of Fixnum to JSON
+ */
 static VALUE rb_yajl_json_ext_fixnum_to_json(int argc, VALUE * argv, VALUE self) {
     VALUE rb_encoder;
     rb_scan_args(argc, argv, "01", &rb_encoder);
@@ -559,6 +608,18 @@ static VALUE rb_yajl_json_ext_fixnum_to_json(int argc, VALUE * argv, VALUE self)
     return rb_yajl_encoder_encode(1, &self, rb_encoder);
 }
 
+/*
+ * Document-class: Float
+ */
+/*
+ * Document-method: to_json
+ *
+ * call-seq: to_json(encoder=Yajl::Encoder.new)
+ *
+ * +encoder+ is an existing Yajl::Encoder used to encode JSON
+ *
+ * Encodes an instance of Float to JSON
+ */
 static VALUE rb_yajl_json_ext_float_to_json(int argc, VALUE * argv, VALUE self) {
     VALUE rb_encoder;
     rb_scan_args(argc, argv, "01", &rb_encoder);
@@ -568,6 +629,18 @@ static VALUE rb_yajl_json_ext_float_to_json(int argc, VALUE * argv, VALUE self)
     return rb_yajl_encoder_encode(1, &self, rb_encoder);
 }
 
+/*
+ * Document-class: String
+ */
+/*
+ * Document-method: to_json
+ *
+ * call-seq: to_json(encoder=Yajl::Encoder.new)
+ *
+ * +encoder+ is an existing Yajl::Encoder used to encode JSON
+ *
+ * Encodes an instance of TrueClass to JSON
+ */
 static VALUE rb_yajl_json_ext_string_to_json(int argc, VALUE * argv, VALUE self) {
     VALUE rb_encoder;
     rb_scan_args(argc, argv, "01", &rb_encoder);
@@ -577,6 +650,18 @@ static VALUE rb_yajl_json_ext_string_to_json(int argc, VALUE * argv, VALUE self)
     return rb_yajl_encoder_encode(1, &self, rb_encoder);
 }
 
+/*
+ * Document-class: TrueClass
+ */
+/*
+ * Document-method: to_json
+ *
+ * call-seq: to_json(encoder=Yajl::Encoder.new)
+ *
+ * +encoder+ is an existing Yajl::Encoder used to encode JSON
+ *
+ * Encodes an instance of TrueClass to JSON
+ */
 static VALUE rb_yajl_json_ext_true_to_json(int argc, VALUE * argv, VALUE self) {
     VALUE rb_encoder;
     rb_scan_args(argc, argv, "01", &rb_encoder);
@@ -586,6 +671,18 @@ static VALUE rb_yajl_json_ext_true_to_json(int argc, VALUE * argv, VALUE self) {
     return rb_yajl_encoder_encode(1, &self, rb_encoder);
 }
 
+/*
+ * Document-class: FalseClass
+ */
+/*
+ * Document-method: to_json
+ *
+ * call-seq: to_json(encoder=Yajl::Encoder.new)
+ *
+ * +encoder+ is an existing Yajl::Encoder used to encode JSON
+ *
+ * Encodes an instance of FalseClass to JSON
+ */
 static VALUE rb_yajl_json_ext_false_to_json(int argc, VALUE * argv, VALUE self) {
     VALUE rb_encoder;
     rb_scan_args(argc, argv, "01", &rb_encoder);
@@ -595,6 +692,18 @@ static VALUE rb_yajl_json_ext_false_to_json(int argc, VALUE * argv, VALUE self)
     return rb_yajl_encoder_encode(1, &self, rb_encoder);
 }
 
+/*
+ * Document-class: NilClass
+ */
+/*
+ * Document-method: to_json
+ *
+ * call-seq: to_json(encoder=Yajl::Encoder.new)
+ *
+ * +encoder+ is an existing Yajl::Encoder used to encode JSON
+ *
+ * Encodes an instance of NilClass to JSON
+ */
 static VALUE rb_yajl_json_ext_nil_to_json(int argc, VALUE * argv, VALUE self) {
     VALUE rb_encoder;
     rb_scan_args(argc, argv, "01", &rb_encoder);
@@ -604,6 +713,16 @@ static VALUE rb_yajl_json_ext_nil_to_json(int argc, VALUE * argv, VALUE self) {
     return rb_yajl_encoder_encode(1, &self, rb_encoder);
 }
 
+/*
+ * Document-class: Yajl::Encoder
+ */
+/*
+ * Document-method: enable_json_gem_compatability
+ *
+ * call-seq: enable_json_gem_compatability
+ *
+ * Enables the JSON gem compatibility API
+ */
 static VALUE rb_yajl_encoder_enable_json_gem_ext(VALUE klass) {
     rb_define_method(rb_cObject, "to_json", rb_yajl_json_ext_object_to_json, -1);
     rb_define_method(rb_cHash, "to_json", rb_yajl_json_ext_hash_to_json, -1);

lib/yajl.rb

@@ -13,7 +13,7 @@
 #
 # Ruby bindings to the excellent Yajl (Yet Another JSON Parser) ANSI C library.
 module Yajl
-  VERSION = "0.5.4"
+  VERSION = "0.5.5"
 
   class Parser
     # A helper method for parse-and-forget use-cases

yajl-ruby.gemspec

@@ -2,11 +2,11 @@
 
 Gem::Specification.new do |s|
   s.name = %q{yajl-ruby}
-  s.version = "0.5.4"
+  s.version = "0.5.5"
 
   s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
   s.authors = ["Brian Lopez", "Lloyd Hilaiel"]
-  s.date = %q{2009-06-16}
+  s.date = %q{2009-06-18}
   s.email = %q{seniorlopez@gmail.com}
   s.extensions = ["ext/extconf.rb"]
   s.extra_rdoc_files = [
@@ -28,6 +28,7 @@ Gem::Specification.new do |s|
      "benchmark/parse_json_and_yaml.rb",
      "benchmark/parse_stream.rb",
      "benchmark/subjects/contacts.json",
+     "benchmark/subjects/contacts.marshal_dump",
      "benchmark/subjects/contacts.yml",
      "benchmark/subjects/item.json",
      "benchmark/subjects/ohai.json",
@@ -69,12 +70,14 @@ Gem::Specification.new do |s|
      "lib/yajl/gzip/stream_reader.rb",
      "lib/yajl/gzip/stream_writer.rb",
      "lib/yajl/http_stream.rb",
+     "lib/yajl/json_gem.rb",
      "spec/encoding/encoding_spec.rb",
      "spec/http/fixtures/http.bzip2.dump",
      "spec/http/fixtures/http.deflate.dump",
      "spec/http/fixtures/http.gzip.dump",
      "spec/http/fixtures/http.raw.dump",
      "spec/http/http_spec.rb",
+     "spec/json_gem_compatibility/compatibility_spec.rb",
      "spec/parsing/active_support_spec.rb",
      "spec/parsing/chunked_spec.rb",
      "spec/parsing/fixtures/fail.15.json",
@@ -147,6 +150,7 @@ Gem::Specification.new do |s|
   s.test_files = [
     "spec/encoding/encoding_spec.rb",
      "spec/http/http_spec.rb",
+     "spec/json_gem_compatibility/compatibility_spec.rb",
      "spec/parsing/active_support_spec.rb",
      "spec/parsing/chunked_spec.rb",
      "spec/parsing/fixtures_spec.rb",