Permalink
Browse files

Refactored internal API for ease of use, new options, thread-safety a…

…nd much much more to come!

initial work on refactor, pushing so I can continue from home
initial refactor working, but slower - will go back and optimize after the API is complete
new initial encoder API is in, benchmarks, spec tests and library code updated for new API
removing debug compile flags for now, changed parsing benchmark to parsing the larger file so I can start work on my optimization pass
setting a default value for indentString to appease the compiler gods
fix loading the rest of the yajl-ruby lib for spec tests when run locally and compiled extension for testing
updated examples to have 1.9 file encoding magic comment
updated twitter_stream.json with a new capture
fixed a bug in the Chunked transfer encoding parser when a received chunk was smaller than the size requested by IO#read
fixed a typo in the chunked_spec
all tests pass again!
add deprecated methods and STDERR messages about it, bump internal version for upcoming release
second refactor pass, this time focused on optimization
* changed instance variable assignment and referencing to an in-C struct to save on context switching between Ruby and C
finish converting chunked parsing over to newer struct architecture
removed ActiveSupport from stream parsing test, it was complaining about who knows what...
remove yajl_set_context hack, don't need it anymore with the new API
fixed another bug in the chunked http parser
increased read buffer size to 65k
switched Data_Get_Struct calls to an inline macro for a little more DRY code
Version bump to 0.5.0
Yajl::Parser.new and Yajl::Encoder.new now both accept options for the constructed parser/encoder
* Yajl::Parser.new(:allow_comments => true, :check_utf8 => true)
* Yajl::Encoder.new(:pretty => true, :indent => "  ")
reorganizing examples dir so I can add more of other types
moved twitter example into http folder
updated changelog for next release, removed require for socket lib in twitter examples
updating gemspec with new structure
  • Loading branch information...
1 parent cd1ca05 commit 8070e10966e09140ae508b775a74020a1d3a0ab4 @brianmario brianmario committed May 23, 2009
Showing with 936 additions and 715 deletions.
  1. +12 −0 CHANGELOG.rdoc
  2. +2 −2 VERSION.yml
  3. +4 −3 benchmark/encode.rb
  4. +4 −3 benchmark/encode_json_and_marshal.rb
  5. +4 −3 benchmark/encode_json_and_yaml.rb
  6. +5 −4 benchmark/parse.rb
  7. +4 −3 benchmark/parse_json_and_marshal.rb
  8. +4 −3 benchmark/parse_json_and_yaml.rb
  9. +15 −17 benchmark/parse_stream.rb
  10. +430 −427 benchmark/subjects/twitter_stream.json
  11. +15 −0 examples/http/twitter_search_api.rb
  12. +8 −4 examples/{ → http}/twitter_stream_api.rb
  13. +3 −1 examples/{ → parsing}/from_file.rb
  14. +3 −1 examples/{ → parsing}/from_stdin.rb
  15. +3 −1 examples/{ → parsing}/from_string.rb
  16. +3 −0 ext/api/yajl_parse.h
  17. +2 −1 ext/extconf.rb
  18. +5 −0 ext/yajl.c
  19. +235 −125 ext/yajl_ext.c
  20. +49 −37 ext/yajl_ext.h
  21. +7 −0 ext/yajl_lex.c
  22. +2 −0 ext/yajl_lex.h
  23. +3 −1 ext/yajl_parser.c
  24. +18 −17 lib/yajl.rb
  25. +1 −1 lib/yajl/bzip2.rb
  26. +1 −1 lib/yajl/bzip2/stream_reader.rb
  27. +1 −1 lib/yajl/bzip2/stream_writer.rb
  28. +1 −1 lib/yajl/deflate.rb
  29. +1 −1 lib/yajl/deflate/stream_reader.rb
  30. +1 −1 lib/yajl/deflate/stream_writer.rb
  31. +1 −1 lib/yajl/gzip.rb
  32. +1 −1 lib/yajl/gzip/stream_reader.rb
  33. +1 −1 lib/yajl/gzip/stream_writer.rb
  34. +14 −7 lib/yajl/http_stream.rb
  35. +14 −9 spec/encoding/encoding_spec.rb
  36. +1 −5 spec/http/http_spec.rb
  37. +5 −3 spec/parsing/active_support_spec.rb
  38. +25 −21 spec/parsing/chunked_spec.rb
  39. +4 −2 spec/parsing/fixtures_spec.rb
  40. +2 −1 spec/parsing/one_off_spec.rb
  41. +8 −1 spec/spec_helper.rb
  42. +14 −4 yajl-ruby.gemspec
View
@@ -1,5 +1,17 @@
= Changelog
+0.5.0 (?)
+ * Refactored internal API so the caller can specify initialization options for the Parser and Encoder
+ respectively. Two new classes were introduced as a result - Yajl::Parser and Yajl::Encoder.
+ The newly refactored codebase is cleaner, thread-safe and removed all of the hack-code that was trickled
+ around to make things work in the previous implementation. She's much more seaworthy now cap'n!
+ ** Yajl::Parser.new accepts two options, :allow_comments and :check_utf8 which both default to true
+ ** Yajl::Encoder.new accepts two options, :pretty and :indent which default to false and " " respectively
+ * cleaned up a lot of state code, that to my knowledge prevented yajl-ruby from being used in a thread-safe environment.
+ * added deprecated messaging to Yajl::Stream.parse and Yajl::Stream.encode - these will likely go away before 0.6.0
+ * fixed a bug in the chunked http response parser regarding partially received chunks
+ * added a Twitter Search API example showing off the HttpStream API
+
0.4.9 (May 20th, 2009)
* fixed some parser state bugs surfaced by edge cases
* added support for Chunked HTTP response bodies in Yajl::HttpStream
View
@@ -1,4 +1,4 @@
---
-:minor: 4
-:patch: 9
+:minor: 5
+:patch: 0
:major: 0
View
@@ -8,16 +8,17 @@
filename = ARGV[0] || 'benchmark/subjects/contacts.json'
json = File.new(filename, 'r')
-hash = Yajl::Stream.parse(json)
+hash = Yajl::Parser.new.parse(json)
json.close
times = ARGV[1] ? ARGV[1].to_i : 1
puts "Starting benchmark encoding #{filename} #{times} times\n\n"
Benchmark.bm { |x|
+ encoder = Yajl::Encoder.new
x.report {
- puts "Yajl::Stream.encode"
+ puts "Yajl::Encoder#encode"
times.times {
- Yajl::Stream.encode(hash, StringIO.new)
+ encoder.encode(hash, StringIO.new)
}
}
x.report {
@@ -6,16 +6,17 @@
filename = ARGV[0] || 'benchmark/subjects/contacts.json'
json = File.new(filename, 'r')
-hash = Yajl::Stream.parse(json)
+hash = Yajl::Parser.new.parse(json)
json.close
times = ARGV[1] ? ARGV[1].to_i : 1
puts "Starting benchmark encoding #{filename} #{times} times\n\n"
Benchmark.bm { |x|
+ encoder = Yajl::Encoder.new
x.report {
- puts "Yajl::Stream.encode"
+ puts "Yajl::Encoder#encode"
times.times {
- Yajl::Stream.encode(hash, StringIO.new)
+ encoder.encode(hash, StringIO.new)
}
}
x.report {
@@ -9,16 +9,17 @@
# JSON Section
filename = 'benchmark/subjects/contacts.json'
json = File.new(filename, 'r')
-hash = Yajl::Stream.parse(json)
+hash = Yajl::Parser.new.parse(json)
json.close
times = ARGV[0] ? ARGV[0].to_i : 1
puts "Starting benchmark encoding #{filename} into JSON #{times} times\n\n"
Benchmark.bm { |x|
+ encoder = Yajl::Encoder.new
x.report {
- puts "Yajl::Stream.encode"
+ puts "Yajl::Encoder#encode"
times.times {
- Yajl::Stream.encode(hash, StringIO.new)
+ encoder.encode(hash, StringIO.new)
}
}
x.report {
View
@@ -5,7 +5,7 @@
require 'json'
require 'activesupport'
-filename = ARGV[0] || 'benchmark/subjects/twitter_search.json'
+filename = ARGV[0] || 'benchmark/subjects/contacts.json'
json = File.new(filename, 'r')
# warm up the filesystem
@@ -15,15 +15,16 @@
times = ARGV[1] ? ARGV[1].to_i : 1
puts "Starting benchmark parsing #{File.size(filename)} bytes of JSON data #{times} times\n\n"
Benchmark.bm { |x|
+ parser = Yajl::Parser.new
x.report {
- puts "Yajl::Stream.parse"
+ puts "Yajl::Parser#parse"
times.times {
json.rewind
- Yajl::Stream.parse(json)
+ parser.parse(json)
}
}
x.report {
- puts "JSON.parser"
+ puts "JSON.parse"
times.times {
json.rewind
JSON.parse(json.read, :max_nesting => false)
@@ -17,15 +17,16 @@
times = ARGV[0] ? ARGV[0].to_i : 1
puts "Starting benchmark parsing #{File.size(filename)} bytes of JSON data #{times} times\n\n"
Benchmark.bm { |x|
+ parser = Yajl::Parser.new
x.report {
- puts "Yajl::Stream.parse"
+ puts "Yajl::Parser#parse"
times.times {
json.rewind
- hash = Yajl::Stream.parse(json)
+ hash = parser.parse(json)
}
}
x.report {
- puts "JSON.parser"
+ puts "JSON.parse"
times.times {
json.rewind
JSON.parse(json.read, :max_nesting => false)
@@ -16,15 +16,16 @@
times = ARGV[0] ? ARGV[0].to_i : 1
puts "Starting benchmark parsing #{File.size(filename)} bytes of JSON data #{times} times\n\n"
Benchmark.bm { |x|
+ parser = Yajl::Parser.new
x.report {
- puts "Yajl::Stream.parse"
+ puts "Yajl::Parser#parse"
times.times {
json.rewind
- Yajl::Stream.parse(json)
+ parser.parse(json)
}
}
x.report {
- puts "JSON.parser"
+ puts "JSON.parse"
times.times {
json.rewind
JSON.parse(json.read, :max_nesting => false)
View
@@ -3,10 +3,7 @@
require 'benchmark'
require 'yajl_ext'
require 'json'
-require 'activesupport'
-
-puts "\nWARNING: I'm still working on getting the streaming parsing to work correctly"
-puts "The results of this benchmark aren't currently accurate.\n\n"
+# require 'activesupport'
filename = 'benchmark/subjects/twitter_stream.json'
json = File.new(filename, 'r')
@@ -18,33 +15,34 @@
times = ARGV[0] ? ARGV[0].to_i : 1
puts "Starting benchmark parsing JSON stream (#{File.size(filename)} bytes of JSON data) #{times} times\n\n"
Benchmark.bm { |x|
- Yajl::Chunked.on_parse_complete = lambda { |obj|
+ parser = Yajl::Parser.new
+ parser.on_parse_complete = lambda { |obj|
# no-op
}
x.report {
- puts "Yajl::Stream.parse"
+ puts "Yajl::Parser#parse"
times.times {
json.rewind
- Yajl::Stream.parse(json)
+ parser.parse(json)
}
}
x.report {
- puts "JSON.parser"
+ puts "JSON.parse"
times.times {
json.rewind
while chunk = json.gets
JSON.parse(chunk, :max_nesting => false)
end
}
}
- x.report {
- puts "ActiveSupport::JSON.decode"
- times.times {
- json.rewind
- while chunk = json.gets
- ActiveSupport::JSON.decode(chunk)
- end
- }
- }
+ # x.report {
+ # puts "ActiveSupport::JSON.decode"
+ # times.times {
+ # json.rewind
+ # while chunk = json.gets
+ # ActiveSupport::JSON.decode(chunk)
+ # end
+ # }
+ # }
}
json.close
Oops, something went wrong.

0 comments on commit 8070e10

Please sign in to comment.