Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Add basic JSON building capabilities. #45

Open
wants to merge 4 commits into from

3 participants

Andrew Harvey Matt Mitchell Naomi Dushay
Andrew Harvey

I've added the ability to update using the json update functionality added in Solr 3.x.

The main caveat is that the json format that Solr uses is strange and I'd go so far as to say broken. I've got it working, but some functionality, such as attributes which would have been on the node in the XML are not possible.

I've not yet implemented commits, optimises and deletes yet, but those won't be far off. For the moment, even with a client configured to update with json, it will fall back to XML where the json hasn't been implemented yet.

Of course the docs also have to be updated, but I've tried to maintain the testing style and coverage level. At some point, I'd like to refactor those specs a little, all the same.

Andrew Harvey and others added some commits
Andrew Harvey Add basic JSON building capabilities.
The main piece of functionality lacking at this point is multi-value
fields. Currently the XML duplicates them. Perhaps it's worth
subclassing document for XML and JSON so that we get the main document
logic exposed as well as proper management of these fields.

Aside from that, the only thing remaining is to have a way of choosing
to add using json rather than XML.
fd57206
Andrew Harvey mootpointer Add RSolr::JSON::Document for json documents
It's a pretty thin sub-class of RSolr::Document. The only change is how
it handles multi-valued fields. Pretty simple if you ask me :)
b422407
Andrew Harvey mootpointer Fix JSON format so it is accepted by Solr
It turns out that Solr's JSON format is either too complex (duplicate
keys) or too simple (removing the ability for session based boosts or
commitwithin). I've opted to implement the simple. I plan to raise an
issue to get the format sorted out.
31377a1
Andrew Harvey mootpointer Plumb in json updating to the client
To update using json, the client needs to initialized with the
:update_format option set to :json.
81df3e5
Matt Mitchell
Owner

Hi. So the status is that this is working, but not complete? I'll have a look and get a feel for the changes. Thanks for this.

  • Matt
Chris Beer cbeer referenced this pull request in projectblacklight/blacklight
Closed

get json-formated data from solr instead of eval-ing a ruby response #661

Naomi Dushay
Owner

Is this addressed by #72 ? I think this should be closed and a new conversation/pull request should be started if the concerns haven't been addressed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Feb 28, 2012
  1. Add basic JSON building capabilities.

    Andrew Harvey authored
    The main piece of functionality lacking at this point is multi-value
    fields. Currently the XML duplicates them. Perhaps it's worth
    subclassing document for XML and JSON so that we get the main document
    logic exposed as well as proper management of these fields.
    
    Aside from that, the only thing remaining is to have a way of choosing
    to add using json rather than XML.
  2. Andrew Harvey

    Add RSolr::JSON::Document for json documents

    mootpointer authored
    It's a pretty thin sub-class of RSolr::Document. The only change is how
    it handles multi-valued fields. Pretty simple if you ask me :)
  3. Andrew Harvey

    Fix JSON format so it is accepted by Solr

    mootpointer authored
    It turns out that Solr's JSON format is either too complex (duplicate
    keys) or too simple (removing the ability for session based boosts or
    commitwithin). I've opted to implement the simple. I plan to raise an
    issue to get the format sorted out.
  4. Andrew Harvey

    Plumb in json updating to the client

    mootpointer authored
    To update using json, the client needs to initialized with the
    :update_format option set to :json.
This page is out of date. Refresh to see the latest.
6 Gemfile
View
@@ -3,6 +3,7 @@ source "http://rubygems.org"
gemspec
gem "builder", ">= 2.1.2"
+gem "jsonify"
group :development do
gem "rake", "~> 0.9.2"
@@ -12,4 +13,7 @@ end
group :test do
gem "rake", "~> 0.9.2"
gem "rspec", "~> 2.6.0"
-end
+ gem "json_spec"
+ gem "json"
+ gem 'pry'
+end
4 lib/rsolr.rb
View
@@ -4,7 +4,7 @@
module RSolr
- %W(Response Char Client Error Connection Uri Xml).each{|n|autoload n.to_sym, "rsolr/#{n.downcase}"}
+ %W(JSON Document Response Char Client Error Connection Uri Xml).each{|n|autoload n.to_sym, "rsolr/#{n.downcase}"}
def self.version; "1.0.7" end
@@ -19,4 +19,4 @@ def self.connect *args
# RSolr.escape
extend Char
-end
+end
24 lib/rsolr/client.rb
View
@@ -14,6 +14,7 @@ def initialize connection, options = {}
proxy_url << "/" unless proxy_url.nil? or proxy_url[-1] == ?/
@proxy = RSolr::Uri.create proxy_url if proxy_url
end
+ @update_format = options.delete(:update_format) || :xml
end
@options = options
end
@@ -66,7 +67,18 @@ def update opts = {}
opts[:headers]['Content-Type'] ||= 'text/xml'
post 'update', opts
end
-
+
+ # POST JSON messages to /update/json with optional params
+ # http://wiki.apache.org/solr/UpdateJSON
+ #
+ # Analagous to +#update+, but with json instead of XML.
+ #
+ def update_json opts = {}
+ opts[:headers] ||= {}
+ opts[:headers]['Content-Type'] ||= 'application/json'
+ post 'update/json', opts
+ end
+
#
# +add+ creates xml "add" documents and sends the xml data to the +update+ method
#
@@ -84,7 +96,11 @@ def update opts = {}
#
def add doc, opts = {}
add_attributes = opts.delete :add_attributes
- update opts.merge(:data => xml.add(doc, add_attributes))
+ if @update_format == :json
+ update_json opts.merge(:data => json.add(doc, add_attributes))
+ else
+ update opts.merge(:data => xml.add(doc, add_attributes))
+ end
end
# send "commit" xml with opts
@@ -135,6 +151,10 @@ def delete_by_query query, opts = {}
def xml
@xml ||= RSolr::Xml::Generator.new
end
+
+ def json
+ @json ||= RSolr::JSON::Generator.new
+ end
# +send_and_receive+ is the main request method responsible for sending requests to the +connection+ object.
#
68 lib/rsolr/document.rb
View
@@ -0,0 +1,68 @@
+module RSolr
+ class Document
+ # "attrs" is a hash for setting the "doc" xml attributes
+ # "fields" is an array of Field objects
+ attr_accessor :attrs, :fields
+
+ # "doc_hash" must be a Hash/Mash object
+ # If a value in the "doc_hash" is an array,
+ # a field object is created for each value...
+ def initialize(doc_hash = {})
+ @fields = []
+ doc_hash.each_pair do |field,values|
+ # create a new field for each value (multi-valued)
+ # put non-array values into an array
+ values = [values] unless values.is_a?(Array)
+ values.each do |v|
+ next if v.to_s.empty?
+ @fields << RSolr::Field.new({:name=>field}, v.to_s)
+ end
+ end
+ @attrs={}
+ end
+
+ # returns an array of fields that match the "name" arg
+ def fields_by_name(name)
+ @fields.select{|f|f.name==name}
+ end
+
+ # returns the *first* field that matches the "name" arg
+ def field_by_name(name)
+ @fields.detect{|f|f.name==name}
+ end
+
+ #
+ # Add a field value to the document. Options map directly to
+ # XML attributes in the Solr <field> node.
+ # See http://wiki.apache.org/solr/UpdateXmlMessages#head-8315b8028923d028950ff750a57ee22cbf7977c6
+ #
+ # === Example:
+ #
+ # document.add_field('title', 'A Title', :boost => 2.0)
+ #
+ def add_field(name, value, options = {})
+ @fields << RSolr::Field.new(options.merge({:name=>name}), value)
+ end
+
+ end
+
+ class Field
+
+ # "attrs" is a hash for setting the "doc" xml attributes
+ # "value" is the text value for the node
+ attr_accessor :attrs, :value
+
+ # "attrs" must be a hash
+ # "value" should be something that responds to #_to_s
+ def initialize(attrs, value)
+ @attrs = attrs
+ @value = value
+ end
+
+ # the value of the "name" attribute
+ def name
+ @attrs[:name]
+ end
+
+ end
+end
41 lib/rsolr/json.rb
View
@@ -0,0 +1,41 @@
+require 'jsonify'
+
+module RSolr::JSON
+ class Generator
+ def build &block
+ b = Jsonify::Builder.new(:format => :pretty)
+ block_given? ? yield(b) : b
+ end
+
+ def add data, add_attrs = nil, &block
+ data = [data] unless data.is_a?(Array)
+ build do |json|
+ json.add data do |doc|
+ doc = RSolr::JSON::Document.new(doc) if doc.respond_to?(:each_pair)
+ yield doc if block_given?
+ doc.attrs.map{|k,v| json[k] = v} if doc.attrs
+ doc.fields.each do |f|
+ if f.attrs.keys.length > 1
+ json[f.name] = f.attrs.merge(:value => f.value)
+ else
+ json[f.name] = f.value
+ end
+ end
+ end
+ json.compile!
+ end
+ end
+ end
+
+ class Document < RSolr::Document
+ def initialize(doc_hash = {})
+ @fields = []
+ @attrs = {}
+ doc_hash.each_pair do |field, values|
+ vals = values.is_a?(Array) ? values.map(&:to_s) : values
+ @fields << RSolr::Field.new({:name => field}, vals)
+ end
+ end
+ end
+end
+
73 lib/rsolr/xml.rb
View
@@ -1,75 +1,6 @@
require 'builder'
module RSolr::Xml
-
- class Document
-
- # "attrs" is a hash for setting the "doc" xml attributes
- # "fields" is an array of Field objects
- attr_accessor :attrs, :fields
-
- # "doc_hash" must be a Hash/Mash object
- # If a value in the "doc_hash" is an array,
- # a field object is created for each value...
- def initialize(doc_hash = {})
- @fields = []
- doc_hash.each_pair do |field,values|
- # create a new field for each value (multi-valued)
- # put non-array values into an array
- values = [values] unless values.is_a?(Array)
- values.each do |v|
- next if v.to_s.empty?
- @fields << RSolr::Xml::Field.new({:name=>field}, v.to_s)
- end
- end
- @attrs={}
- end
-
- # returns an array of fields that match the "name" arg
- def fields_by_name(name)
- @fields.select{|f|f.name==name}
- end
-
- # returns the *first* field that matches the "name" arg
- def field_by_name(name)
- @fields.detect{|f|f.name==name}
- end
-
- #
- # Add a field value to the document. Options map directly to
- # XML attributes in the Solr <field> node.
- # See http://wiki.apache.org/solr/UpdateXmlMessages#head-8315b8028923d028950ff750a57ee22cbf7977c6
- #
- # === Example:
- #
- # document.add_field('title', 'A Title', :boost => 2.0)
- #
- def add_field(name, value, options = {})
- @fields << RSolr::Xml::Field.new(options.merge({:name=>name}), value)
- end
-
- end
-
- class Field
-
- # "attrs" is a hash for setting the "doc" xml attributes
- # "value" is the text value for the node
- attr_accessor :attrs, :value
-
- # "attrs" must be a hash
- # "value" should be something that responds to #_to_s
- def initialize(attrs, value)
- @attrs = attrs
- @value = value
- end
-
- # the value of the "name" attribute
- def name
- @attrs[:name]
- end
-
- end
-
class Generator
def build &block
@@ -109,7 +40,7 @@ def add data, add_attrs = nil, &block
build do |xml|
xml.add(add_attrs) do |add_node|
data.each do |doc|
- doc = RSolr::Xml::Document.new(doc) if doc.respond_to?(:each_pair)
+ doc = RSolr::Document.new(doc) if doc.respond_to?(:each_pair)
yield doc if block_given?
add_node.doc(doc.attrs) do |doc_node|
doc.fields.each do |field_obj|
@@ -162,4 +93,4 @@ def delete_by_query(queries)
end
-end
+end
56 spec/api/client_spec.rb
View
@@ -55,6 +55,13 @@ def client
client.xml.should be_a RSolr::Xml::Generator
end
end
+
+ context "json" do
+ include ClientHelper
+ it "should return an instance of RSolr::JSON::Generator" do
+ client.json.should be_a RSolr::JSON::Generator
+ end
+ end
context "add" do
include ClientHelper
@@ -78,8 +85,55 @@ def client
and_return("<xml/>")
client.add({:id=>1}, :add_attributes => {:commitWith=>10})
end
+
+ context 'when the client is configured for json updates' do
+ let(:json_client) do
+ connection = RSolr::Connection.new
+ RSolr::Client.new connection, :update_format => :json
+ end
+ it "should send json to the connection's #post method" do
+ json_client.connection.should_receive(:execute).
+ with(
+ json_client, hash_including({
+ :path => 'update/json',
+ :headers => {"Content-Type" => 'application/json'},
+ :method => :post,
+ :data => '{"hello":"this is json"}'
+ })
+ ).
+ and_return(
+ :body => "",
+ :status => 200,
+ :headers => {"Content-Type"=>"text/xml"}
+ )
+ json_client.json.should_receive(:add).
+ with({:id => 1}, {:commitWith=>10}).
+ and_return('{"hello":"this is json"}')
+ json_client.add({:id=>1}, :add_attributes => {:commitWith=>10})
+ end
+ end
+ end
+
+ context "update_json" do
+ include ClientHelper
+ it "should send json to the connection's #post method" do
+ client.connection.should_receive(:execute).
+ with(
+ client, hash_including({
+ :path => 'update/json',
+ :headers => {'Content-Type'=>'application/json'},
+ :method => :post,
+ :data => '{"optimise" : {}}'
+ })
+ ).
+ and_return(
+ :body => "",
+ :status => 200,
+ :headers => {"Content-Type"=>"text/xml"}
+ )
+ client.update_json(:data => '{"optimise" : {}}')
+ end
end
-
context "update" do
include ClientHelper
it "should send data to the connection's #post method" do
62 spec/api/json_spec.rb
View
@@ -0,0 +1,62 @@
+require 'spec_helper'
+require 'json_spec'
+require 'pry'
+
+
+
+describe 'RSolr::JSON' do
+
+ let(:generator){ RSolr::JSON::Generator.new }
+
+ context :add do
+
+ it 'should yield a Document object when #add is called with a block' do
+ documents = [{:id=>1, :name=>'sam', :cat=>['cat 1', 'cat 2']}]
+ result = generator.add(documents) do |doc|
+ doc.field_by_name(:name).attrs[:boost] = 10
+ doc.fields.size.should == 3
+ doc.fields_by_name(:cat).size.should == 1
+ end
+ result.should be_json_eql('["cat 1", "cat 2"]').at_path('add/0/cat')
+ result.should be_json_eql('10').at_path('add/0/name/boost')
+ end
+
+ # add a single hash ("doc")
+ it 'should create an add from a hash' do
+ data = {
+ :id=>"1",
+ :name=>'matt'
+ }
+ result = generator.add(data)
+ result.should have_json_path("add")
+ result.should have_json_size(1).at_path("add")
+ result.should be_json_eql(data.to_json).at_path('add/0')
+ end
+
+ # add an array of hashes
+ it 'should create many adds from an array of hashes' do
+ data = [
+ {
+ :id=>"1",
+ :name=>'matt'
+ },
+ {
+ :id=>"2",
+ :name=>'sam'
+ }
+ ]
+ message = generator.add(data)
+ message.should have_json_size(2).at_path("add")
+ end
+ end
+
+ it 'should create multiple fields from array values' do
+ data = {
+ :id => 1,
+ :name => ['matt1', 'matt2']
+ }
+ result = generator.add(data)
+ result.should be_json_eql('["matt1","matt2"]').at_path('add/0/name')
+ end
+
+end
6 spec/api/xml_spec.rb
View
@@ -71,7 +71,7 @@
end
it 'should create an add from a single Message::Document' do
- document = RSolr::Xml::Document.new
+ document = RSolr::Document.new
document.add_field('id', 1)
document.add_field('name', 'matt', :boost => 2.0)
result = generator.add(document)
@@ -84,7 +84,7 @@
it 'should create adds from multiple Message::Documents' do
documents = (1..2).map do |i|
- doc = RSolr::Xml::Document.new
+ doc = RSolr::Document.new
doc.add_field('id', i)
doc.add_field('name', "matt#{i}")
doc
@@ -118,4 +118,4 @@
end
end
-end
+end
Something went wrong with that request. Please try again.