Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Fix for memory issues #18

Closed
wants to merge 2 commits into from

4 participants

@tlipinski

Recently I had quite big xml to process (about 45MB xml with 190,000 elements originating from root, and lots of other nested elements) but happymapper keeped crashing with so much data due to full memory consumption. I've made some changes in happymapper's codebase so that it doesn't return whole collection of objects but yields it part by part if I set :limit option. Now if I set it to 1000 for instance, memory usage jumps up only by 20% and stays on this level during whole processing.

@mcollina

This is awesome. I'll move from ROXML to your branch asap :).
Could this be merged upstream?

@tlipinski

Matteo, @burtlo pulled and documented these changes into his fork (option changed to :in_groups_of) burtlo/happymapper@5538d3f.

@ivar

@jnunemaker any chance you could take a look at this ? It would be great to get this functionality into the main branch..

@jnunemaker jnunemaker closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Jun 21, 2011
  1. yield partial result by using :limit option

    pplcanfly authored
Commits on Sep 10, 2011
  1. Edited README.rdoc via GitHub

    Tomasz Lipinski authored
This page is out of date. Refresh to see the latest.
Showing with 45 additions and 67 deletions.
  1. +2 −54 README.rdoc
  2. +25 −13 lib/happymapper.rb
  3. +18 −0 spec/happymapper_spec.rb
View
56 README.rdoc
@@ -1,55 +1,3 @@
-= happymapper
+This fork adds an option that allows user to limit results returned by parser.
-== DESCRIPTION:
-
-XML to object mapping library. I have included examples to help get you going. The specs should also point you in the right direction.
-
-== FEATURES:
-
-* Easy to define xml attributes and elements for an object
-* Fast because it uses libxml-ruby under the hood
-* Automatic conversion of xml to defined objects
-
-== EXAMPLES:
-
-Here is a simple example that maps Twitter statuses and users.
-
- class User
- include HappyMapper
-
- element :id, Integer
- element :name, String
- element :screen_name, String
- element :location, String
- element :description, String
- element :profile_image_url, String
- element :url, String
- element :protected, Boolean
- element :followers_count, Integer
- end
-
- class Status
- include HappyMapper
-
- element :id, Integer
- element :text, String
- element :created_at, Time
- element :source, String
- element :truncated, Boolean
- element :in_reply_to_status_id, Integer
- element :in_reply_to_user_id, Integer
- element :favorited, Boolean
- has_one :user, User
- end
-
-See examples directory in the gem for more examples.
-
-http://github.com/jnunemaker/happymapper/tree/master/examples/
-
-== INSTALL:
-
-* gem install happymapper
-
-== DOCS:
-
-http://rdoc.info/projects/jnunemaker/happymapper
View
38 lib/happymapper.rb
@@ -96,24 +96,36 @@ def parse(xml, options = {})
xpath += tag_name
nodes = node.find(xpath, Array(namespace))
- collection = nodes.collect do |n|
- obj = new
- attributes.each do |attr|
- obj.send("#{attr.method_name}=",
- attr.from_xml_node(n, namespace))
- end
+ limit = options[:limit] || nodes.size
+ return [] if limit == 0
- elements.each do |elem|
- obj.send("#{elem.method_name}=",
- elem.from_xml_node(n, namespace))
- end
+ collection = []
+ nodes.each_slice(limit) do |slice|
+ part = slice.collect do |n|
+ obj = new
+
+ attributes.each do |attr|
+ obj.send("#{attr.method_name}=",
+ attr.from_xml_node(n, namespace))
+ end
+
+ elements.each do |elem|
+ obj.send("#{elem.method_name}=",
+ elem.from_xml_node(n, namespace))
+ end
- obj.send("#{@content}=", n.content) if @content
+ obj.send("#{@content}=", n.content) if @content
- obj.class.after_parse_callbacks.each { |callback| callback.call(obj) }
+ obj.class.after_parse_callbacks.each { |callback| callback.call(obj) }
- obj
+ obj
+ end
+ if options[:limit] && block_given?
+ yield part
+ else
+ collection += part
+ end
end
# per http://libxml.rubyforge.org/rdoc/classes/LibXML/XML/Document.html#M000354
View
18 spec/happymapper_spec.rb
@@ -395,4 +395,22 @@ def mapping
end
end
+ describe "with limit option" do
+ it "should return results with limited size: 6" do
+ sizes = []
+ posts = Post.parse(fixture_file('posts.xml'), :limit => 6) do |a|
+ sizes << a.size
+ end
+ sizes.should == [6, 6, 6, 2]
+ end
+
+ it "should return results with limited size: 10" do
+ sizes = []
+ posts = Post.parse(fixture_file('posts.xml'), :limit => 10) do |a|
+ sizes << a.size
+ end
+ sizes.should == [10, 10]
+ end
+ end
+
end
Something went wrong with that request. Please try again.