New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

performance issues #287

Closed
doughsay opened this Issue Aug 14, 2014 · 22 comments

Comments

Projects
None yet
7 participants
@doughsay

doughsay commented Aug 14, 2014

First let me say that this is an amazing gem, I really enjoy working with it and it really fits my needs exactly. But I've just realized that I might not be able to use it due to performance issues, which is a real shame.

Take this rather contrived example below to see how badly it seems to perform for me:

class Foo
  attr_accessor :foo
end

class Bar
  include Mongoid::Document
  field :bar, :type => String
end

class Baz
  include Virtus.model
  attribute :baz, String
end

time_me = ->(proc) {
  start_time = Time.now.to_f
  proc.call()
  end_time = Time.now.to_f
  time_in_millis = ((end_time - start_time) * 1000).to_i
  puts time_in_millis
}

time_me.call ->{1000.times{ foo = Foo.new; foo.foo = 'foo'; }}
# => 0

time_me.call ->{1000.times{ bar = Bar.new; bar.bar = 'bar'; }}
# => 21

time_me.call ->{1000.times{ baz = Baz.new; baz.baz = 'baz'; }}
# => 592

As you can see, it appears Virtus models perform 30x slower (than mongoid documents) in simple cases like this, where mongoid is providing the same functionality, i.e. attribute definition and type coercion.

Basically in my case, what I'm seeing is Virtus taking almost 2 seconds to "hydrate" less than 10 models each with about 100 attributes. This makes it pretty much unusable... I really hope there's something that can be done about it because I'd love to be able to use Virtus.

@coladarci

This comment has been minimized.

Show comment
Hide comment
@coladarci

coladarci Aug 14, 2014

👍 - such a shame..

coladarci commented Aug 14, 2014

👍 - such a shame..

@solnic solnic added this to the 2.0.0 milestone Aug 14, 2014

@solnic

This comment has been minimized.

Show comment
Hide comment
@solnic

solnic Aug 14, 2014

Owner

This will be addressed in virtus 2.0. Poor performance is probably caused by its coercion backend which supports more types of coercions than Mongoid and relies on stuff like regexp parsing which is just very slow.

I'm marking this issue as a todo for 2.0.

Owner

solnic commented Aug 14, 2014

This will be addressed in virtus 2.0. Poor performance is probably caused by its coercion backend which supports more types of coercions than Mongoid and relies on stuff like regexp parsing which is just very slow.

I'm marking this issue as a todo for 2.0.

@coladarci

This comment has been minimized.

Show comment
Hide comment
@coladarci

coladarci Aug 14, 2014

@solnic, despite it probably not being a good idea, in your estimation, how complicated would it be for us to fork and remove the extra bloat if we don't need all of the extra support for the types? (in general we stay away from this method, but we do love this gem...)

coladarci commented Aug 14, 2014

@solnic, despite it probably not being a good idea, in your estimation, how complicated would it be for us to fork and remove the extra bloat if we don't need all of the extra support for the types? (in general we stay away from this method, but we do love this gem...)

@elskwid

This comment has been minimized.

Show comment
Hide comment
@elskwid

elskwid Aug 14, 2014

Collaborator

@doughsay - I don't see the same results here. Take a look at these numbers after running this gist:

> be ruby coerce-speed.rb
Rehearsal -------------------------------------------------------
Poro                  0.020000   0.000000   0.020000 (  0.025502)
MongoidDoc            0.460000   0.000000   0.460000 (  0.464085)
VirtusModel           0.210000   0.010000   0.220000 (  0.219621)
VirtusModelNoCoerce   0.060000   0.000000   0.060000 (  0.062512)
---------------------------------------------- total: 0.760000sec

                          user     system      total        real
Poro                  0.010000   0.000000   0.010000 (  0.004206)
MongoidDoc            0.420000   0.010000   0.430000 (  0.430198)
VirtusModel           0.210000   0.000000   0.210000 (  0.242876)
VirtusModelNoCoerce   0.050000   0.000000   0.050000 (  0.056276)
Collaborator

elskwid commented Aug 14, 2014

@doughsay - I don't see the same results here. Take a look at these numbers after running this gist:

> be ruby coerce-speed.rb
Rehearsal -------------------------------------------------------
Poro                  0.020000   0.000000   0.020000 (  0.025502)
MongoidDoc            0.460000   0.000000   0.460000 (  0.464085)
VirtusModel           0.210000   0.010000   0.220000 (  0.219621)
VirtusModelNoCoerce   0.060000   0.000000   0.060000 (  0.062512)
---------------------------------------------- total: 0.760000sec

                          user     system      total        real
Poro                  0.010000   0.000000   0.010000 (  0.004206)
MongoidDoc            0.420000   0.010000   0.430000 (  0.430198)
VirtusModel           0.210000   0.000000   0.210000 (  0.242876)
VirtusModelNoCoerce   0.050000   0.000000   0.050000 (  0.056276)
@elskwid

This comment has been minimized.

Show comment
Hide comment
@elskwid

elskwid Aug 14, 2014

Collaborator

@doughsay - that wasn't to say that what you're seeing isn't happening, just that I couldn't see it in my example. 2 seconds for 10 models sounds like a lot of time.

@coladarci - as you can see in my gist above, you can opt out of coercion with Virtus.model(coerce: false). Is that enough?

Collaborator

elskwid commented Aug 14, 2014

@doughsay - that wasn't to say that what you're seeing isn't happening, just that I couldn't see it in my example. 2 seconds for 10 models sounds like a lot of time.

@coladarci - as you can see in my gist above, you can opt out of coercion with Virtus.model(coerce: false). Is that enough?

@doughsay

This comment has been minimized.

Show comment
Hide comment
@doughsay

doughsay Aug 14, 2014

@elskwid - Thanks a lot for that, this indicates that it might be something else on my end causing the slowdown. That's good news; I'll do some more investigation on my end.

My results from your gist:
ruby-1.9.3-p547 + mongoid-3.1.6 + virtus-1.0.3

Rehearsal -------------------------------------------------------
Poro                  0.010000   0.000000   0.010000 (  0.003283)
MongoidDoc            0.280000   0.000000   0.280000 (  0.283630)
VirtusModel           0.480000   0.010000   0.490000 (  0.506913)
VirtusModelNoCoerce   0.030000   0.000000   0.030000 (  0.030868)
---------------------------------------------- total: 0.810000sec

                          user     system      total        real
Poro                  0.010000   0.000000   0.010000 (  0.003304)
MongoidDoc            0.270000   0.000000   0.270000 (  0.263741)
VirtusModel           0.480000   0.010000   0.490000 (  0.495392)
VirtusModelNoCoerce   0.040000   0.000000   0.040000 (  0.041859)

ruby-2.1.2 + mongoid-4.0.0 + virtus-1.0.3

Rehearsal -------------------------------------------------------
Poro                  0.010000   0.000000   0.010000 (  0.004674)
MongoidDoc            0.330000   0.000000   0.330000 (  0.329595)
VirtusModel           0.130000   0.000000   0.130000 (  0.135011)
VirtusModelNoCoerce   0.040000   0.000000   0.040000 (  0.042119)
---------------------------------------------- total: 0.510000sec

                          user     system      total        real
Poro                  0.000000   0.000000   0.000000 (  0.004560)
MongoidDoc            0.300000   0.000000   0.300000 (  0.304607)
VirtusModel           0.130000   0.010000   0.140000 (  0.141968)
VirtusModelNoCoerce   0.040000   0.000000   0.040000 (  0.033637)

Ruby 2 is definitely faster.

doughsay commented Aug 14, 2014

@elskwid - Thanks a lot for that, this indicates that it might be something else on my end causing the slowdown. That's good news; I'll do some more investigation on my end.

My results from your gist:
ruby-1.9.3-p547 + mongoid-3.1.6 + virtus-1.0.3

Rehearsal -------------------------------------------------------
Poro                  0.010000   0.000000   0.010000 (  0.003283)
MongoidDoc            0.280000   0.000000   0.280000 (  0.283630)
VirtusModel           0.480000   0.010000   0.490000 (  0.506913)
VirtusModelNoCoerce   0.030000   0.000000   0.030000 (  0.030868)
---------------------------------------------- total: 0.810000sec

                          user     system      total        real
Poro                  0.010000   0.000000   0.010000 (  0.003304)
MongoidDoc            0.270000   0.000000   0.270000 (  0.263741)
VirtusModel           0.480000   0.010000   0.490000 (  0.495392)
VirtusModelNoCoerce   0.040000   0.000000   0.040000 (  0.041859)

ruby-2.1.2 + mongoid-4.0.0 + virtus-1.0.3

Rehearsal -------------------------------------------------------
Poro                  0.010000   0.000000   0.010000 (  0.004674)
MongoidDoc            0.330000   0.000000   0.330000 (  0.329595)
VirtusModel           0.130000   0.000000   0.130000 (  0.135011)
VirtusModelNoCoerce   0.040000   0.000000   0.040000 (  0.042119)
---------------------------------------------- total: 0.510000sec

                          user     system      total        real
Poro                  0.000000   0.000000   0.000000 (  0.004560)
MongoidDoc            0.300000   0.000000   0.300000 (  0.304607)
VirtusModel           0.130000   0.010000   0.140000 (  0.141968)
VirtusModelNoCoerce   0.040000   0.000000   0.040000 (  0.033637)

Ruby 2 is definitely faster.

@elskwid

This comment has been minimized.

Show comment
Hide comment
@elskwid

elskwid Aug 14, 2014

Collaborator

@doughsay - Excellent! Let me know if you need a hand or a second pair of eyes on something. I find this kind of thing very interesting. Good luck!

Collaborator

elskwid commented Aug 14, 2014

@doughsay - Excellent! Let me know if you need a hand or a second pair of eyes on something. I find this kind of thing very interesting. Good luck!

@doughsay

This comment has been minimized.

Show comment
Hide comment
@doughsay

doughsay Aug 14, 2014

@elskwid - I've figure out that it must come from rails, or at least some other gem we're including in our project. I added your gist as a rake task in our project and got this result:

Rehearsal -------------------------------------------------------
Poro                  0.010000   0.000000   0.010000 (  0.004219)
MongoidDoc            0.270000   0.020000   0.290000 (  0.300970)
VirtusModel           4.150000   0.020000   4.170000 (  4.219307)
VirtusModelNoCoerce   0.040000   0.000000   0.040000 (  0.041164)
---------------------------------------------- total: 4.510000sec

                          user     system      total        real
Poro                  0.000000   0.000000   0.000000 (  0.004330)
MongoidDoc            0.220000   0.000000   0.220000 (  0.227933)
VirtusModel           3.880000   0.010000   3.890000 (  3.919318)
VirtusModelNoCoerce   0.040000   0.000000   0.040000 (  0.037428)

Any idea what could be causing this? Also, I don't really know how to continue my investigation from this point...

This is rails 3.2.13, and those results were using ruby-2.1.2, but we usually run this project with 1.9.3.

doughsay commented Aug 14, 2014

@elskwid - I've figure out that it must come from rails, or at least some other gem we're including in our project. I added your gist as a rake task in our project and got this result:

Rehearsal -------------------------------------------------------
Poro                  0.010000   0.000000   0.010000 (  0.004219)
MongoidDoc            0.270000   0.020000   0.290000 (  0.300970)
VirtusModel           4.150000   0.020000   4.170000 (  4.219307)
VirtusModelNoCoerce   0.040000   0.000000   0.040000 (  0.041164)
---------------------------------------------- total: 4.510000sec

                          user     system      total        real
Poro                  0.000000   0.000000   0.000000 (  0.004330)
MongoidDoc            0.220000   0.000000   0.220000 (  0.227933)
VirtusModel           3.880000   0.010000   3.890000 (  3.919318)
VirtusModelNoCoerce   0.040000   0.000000   0.040000 (  0.037428)

Any idea what could be causing this? Also, I don't really know how to continue my investigation from this point...

This is rails 3.2.13, and those results were using ruby-2.1.2, but we usually run this project with 1.9.3.

@doughsay

This comment has been minimized.

Show comment
Hide comment
@doughsay

doughsay Aug 14, 2014

Found it! gem 'better_errors', '1.1.0' causes performance to go to 💩

doughsay commented Aug 14, 2014

Found it! gem 'better_errors', '1.1.0' causes performance to go to 💩

@elskwid

This comment has been minimized.

Show comment
Hide comment
@elskwid

elskwid Aug 14, 2014

Collaborator

Oh snap! Great detective work. I wonder if it's actually binding_of_caller since that gets in the callstack.

Collaborator

elskwid commented Aug 14, 2014

Oh snap! Great detective work. I wonder if it's actually binding_of_caller since that gets in the callstack.

@doughsay

This comment has been minimized.

Show comment
Hide comment
@doughsay

doughsay Aug 15, 2014

Yep, that's it. You can still include better_errors, just not binding_of_caller. Good call.

doughsay commented Aug 15, 2014

Yep, that's it. You can still include better_errors, just not binding_of_caller. Good call.

@elskwid

This comment has been minimized.

Show comment
Hide comment
@elskwid

elskwid Aug 15, 2014

Collaborator

@doughsay - I'm going to close this. Let us know if you run into any more trouble.

Collaborator

elskwid commented Aug 15, 2014

@doughsay - I'm going to close this. Let us know if you run into any more trouble.

@elskwid elskwid closed this Aug 15, 2014

@skade

This comment has been minimized.

Show comment
Hide comment
@skade

skade Aug 15, 2014

To add an anecdote (I have to dig for numbers), we found that the coercer plays really well with the JRuby JIT.

skade commented Aug 15, 2014

To add an anecdote (I have to dig for numbers), we found that the coercer plays really well with the JRuby JIT.

@jgdreyes

This comment has been minimized.

Show comment
Hide comment
@jgdreyes

jgdreyes Nov 26, 2014

@doughsay - I was seeing performance issues with Virtus as well. Though I love the coercion, type specifier and default values for my form objects, I don't always need these features. I ran a benchmark as well (outside of rails) using value objects that are mutable: https://github.com/codeshoppe/simple_attrs_benchmark. I found that Virtus still performs slower than a few other gems.

I still use Virtus for form_objects and harness all of its powerful features, but I have switched to simple_attrs for a simple attribute DSL.

(Note that the performance issues I see worsen as I add more attributes.)

jgdreyes commented Nov 26, 2014

@doughsay - I was seeing performance issues with Virtus as well. Though I love the coercion, type specifier and default values for my form objects, I don't always need these features. I ran a benchmark as well (outside of rails) using value objects that are mutable: https://github.com/codeshoppe/simple_attrs_benchmark. I found that Virtus still performs slower than a few other gems.

I still use Virtus for form_objects and harness all of its powerful features, but I have switched to simple_attrs for a simple attribute DSL.

(Note that the performance issues I see worsen as I add more attributes.)

@solnic

This comment has been minimized.

Show comment
Hide comment
@solnic

solnic Nov 26, 2014

Owner

@jgdreyes performance will be addressed in virtus 2.0 rewrite, there are lots of features that I think should be removed, some features that should be optional and probably a couple of low-hanging fruits which could speed things up. Stay tuned :)

Owner

solnic commented Nov 26, 2014

@jgdreyes performance will be addressed in virtus 2.0 rewrite, there are lots of features that I think should be removed, some features that should be optional and probably a couple of low-hanging fruits which could speed things up. Stay tuned :)

@jgdreyes

This comment has been minimized.

Show comment
Hide comment
@jgdreyes

jgdreyes Nov 26, 2014

@solnic I did read that and I'm super excited. I really do love Virtus and still use it in production. It fits my needs perfectly for most scenarios.

Specifically I'd like to see performance improvements as the number of attributes grow within a single virtus model. If you look at the benchmark code I added in my previous comment, I instantiate 100,000 objects with 25 attributes. The use case for this is to consume an external API that returns back an array of hashes. I'd like to work with plain old ruby objects instead of hashes so that I can layer on additional functionality to each record. However, with my little spike it was clear that the performance wasn't acceptable. Temporarily I've switched to using a simple DSL, but would love to continue using Virtus. :) Keep up the great work and thanks for the reply!

jgdreyes commented Nov 26, 2014

@solnic I did read that and I'm super excited. I really do love Virtus and still use it in production. It fits my needs perfectly for most scenarios.

Specifically I'd like to see performance improvements as the number of attributes grow within a single virtus model. If you look at the benchmark code I added in my previous comment, I instantiate 100,000 objects with 25 attributes. The use case for this is to consume an external API that returns back an array of hashes. I'd like to work with plain old ruby objects instead of hashes so that I can layer on additional functionality to each record. However, with my little spike it was clear that the performance wasn't acceptable. Temporarily I've switched to using a simple DSL, but would love to continue using Virtus. :) Keep up the great work and thanks for the reply!

@mbj

This comment has been minimized.

Show comment
Hide comment
@mbj

mbj Nov 27, 2014

Collaborator

@solnic I'd be interested in a small pairing where we figure out if morpher can help you to reduce the code and actually speed stuff up for 2.0. Morpher is pretty fast in non evaluation history mode and allows the VM to inline most of the method calls (Jruby / rbx), even on MRI I was not able to measure a difference from hand written code for complex nested transforms. Are you available for such a pairing this / next week?

Collaborator

mbj commented Nov 27, 2014

@solnic I'd be interested in a small pairing where we figure out if morpher can help you to reduce the code and actually speed stuff up for 2.0. Morpher is pretty fast in non evaluation history mode and allows the VM to inline most of the method calls (Jruby / rbx), even on MRI I was not able to measure a difference from hand written code for complex nested transforms. Are you available for such a pairing this / next week?

@solnic

This comment has been minimized.

Show comment
Hide comment
@solnic

solnic Nov 27, 2014

Owner

@mbj thanks Markus, before I dive into any coding I need to define what Virtus 2.0 is supposed to be :) let me first come up with a list of things I want it to support and then we'll be able to figure out how morpher fits here. Does that sound reasonable?

Owner

solnic commented Nov 27, 2014

@mbj thanks Markus, before I dive into any coding I need to define what Virtus 2.0 is supposed to be :) let me first come up with a list of things I want it to support and then we'll be able to figure out how morpher fits here. Does that sound reasonable?

@mbj

This comment has been minimized.

Show comment
Hide comment
@mbj

mbj Nov 27, 2014

Collaborator

@solnic Yeah it does. My point was to give you some examples on how to map virtus 1.0 to morpher + anima. To give you the ideas you need for 2.0 ;)

Collaborator

mbj commented Nov 27, 2014

@solnic Yeah it does. My point was to give you some examples on how to map virtus 1.0 to morpher + anima. To give you the ideas you need for 2.0 ;)

@solnic

This comment has been minimized.

Show comment
Hide comment
@solnic

solnic Nov 27, 2014

Owner

@mbj virtus 1.0 can't be used with anima because it'd break its public interface, that's why I plan to use morpher + maybe anima in 2.0. Main problem is that anima requires all keys to be present in the attribute hash that's passed to the constructor and probably a couple more differences

Owner

solnic commented Nov 27, 2014

@mbj virtus 1.0 can't be used with anima because it'd break its public interface, that's why I plan to use morpher + maybe anima in 2.0. Main problem is that anima requires all keys to be present in the attribute hash that's passed to the constructor and probably a couple more differences

@mbj

This comment has been minimized.

Show comment
Hide comment
@mbj

mbj Nov 27, 2014

Collaborator

@solnic I was not saying you should use virtus 1.0 backed by anima + morpher, I was saying we should talk on how it could be used to see a path to 2.0 as an explorational thought experiment.

Collaborator

mbj commented Nov 27, 2014

@solnic I was not saying you should use virtus 1.0 backed by anima + morpher, I was saying we should talk on how it could be used to see a path to 2.0 as an explorational thought experiment.

@mbj

This comment has been minimized.

Show comment
Hide comment
@mbj

mbj Nov 27, 2014

Collaborator

@solnic BTW computing absent keys is very easy and a total morpher transformer in front of the anima instantiation can easily fill the gabs with nil valued keys.

Collaborator

mbj commented Nov 27, 2014

@solnic BTW computing absent keys is very easy and a total morpher transformer in front of the anima instantiation can easily fill the gabs with nil valued keys.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment