New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Module elimination #219

Open
der-flo opened this Issue Jul 4, 2017 · 6 comments

Comments

Projects
None yet
3 participants
@der-flo

der-flo commented Jul 4, 2017

#202 (method elimination inside the C extension) is a very interesting feature, but it doesn't fit my use-case. I want to profile e. g. RSpec tests. Of course I'm primarily interested in the performance of my code, not the code inside of utilising libraries. So I have to exclude RSpec, VCR, ActiveRecord, ... . This is possible with the following approach:

[
  RSpec::Core::Runner,
  RSpec::Core::ExampleGroup,
  RSpec::Core::Hooks::HookCollections,
  RSpec::Core::Example::Procsy,
  RSpec::Core::Hooks::AroundHook,
  RSpec::Core::Configuration,
  RSpec::Core::Example,
  RSpec::Core::Hooks::BeforeHook,
  <...>
].each do |clz|
  profile.exclude_methods! clz, clz.instance_methods
  profile.exclude_methods! clz, clz.private_instance_methods
  profile.exclude_singleton_methods! clz, clz.methods
end

But this has 2 major drawbacks: First I have to list many, also private classes, and second it affects performance dramatically. Sure, code like this generates many, many exclusion rules.

The legacy way to eliminate methods using regexes allows excluding complete modules/namespaces, so the rules get reduced to an acceptable minimum. But, as we all know, this approach has the drawback to profile massive amounts of data and filters afterwards, slows us down again.

What about adding the ability to exclude modules with all its methods? This should not affect the performance. Or going a step further and extending this concept to namespaces / parent modules?

@skaes

This comment has been minimized.

Show comment
Hide comment
@skaes

skaes Jul 8, 2017

Member

Excluding a complete namespace sounds like an interesting feature to me. But I don't think matching on module names or traversing instance hierarchies would be fast enough, so we'd probably would need to enter all the relevant names into the lookup table for excluded methods.

Member

skaes commented Jul 8, 2017

Excluding a complete namespace sounds like an interesting feature to me. But I don't think matching on module names or traversing instance hierarchies would be fast enough, so we'd probably would need to enter all the relevant names into the lookup table for excluded methods.

@skaes

This comment has been minimized.

Show comment
Hide comment
@skaes

skaes Jul 8, 2017

Member

@der-flo For the time being, you can simply eliminate the unwanted methods after profiling. Or use the following hack:

def exclude_module(profile, mod)
  ObjectSpace.each_object(Module) do |m|
    next unless m.name =~ /\A#{mod}(:?::.*)?\z/
    profile.exclude_methods! m, m.instance_methods
    profile.exclude_methods! m, m.private_instance_methods
    profile.exclude_singleton_methods! m, m.methods
  end
end
Member

skaes commented Jul 8, 2017

@der-flo For the time being, you can simply eliminate the unwanted methods after profiling. Or use the following hack:

def exclude_module(profile, mod)
  ObjectSpace.each_object(Module) do |m|
    next unless m.name =~ /\A#{mod}(:?::.*)?\z/
    profile.exclude_methods! m, m.instance_methods
    profile.exclude_methods! m, m.private_instance_methods
    profile.exclude_singleton_methods! m, m.methods
  end
end
@der-flo

This comment has been minimized.

Show comment
Hide comment
@der-flo

der-flo Jul 14, 2017

@skaes Thank your for your suggestion. I played around a bit and it seems quite useful!
Furthermore I don't see the extreme performance affectation anymore. I can't explain that yet.

One shortcoming of ObjectSpace.each_object(Module) ... is that it obviously only iterates over the currently loaded modules. In eg. a Rails environment many modules are typically autoloaded, so I cannot exclude them with this strategy. I have to require them manuallly, what makes the maintenance of such a list quite complicated.

I'll dig deeper in this topic they next days. However a native implementation of module exclusion would absolutly make sense, but I understand that this requirement is not trivial.

der-flo commented Jul 14, 2017

@skaes Thank your for your suggestion. I played around a bit and it seems quite useful!
Furthermore I don't see the extreme performance affectation anymore. I can't explain that yet.

One shortcoming of ObjectSpace.each_object(Module) ... is that it obviously only iterates over the currently loaded modules. In eg. a Rails environment many modules are typically autoloaded, so I cannot exclude them with this strategy. I have to require them manuallly, what makes the maintenance of such a list quite complicated.

I'll dig deeper in this topic they next days. However a native implementation of module exclusion would absolutly make sense, but I understand that this requirement is not trivial.

@skaes

This comment has been minimized.

Show comment
Hide comment
@skaes

skaes Jul 14, 2017

Member

@der-flo you can just eager load the rails app to mitigate the problem. Also, you can add a catch all elimination call for the modules which have not been called:

 result.eliminate_methods!([/\A#{mod}(:?::.*)?\z/])

This is then slightly less efficient than eliminating them during the profile run, but hey, it works.

Member

skaes commented Jul 14, 2017

@der-flo you can just eager load the rails app to mitigate the problem. Also, you can add a catch all elimination call for the modules which have not been called:

 result.eliminate_methods!([/\A#{mod}(:?::.*)?\z/])

This is then slightly less efficient than eliminating them during the profile run, but hey, it works.

@skaes

This comment has been minimized.

Show comment
Hide comment
@skaes

skaes Dec 16, 2017

Member

@schleyfox what's your take on this?

Member

skaes commented Dec 16, 2017

@schleyfox what's your take on this?

@schleyfox

This comment has been minimized.

Show comment
Hide comment
@schleyfox

schleyfox Dec 18, 2017

Contributor

I think @skaes suggestion to use the legacy method elimination is your best bet given your requirements. Native method exclusion has a performance advantage because it's only a lookup on a method table keyed by (class pointer, method id). If we don't have a class to have a pointer to, we're stuck in the land of strings.

Is the performance impact from your first attempt the result of creating all the exclusion rules or felt while running the profile (e.g. is this making the method table lookup less efficient in your experience)?

It's also worth considering that as you eliminate methods (or modules), you lose context and the ability to explain why the performance is the way it is. Especially if you are lazy loading the rails app while profiling, many methods will be slow because they are finding constants, rather than for any other reason. Selecting the appropriate visualization can often help with interpretation without hiding the details or framework code.

Contributor

schleyfox commented Dec 18, 2017

I think @skaes suggestion to use the legacy method elimination is your best bet given your requirements. Native method exclusion has a performance advantage because it's only a lookup on a method table keyed by (class pointer, method id). If we don't have a class to have a pointer to, we're stuck in the land of strings.

Is the performance impact from your first attempt the result of creating all the exclusion rules or felt while running the profile (e.g. is this making the method table lookup less efficient in your experience)?

It's also worth considering that as you eliminate methods (or modules), you lose context and the ability to explain why the performance is the way it is. Especially if you are lazy loading the rails app while profiling, many methods will be slow because they are finding constants, rather than for any other reason. Selecting the appropriate visualization can often help with interpretation without hiding the details or framework code.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment