Introduce MemoryProf for detecting test examples that cause memory sp…

…ikes

docs/profilers/memory_prof.md

@@ -0,0 +1,85 @@
+# MemoryProf
+
+MemoryProf tracks memory usage during your test suite run, and can help to detect test examples and groups that cause memory spikes. Memory profiling supports two metrics: RSS and allocations.
+
+Example output:
+
+```sh
+[TEST PROF INFO] MemoryProf results
+
+Final RSS: 673KB
+
+Top 5 groups (by RSS):
+
+AnswersController (./spec/controllers/answers_controller_spec.rb:3) – +80KB (13.50%)
+QuestionsController (./spec/controllers/questions_controller_spec.rb:3) – +32KB  (9.08%)
+CommentsController (./spec/controllers/comments_controller_spec.rb:3) – +16KB (3.27%)
+
+Top 5 examples (by RSS):
+
+destroys question (./spec/controllers/questions_controller_spec.rb:38) – +144KB (24.38%)
+change comments count (./spec/controllers/comments_controller_spec.rb:7) – +120KB (20.00%)
+change Votes count (./spec/shared_examples/controllers/voted_examples.rb:23) – +90KB (16.36%)
+change Votes count (./spec/shared_examples/controllers/voted_examples.rb:23) – +64KB (12.86%)
+fails (./spec/shared_examples/controllers/invalid_examples.rb:3) – +32KB (5.00%)
+```
+
+The examples block shows the amount of memory used by each example, and the groups block displays the memory allocated by other code defined in the groups. For example, RSpec groups may include heavy `before(:all)` (or `before_all`) setup blocks, so it is helpful to see which groups use the most amount of memory outside of their examples.
+
+## Instructions
+
+To activate MemoryProf with:
+
+### RSpec
+
+Use `TEST_MEM_PROF` environment variable to set which metric to use:
+
+```sh
+TEST_MEM_PROF='rss' rspec ...
+TEST_MEM_PROF='alloc' rake rspec ...
+```
+
+### Minitest
+
+Use `TEST_MEM_PROF` environment variable to set which metric to use:
+
+```sh
+TEST_MEM_PROF='rss' rake test
+TEST_MEM_PROF='alloc' rspec ...
+```
+
+or use CLI options as well:
+
+```sh
+# Run a specific file using CLI option
+ruby test/my_super_test.rb --mem-prof=rss
+
+# Show the list of possible options:
+ruby test/my_super_test.rb --help
+```
+
+## Configuration
+
+By default, MemoryProf tracks the top 5 examples and groups that use the largest amount of memory.
+You can set how many examples/groups to display with the option:
+
+```sh
+TEST_MEM_PROF='rss' TEST_MEM_PROF_COUNT=10 rspec ...
+```
+
+or with CLI options for Minitest:
+
+```sh
+# Run a specific file using CLI option
+ruby test/my_super_test.rb --mem-prof=rs --mem-prof-top-count=10
+```
+
+## Supported Ruby Engines & OS
+
+Currently the allocation mode is not supported for JRuby.
+
+Since RSS depends on the OS, MemoryProf uses different tools to retrieve it:
+
+* Linux – `/proc/$pid/statm` file,
+* macOS, Solaris, BSD – `ps`,
+* Windows – `Get-Process`, requires PowerShell to be installed.

lib/minitest/test_prof_plugin.rb

@@ -2,6 +2,7 @@
 
 require "test_prof/event_prof/minitest"
 require "test_prof/factory_doctor/minitest"
+require "test_prof/memory_prof/minitest"
 
 module Minitest # :nodoc:
   module TestProf # :nodoc:
@@ -13,6 +14,8 @@ def self.configure_options(options = {})
         opts[:per_example] = true if ENV["EVENT_PROF_EXAMPLES"]
         opts[:fdoc] = true if ENV["FDOC"]
         opts[:sample] = true if ENV["SAMPLE"] || ENV["SAMPLE_GROUPS"]
+        opts[:mem_prof_mode] = ENV["TEST_MEM_PROF"] if ENV["TEST_MEM_PROF"]
+        opts[:mem_prof_top_count] = ENV["TEST_MEM_PROF_COUNT"] if ENV["TEST_MEM_PROF_COUNT"]
       end
     end
   end
@@ -33,13 +36,20 @@ def self.plugin_test_prof_options(opts, options)
     opts.on "--factory-doctor", TrueClass, "Enable Factory Doctor for your examples" do |flag|
       options[:fdoc] = flag
     end
+    opts.on "--mem-prof=MODE", "Enable MemoryProf for your examples" do |flag|
+      options[:mem_prof_mode] = flag
+    end
+    opts.on "--mem-prof-top-count=N", "Limits MemoryProf results with N groups/examples" do |flag|
+      options[:mem_prof_top_count] = flag
+    end
   end
 
   def self.plugin_test_prof_init(options)
     options = TestProf.configure_options(options)
 
     reporter << TestProf::EventProfReporter.new(options[:io], options) if options[:event]
     reporter << TestProf::FactoryDoctorReporter.new(options[:io], options) if options[:fdoc]
+    reporter << TestProf::MemoryProfReporter.new(options[:io], options) if options[:mem_prof_mode]
 
     ::TestProf::MinitestSample.call if options[:sample]
   end

lib/test_prof.rb

@@ -8,6 +8,7 @@
 require "test_prof/event_prof"
 require "test_prof/factory_doctor"
 require "test_prof/factory_prof"
+require "test_prof/memory_prof"
 require "test_prof/rspec_stamp"
 require "test_prof/tag_prof"
 require "test_prof/rspec_dissect" if TestProf.rspec?

lib/test_prof/memory_prof.rb

@@ -0,0 +1,78 @@
+# frozen_string_literal: true
+
+require "test_prof/memory_prof/tracker"
+require "test_prof/memory_prof/printer"
+
+module TestProf
+  # MemoryProf can help in detecting test examples causing memory spikes.
+  # It supports two metrics: RSS and allocations.
+  #
+  # Example:
+  #
+  #   TEST_MEM_PROF='rss' rspec ...
+  #   TEST_MEM_PROF='alloc' rspec ...
+  #
+  # By default MemoryProf shows the top 5 examples and groups (for RSpec) but you can
+  # set how many items to display with `TEST_MEM_PROF_COUNT`:
+  #
+  #   TEST_MEM_PROF='rss' TEST_MEM_PROF_COUNT=10 rspec ...
+  #
+  # The examples block shows the amount of memory used by each example, and the groups
+  # block displays the memory allocated by other code defined in the groups. For example,
+  # RSpec groups may include heavy `before(:all)` (or `before_all`) setup blocks, so it is
+  # helpful to see which groups use the most amount of memory outside of their examples.
+
+  module MemoryProf
+    # MemoryProf configuration
+    class Configuration
+      attr_reader :mode, :top_count
+
+      def initialize
+        self.mode = ENV["TEST_MEM_PROF"]
+        self.top_count = ENV["TEST_MEM_PROF_COUNT"]
+      end
+
+      def mode=(value)
+        @mode = (value == "alloc") ? :alloc : :rss
+      end
+
+      def top_count=(value)
+        @top_count = value.to_i
+        @top_count = 5 unless @top_count.positive?
+      end
+    end
+
+    class << self
+      TRACKERS = {
+        alloc: AllocTracker,
+        rss: RssTracker
+      }.freeze
+
+      PRINTERS = {
+        alloc: AllocPrinter,
+        rss: RssPrinter
+      }.freeze
+
+      def config
+        @config ||= Configuration.new
+      end
+
+      def configure
+        yield config
+      end
+
+      def tracker
+        tracker = TRACKERS[config.mode]
+        tracker.new(config.top_count)
+      end
+
+      def printer(tracker)
+        printer = PRINTERS[config.mode]
+        printer.new(tracker)
+      end
+    end
+  end
+end
+
+require "test_prof/memory_prof/rspec" if TestProf.rspec?
+require "test_prof/memory_prof/minitest" if TestProf.minitest?

lib/test_prof/memory_prof/minitest.rb

@@ -0,0 +1,56 @@
+# frozen_string_literal: true
+
+require "minitest/base_reporter"
+
+module Minitest
+  module TestProf
+    class MemoryProfReporter < BaseReporter # :nodoc:
+      attr_reader :tracker, :printer, :current_example
+
+      def initialize(io = $stdout, options = {})
+        super
+
+        configure_profiler(options)
+
+        @tracker = ::TestProf::MemoryProf.tracker
+        @printer = ::TestProf::MemoryProf.printer(tracker)
+
+        @current_example = nil
+      end
+
+      def prerecord(group, example)
+        set_current_example(group, example)
+        tracker.example_started(current_example)
+      end
+
+      def record(example)
+        tracker.example_finished(current_example)
+      end
+
+      def start
+        tracker.start
+      end
+
+      def report
+        tracker.finish
+        printer.print
+      end
+
+      private
+
+      def set_current_example(group, example)
+        @current_example = {
+          name: example.gsub(/^test_(?:\d+_)?/, ""),
+          location: location_with_line_number(group, example)
+        }
+      end
+
+      def configure_profiler(options)
+        ::TestProf::MemoryProf.configure do |config|
+          config.mode = options[:mem_prof_mode]
+          config.top_count = options[:mem_prof_top_count] if options[:mem_prof_top_count]
+        end
+      end
+    end
+  end
+end

lib/test_prof/memory_prof/printer.rb

@@ -0,0 +1,93 @@
+# frozen_string_literal: true
+
+require "test_prof/memory_prof/printer/number_to_human"
+require "test_prof/ext/string_truncate"
+
+module TestProf
+  module MemoryProf
+    class Printer
+      include Logging
+      using StringTruncate
+
+      def initialize(tracker)
+        @tracker = tracker
+      end
+
+      def print
+        messages = [
+          "MemoryProf results\n\n",
+          print_total,
+          print_block("groups", tracker.groups),
+          print_block("examples", tracker.examples)
+        ]
+
+        log :info, messages.join
+      end
+
+      private
+
+      attr_reader :tracker
+
+      def print_block(name, items)
+        return if items.empty?
+
+        <<~GROUP
+          Top #{tracker.top_count} #{name} (by #{mode}):
+
+          #{print_items(items)}
+        GROUP
+      end
+
+      def print_items(items)
+        messages =
+          items.map do |item|
+            <<~ITEM
+              #{item[:name].truncate(30)} (#{item[:location]}) – +#{memory_amount(item)} (#{memory_percentage(item)}%)
+            ITEM
+          end
+
+        messages.join
+      end
+
+      def memory_percentage(item)
+        (100.0 * item[:memory] / tracker.total_memory).round(2)
+      end
+
+      def number_to_human(value)
+        NumberToHuman.convert(value)
+      end
+    end
+
+    class AllocPrinter < Printer
+      private
+
+      def mode
+        "allocations"
+      end
+
+      def print_total
+        "Total allocations: #{tracker.total_memory}\n\n"
+      end
+
+      def memory_amount(item)
+        item[:memory]
+      end
+    end
+
+    class RssPrinter < Printer
+      private
+
+      def mode
+        "RSS"
+      end
+
+      def print_total
+        "Final RSS: #{number_to_human(tracker.total_memory)}\n\n"
+      end
+
+      def memory_amount(item)
+        number_to_human(item[:memory])
+      end
+    end
+  end
+end