+
+
+
+TestProf はテストスイートの性能を分析するための様々なツールの詰め合わせです。
+
+どうしてテストスイートの性能が重要なのでしょうか？
+第一に、テストは開発者のためのフィードバックループの一部です。(参考: [@searls](https://github.com/searls) [talk](https://vimeo.com/145917204))
+そして第二に、テストはデプロイサイクルの一部でもあります。
+
+はっきりと言うと、遅いテストは時間の無駄であり、あなたの活動を非生産的なものにします。
+
+TestProf ツールボックスは以下のようなツールを備えており、テストスイートに含まれるボトルネックを特定するのに役立つでしょう。
+
+- 一般的な Ruby プロファイラのための プラグ・アンド・プレイ ([`ruby-prof`](https://github.com/ruby-prof/ruby-prof), [`stackprof`](https://github.com/tmm1/stackprof))
+
+- Factory の使用状況についての分析器とプロファイラ
+
+- ActiveSupport を活用したプロファイラ
+
+- より速いテストを書くための、RSpec と minitest のための[ヘルパー](#recipes)
+
+- RuboCop で使える Cop (検査ルール)
+
+- その他色々
+
+📑 [ドキュメント](https://test-prof.evilmartians.io)
+
+
+
+
+
+
+
+ 
+
+
+ スポンサー
+
-* レシピー
- * [`before_all`](/recipes/before_all.md)
- * [`let_it_be`](/recipes/let_it_be.md)
- * [AnyFixture](/recipes/any_fixture.md)
- * [FactoryDefault](/recipes/factory_default.md)
- * [FactoryAllStub](/recipes/factory_all_stub.md)
- * [RSpec Stamp](/recipes/rspec_stamp.md)
- * [テストのサンプリング](/recipes/tests_sampling.md)
- * [Railsのログ出力](/recipes/logging.md)
+* プロファイラ
+ * [Ruby profilers](/profilers/ruby_profilers.md)
+ * [Event Profiler](/profilers/event_prof.md)
+ * [Tag Profiler](/profilers/tag_prof.md)
+ * [Factory Doctor](/profilers/factory_doctor.md)
+ * [Factory Profiler](/profilers/factory_prof.md)
+ * [RSpecDissect Profiler](/profilers/rspec_dissect.md)
+ * [Memory Profiler](/profilers/memory_prof.md)
+
+* レシピ
+ * [`before_all` Hook](/recipes/before_all.md)
+ * [`let_it_be` Helper](/recipes/let_it_be.md)
+ * [AnyFixture](/recipes/any_fixture.md)
+ * [FactoryDefault](/recipes/factory_default.md)
+ * [FactoryAllStub](/recipes/factory_all_stub.md)
+ * [RSpec スタンプ](/recipes/rspec_stamp.md)
+ * [テストのサンプリング](/recipes/tests_sampling.md)
+ * [Rails のロギング](/recipes/logging.md)
* その他
- * [RuboCopのコップ](/misc/rubocop.md)
+ * [RuboCop の検査ルール](/misc/rubocop.md)
diff --git a/docs/assets/factory-flame.gif b/docs/assets/factory-flame.gif
new file mode 100644
index 0000000..5349557
Binary files /dev/null and b/docs/assets/factory-flame.gif differ
diff --git a/docs/assets/images/coggle.png b/docs/assets/images/coggle.png
new file mode 100644
index 0000000..422071f
Binary files /dev/null and b/docs/assets/images/coggle.png differ
diff --git a/docs/assets/images/logo.svg b/docs/assets/images/logo.svg
new file mode 100644
index 0000000..c412abd
--- /dev/null
+++ b/docs/assets/images/logo.svg
@@ -0,0 +1 @@
+
diff --git a/docs/assets/images/testprof.png b/docs/assets/images/testprof.png
new file mode 100644
index 0000000..b2ad48b
Binary files /dev/null and b/docs/assets/images/testprof.png differ
diff --git a/docs/assets/tag-prof.gif b/docs/assets/tag-prof.gif
new file mode 100644
index 0000000..8980c98
Binary files /dev/null and b/docs/assets/tag-prof.gif differ
diff --git a/docs/getting_started.md b/docs/getting_started.md
index 315cd78..4d514bc 100644
--- a/docs/getting_started.md
+++ b/docs/getting_started.md
@@ -1,67 +1,55 @@
-# 入門
-
-## 前提条件
-
-対応しているRubyのバージョン
-
-* Ruby (MRI) >= 2.5.0
- * Ruby 2.2の場合は、TestProf < 0.7.0、
- * Ruby 2.3の場合は、TestProf ~> 0.7.0、
- * Ruby 2.4の場合は、TestProf < 0.12.0 を使用してください。
-
-* JRuby >= 9.1.0.0（一部のツールはバージョン 9.2.7+ が必要）
-
-RSpecの場合は、対応のバージョンは >=3.5.0 です。もっと古いRSpecには TestProf < 0.8.0 が必要です。
+# はじめに
## インストール
-ジェム「`test-prof`」を追加してください。
+`test-prof` gem をアプリケーションに追加してください。
```ruby
group :test do
- gem "test-prof"
+ gem "test-prof", "~> 1.0"
end
```
-これでインストールが終わりです!
+これだけで完了です。TestProfを使用する準備ができました！ [プロファイル](/#profilers).
## 設定
-TestProfは、全ツールで使用されるいくつかのグローバル設定があります。
+TestProf には、全ツールで使用されるグローバル設定がいくつかあります。
```ruby
TestProf.configure do |config|
# レポートなどを保存するフォルダー (デフォルトは'tmp/test_prof')
config.output_dir = "tmp/test_prof"
- # レポートファイルの名前ににタイムスタンプを付ける
+ # レポートに対し一意なファイル名を付与する(単に、現在のタイムスタンプを追加する)
config.timestamps = true
- # 出力をハイライトする
+ # 色付きで出力する
config.color = true
- # ログ出力の宛先（ファイルまたはSTDOUT）
+ # ログの出力先 (デフォルト)
config.output = $stdout
- # カスタムのロガーインスタンスを指定することもできます
+ # あるいは、カスタムのロガーインスタンスを指定することもできます
config.logger = MyLogger.new
end
```
-### レポート区別用の識別子
-
-また、「`TEST_PROF_REPORT`」という環境変数を使用して、レポート名に識別子を追加することができます。これは、異なるセットアップでのレポートを比較したい場合に便利です。
+また、「`TEST_PROF_REPORT`」という環境変数を使用して、レポート名に識別子を追加することができます。
+これは、異なるセットアップ間でレポートを比較したい場合に役立ちます。
**例：** `bootsnap`を使う場合と使わない場合のロード時間を[`stackprof`](./profilers/stack_prof.md)で比較してみましょう。
```sh
-# 一番目のレポートに、「-with-bootsnap」を付けます
+# 一番目のレポートに、接頭語「-with-bootsnap」を付けます
$ TEST_STACK_PROF=boot TEST_PROF_REPORT=with-bootsnap bundle exec rake
$ #=> StackProf report generated: tmp/test_prof/stack-prof-report-wall-raw-boot-with-bootsnap.dump
-# 二番目のレポートは、bootnapを無効にし、名前に「-no-bootsnap」を付けて作成します
+
+# bootsnapを無効にし、二番目のレポートを作成したい
+# Assume that you disabled bootsnap and want to generate a new report
$ TEST_STACK_PROF=boot TEST_PROF_REPORT=no-bootsnap bundle exec rake
$ #=> StackProf report generated: tmp/test_prof/stack-prof-report-wall-raw-boot-no-bootsnap.dump
```
-これで、分かりやすい名前の2つのレポートができました。
+これで、分かりやすい名前のレポートが二つできました。
diff --git a/docs/misc/rubocop.md b/docs/misc/rubocop.md
new file mode 100644
index 0000000..14800e8
--- /dev/null
+++ b/docs/misc/rubocop.md
@@ -0,0 +1,57 @@
+# Custom RuboCop Cops
+
+TestProf comes with the [RuboCop](https://github.com/bbatsov/rubocop) cops that help you write more performant tests.
+
+To enable them, require `test_prof/rubocop` in your RuboCop configuration:
+
+```yml
+# .rubocop.yml
+require:
+ - 'test_prof/rubocop'
+```
+
+To configure cops to your needs:
+
+```yml
+RSpec/AggregateExamples:
+ AddAggregateFailuresMetadata: false
+```
+
+Or you can just require it dynamically:
+
+```sh
+bundle exec rubocop -r 'test_prof/rubocop' --only RSpec/AggregateExamples
+```
+
+## RSpec/AggregateExamples
+
+This cop encourages you to use one of the greatest features of the recent RSpec – aggregating failures within an example.
+
+Instead of writing one example per assertion, you can group _independent_ assertions together, thus running all setup hooks only once.
+That can dramatically increase your performance (by reducing the total number of examples).
+
+Consider an example:
+
+```ruby
+# bad
+it { is_expected.to be_success }
+it { is_expected.to have_header("X-TOTAL-PAGES", 10) }
+it { is_expected.to have_header("X-NEXT-PAGE", 2) }
+its(:status) { is_expected.to eq(200) }
+
+# good
+it "returns the second page", :aggregate_failures do
+ is_expected.to be_success
+ is_expected.to have_header("X-TOTAL-PAGES", 10)
+ is_expected.to have_header("X-NEXT-PAGE", 2)
+ expect(subject.status).to eq(200)
+end
+```
+
+Auto-correction will typically add `:aggregate_failures` to examples, but if your project enables it globally, or selectively by e.g. deriving metadata from file location, you may opt-out of adding it using `AddAggregateFailuresMetadata` config option.
+
+This cop supports auto-correct feature, so you can automatically refactor you legacy tests!
+
+**NOTE**: `its` examples shown here have been deprecated as of RSpec 3, but users of the [rspec-its gem](https://github.com/rspec/rspec-its) can leverage this cop to cut out that dependency.
+
+**NOTE**: auto-correction of examples using block matchers, such as `change` is deliberately not supported.
diff --git a/docs/playbook.md b/docs/playbook.md
new file mode 100644
index 0000000..6dbb9da
--- /dev/null
+++ b/docs/playbook.md
@@ -0,0 +1,203 @@
+# Playbook
+
+This document aims to help you get started with profiling test suites and answers the following questions: which profiles to run first? How do we interpret the results to choose the next steps? Etc.
+
+**NOTE**: This document assumes you're working with a Ruby on Rails application and RSpec testing framework. The ideas can easily be translated into other frameworks.
+
+## Step 0. Configuration basics
+
+Low-hanging configuration fruits:
+
+- Disable logging in tests—it's useless. If you really need it, use our [logging utils](./recipes/logging.md).
+
+```ruby
+config.logger = ActiveSupport::TaggedLogging.new(Logger.new(nil))
+config.log_level = :fatal
+```
+
+- Disable coverage and built-in profiling by default. Use env var to enable it (e.g., `COVERAGE=true`)
+
+## Step 1. General profiling
+
+It helps to identify not-so-low hanging fruits. We recommend using [StackProf](./profilers/stack_prof.md), so you must install it first (if not yet):
+
+```sh
+bundle add stackprof
+```
+
+Configure Test Prof to generate JSON profiles by default:
+
+```ruby
+TestProf::StackProf.configure do |config|
+ config.format = "json"
+end
+```
+
+We recommend using [speedscope](https://www.speedscope.app) to analyze these profiles.
+
+### Step 1.1. Application boot profiling
+
+```sh
+TEST_STACK_PROF=boot rspec ./spec/some_spec.rb
+```
+
+**NOTE:** running a single spec/test is enough for this profiling.
+
+What to look for? Some examples:
+
+- No [Bootsnap](https://github.com/Shopify/bootsnap) used or not configured to cache everything (e.g., YAML files)
+- Slow Rails initializers that are not needed in tests.
+
+### Step 1.2. Sampling tests profiling
+
+The idea is to run a random subset of tests multiple times to reveal some application-wide problems. You must enable the [sampling feature](./recipes/tests_sampling.md) first:
+
+```rb
+# For RSpec in your spec_helper.rb
+require "test_prof/recipes/rspec/sample"
+
+# For Minitest in your test_helper.rb
+require "test_prof/recipes/minitest/sample"
+```
+
+Then run **multiple times** and analyze the obtained flamegraphs:
+
+```sh
+SAMPLE=100 bin/rails test
+# or
+SAMPLE=100 bin/rspec
+```
+
+Common findings:
+
+- Encryption calls (`*crypt*`-whatever): relax the settings in the test env
+- Log calls: are you sure you disabled logs?
+- Databases: maybe there are some low-hanging fruits (like using DatabaseCleaner truncation for every test instead of transactions)
+- Network: should not be there for unit tests, inevitable for browser tests; use [Webmock](https://github.com/bblimke/webmock) to disable HTTP calls completely.
+
+## Step 2. Narrow down the scope
+
+This is an important step for large codebases. We must prioritize quick fixes that bring the most value (time reduction) over dealing with complex, slow tests individually (even if they're the slowest ones). For that, we first identify the **types of tests** contributing the most to the overall run time.
+
+We use [TagProf](./profilers/tag_prof.md) for that:
+
+```sh
+TAG_PROF=type TAG_PROF_FORMAT=html TAG_PROF_EVENT=sql.active_record,factory.create bin/rspec
+```
+
+Looking at the generated diagram, you can identify the two most time-consuming test types (usually models and/or controllers among them).
+
+We assume that it's easier to find a common slowness cause for the whole group and fix it than dealing with individual tests. Given that assumption, we continue the process only within the selected group (let's say, models).
+
+## Step 3. Specialized profiling
+
+Within the selected group, we can first perform quick event-based profiling via [EventProf](./profilers/event_prof.md). (Maybe, with sampling enabled as well).
+
+### Step 3.1. Dependencies configuration
+
+At this point, we may identify some misconfigured or misused dependencies/gems. Common examples:
+
+- Inlined Sidekiq jobs:
+
+```sh
+EVENT_PROF=sidekiq.inline bin/rspec spec/models
+```
+
+- Wisper broadcasts ([patch required](https://gist.github.com/palkan/aa7035cebaeca7ed76e433981f90c07b)):
+
+```sh
+EVENT_PROF=wisper.publisher.broadcast bin/rspec spec/models
+```
+
+- PaperTrail logs creation:
+
+Enable custom profiling:
+
+```rb
+TestProf::EventProf.monitor(PaperTrail::RecordTrail, "paper_trail.record", :record_create)
+TestProf::EventProf.monitor(PaperTrail::RecordTrail, "paper_trail.record", :record_destroy)
+TestProf::EventProf.monitor(PaperTrail::RecordTrail, "paper_trail.record", :record_update)
+```
+
+Run tests:
+
+```sh
+EVENT_PROF=paper_trail.record bin/rspec spec/models
+```
+
+See [the Sidekiq example](https://evilmartians.com/chronicles/testprof-a-good-doctor-for-slow-ruby-tests#background-jobs) on how to quickly fix such problems using [RSpecStamp](./recipes/rspec_stamp.md).
+
+### Step 3.2. Data generation
+
+Identify the slowest tests based on the amount of time spent in the database or factories (if any):
+
+```sh
+# Database interactions
+EVENT_PROF=sql.active_record bin/rspec spec/models
+
+# Factories
+EVENT_PROF=factory.create bin/rspec spec/models
+```
+
+Now, we can narrow our scope further to the top 10 files from the generated reports. If you use factories, use the `factory.create` report.
+
+**TIP:** In RSpec, you can mark the slowest examples with a custom tag automatically using the following command:
+
+```sh
+EVENT_PROF=factory.create EVEN_PROF_STAMP=slow:factory bin/rspec spec/models
+```
+
+## Step 4. Factories usage
+
+Identify the most used factories among the `slow:factory` tests:
+
+```sh
+FPROF=1 bin/rspec --tag slow:factory
+```
+
+If you see some factories used much more times than the total number of examples, you deal with _factory cascades_.
+
+Visualize the cascades:
+
+```sh
+FPROF=flamegraph bin/rspec --tag slow:factory
+```
+
+The visualization should help to identify the factories to be fixed. You find possible solutions in [this post](https://evilmartians.com/chronicles/testprof-2-factory-therapy-for-your-ruby-tests-rspec-minitest).
+
+### Step 4.1. Factory defaults
+
+One option to fix cascades produced by model associations is to use [factory defaults](./recipes/factory_default.md). To estimate the potential impact and identify factories to apply this pattern to, run the following profiler:
+
+```sh
+FACTORY_DEFAULT_PROF=1 bin/rspec --tag slow:factory
+```
+
+Try adding `create_default` and measure the impact:
+
+```sh
+FACTORY_DEFAULT_SUMMARY=1 bin/rspec --tag slow:factory
+
+# More hits — better
+FactoryDefault summary: hit=11 miss=3
+```
+
+### Step 4.2. Factory fixtures
+
+Back to the `FPROF=1` report, see if you have some records created for every example (typically, `user`, `account`, `team`). Consider replacing them with fixtures using [AnyFixture](./recipes/any_fixture.md).
+
+## Step 5. Reusable setup
+
+It's common to have the same setup shared across multiple examples. You can measure the time spent in `let` / `before` compared to the actual example time using [RSpecDissect](./profilers/rspec_dissect.md):
+
+```sh
+RD_PROF=1 bin/rspec
+```
+
+Take a look at the slowest groups and try to replace `let`/`let!` with [let_it_be](./recipes/let_it_be.md) and `before` with [before_all](./recipes/before_all.md).
+
+**IMPORTANT:** Knapsack Pro users must be aware that per-example balancing eliminates the positive effect of using `let_it_be` / `before_all`. You must switch to per-file balancing while at the same time keeping your files small—that's how you can maximize the effect of using Test Prof optimizations.
+
+## Conclusion
+
+After applying the steps above to a given group of tests, you should develop the patterns and techniques optimized for your codebase. Then, all you need is to extrapolate them to other groups. Good luck!
diff --git a/docs/profilers/event_prof.md b/docs/profilers/event_prof.md
new file mode 100644
index 0000000..bacaee0
--- /dev/null
+++ b/docs/profilers/event_prof.md
@@ -0,0 +1,244 @@
+# EventProf
+
+EventProf collects instrumentation (such as ActiveSupport::Notifications) metrics during your test suite run.
+
+It works very similar to `rspec --profile` but can track arbitrary events.
+
+Example output:
+
+```sh
+[TEST PROF INFO] EventProf results for sql.active_record
+
+Total time: 00:00.256 of 00:00.512 (50.00%)
+Total events: 1031
+
+Top 5 slowest suites (by time):
+
+AnswersController (./spec/controllers/answers_controller_spec.rb:3) – 00:00.119 (549 / 20) of 00:00.200 (59.50%)
+QuestionsController (./spec/controllers/questions_controller_spec.rb:3) – 00:00.105 (360 / 18) of 00:00.125 (84.00%)
+CommentsController (./spec/controllers/comments_controller_spec.rb:3) – 00:00.032 (122 / 4) of 00:00.064 (50.00%)
+
+Top 5 slowest tests (by time):
+
+destroys question (./spec/controllers/questions_controller_spec.rb:38) – 00:00.022 (29) of 00:00.064 (34.38%)
+change comments count (./spec/controllers/comments_controller_spec.rb:7) – 00:00.011 (34) of 00:00.022 (50.00%)
+change Votes count (./spec/shared_examples/controllers/voted_examples.rb:23) – 00:00.008 (25) of 00:00.022 (36.36%)
+change Votes count (./spec/shared_examples/controllers/voted_examples.rb:23) – 00:00.008 (32) of 00:00.035 (22.86%)
+fails (./spec/shared_examples/controllers/invalid_examples.rb:3) – 00:00.007 (34) of 00:00.014 (50.00%)
+
+```
+
+## Instructions
+
+Currently, EventProf supports only ActiveSupport::Notifications
+
+To activate EventProf with:
+
+### RSpec
+
+Use `EVENT_PROF` environment variable set to event name:
+
+```sh
+# Collect SQL queries stats for every suite and example
+EVENT_PROF='sql.active_record' rspec ...
+```
+
+You can track multiple events simultaneously:
+
+```sh
+EVENT_PROF='sql.active_record,perform.active_job' rspec ...
+```
+
+### Minitest
+
+Use `EVENT_PROF` environment variable set to event name:
+
+```sh
+# Collect SQL queries stats for every suite and example
+EVENT_PROF='sql.active_record' rake test
+```
+
+or use CLI options as well:
+
+```sh
+# Run a specific file using CLI option
+ruby test/my_super_test.rb --event-prof=sql.active_record
+
+# Show the list of possible options:
+ruby test/my_super_test.rb --help
+```
+
+### Using with Minitest::Reporters
+
+If you're using `Minitest::Reporters` in your project you have to explicitly declare it
+in your test helper file:
+
+```sh
+require 'minitest/reporters'
+Minitest::Reporters.use! [YOUR_FAVORITE_REPORTERS]
+```
+
+#### NOTICE
+
+When you have `minitest-reporters` installed as a gem but not declared in your `Gemfile`
+make sure to always prepend your test run command with `bundle exec` (but we sure that you always do it).
+Otherwise, you'll get an error caused by Minitest plugin system, which scans all the entries in the
+`$LOAD_PATH` for any `minitest/*_plugin.rb`, thus initialization of `minitest-reporters` plugin which is
+available in that case doesn't happens correctly.
+
+See [Rails guides](http://guides.rubyonrails.org/active_support_instrumentation.html)
+for the list of available events if you're using Rails.
+
+If you're using [rom-rb](http://rom-rb.org) you might be interested in profiling `'sql.rom'` event.
+
+## Configuration
+
+By default, EventProf collects information only about top-level groups (aka suites),
+but you can also profile individual examples. Just set the configuration option:
+
+```ruby
+TestProf::EventProf.configure do |config|
+ config.per_example = true
+end
+```
+
+Or provide the `EVENT_PROF_EXAMPLES=1` env variable.
+
+Another useful configuration parameter – `rank_by`. It's responsible for sorting stats –
+either by the time spent in the event or by the number of occurrences:
+
+```sh
+EVENT_PROF_RANK=count EVENT_PROF='instantiation.active_record' be rspec
+```
+
+See [event_prof.rb](https://github.com/test-prof/test-prof/tree/master/lib/test_prof/event_prof.rb) for all available configuration options and their usage.
+
+## Using with RSpecStamp
+
+EventProf can be used with [RSpec Stamp](../recipes/rspec_stamp.md) to automatically mark _slow_ examples with custom tags. For example:
+
+```sh
+EVENT_PROF="sql.active_record" EVENT_PROF_STAMP="slow:sql" rspec ...
+```
+
+After running the command above the slowest example groups (and examples if configured) would be marked with the `slow: :sql` tag.
+
+## Custom Instrumentation
+
+To use EventProf with your instrumentation engine just complete the two following steps:
+
+- Add a wrapper for your instrumentation:
+
+```ruby
+# Wrapper over your instrumentation
+module MyEventsWrapper
+ # Should contain the only one method
+ def self.subscribe(event)
+ raise ArgumentError, "Block is required!" unless block_given?
+
+ ::MyEvents.subscribe(event) do |start, finish, *|
+ yield (finish - start)
+ end
+ end
+end
+```
+
+- Set instrumenter in the config:
+
+```ruby
+TestProf::EventProf.configure do |config|
+ config.instrumenter = MyEventsWrapper
+end
+```
+
+## Custom Events
+
+### `"factory.create"`
+
+FactoryGirl provides its own instrumentation ('factory_girl.run_factory'); but there is a caveat – it fires an event every time a factory is used, even when we use factory for nested associations. Thus it's not possible to calculate the total time spent in factories due to the double calculation.
+
+EventProf comes with a little patch for FactoryGirl which provides instrumentation only for top-level `FactoryGirl.create` calls. It is loaded automatically if you use `"factory.create"` event:
+
+```sh
+EVENT_PROF=factory.create bundle exec rspec
+```
+
+Also supports Fabrication (tracks implicit and explicit `Fabricate.create` calls).
+
+### `"sidekiq.jobs"`
+
+Collects statistics about Sidekiq jobs that have been run inline:
+
+```sh
+EVENT_PROF=sidekiq.jobs bundle exec rspec
+```
+
+**NOTE**: automatically sets `rank_by` to `count` ('cause it doesn't make sense to collect the information about time spent – see below).
+
+### `"sidekiq.inline"`
+
+Collects statistics about Sidekiq jobs that have been run inline (excluding nested jobs):
+
+```sh
+EVENT_PROF=sidekiq.inline bundle exec rspec
+```
+
+Use this event to profile the time spent running Sidekiq jobs.
+
+## Profile arbitrary methods
+
+You can also add your custom events to profile specific methods (for example, after figuring out some hot calls with [RubyProf](./ruby_prof.md) or [StackProf](./stack_prof.md)).
+
+For example, having a class doing some heavy work:
+
+```ruby
+class Work
+ def do_smth(*args)
+ # do something
+ end
+end
+```
+
+You can profile it by adding a _monitor_:
+
+```ruby
+# provide a class, event name and methods to monitor
+TestProf::EventProf.monitor(Work, "my.work", :do_smth)
+```
+
+And then run EventProf as usual:
+
+```sh
+EVENT_PROF=my.work bundle exec rake test
+```
+
+You can also provide additional options:
+
+- `top_level: true | false` (defaults to `false`): defines whether you want to take into account only top-level invocations and ignore nested triggers of this event (that's how "factory.create" is [implemented](https://github.com/test-prof/test-prof/blob/master/lib/test_prof/event_prof/custom_events/factory_create.rb))
+- `guard: Proc` (defaults to `nil`): provide a Proc which could prevent from triggering an event: the method is instrumented only if `guard` returns `true`; `guard` is executed using `instance_exec` and the method arguments are passed to it.
+
+For example:
+
+```ruby
+TestProf::EventProf.monitor(
+ Sidekiq::Client,
+ "sidekiq.inline",
+ :raw_push,
+ top_level: true,
+ guard: ->(*) { Sidekiq::Testing.inline? }
+)
+```
+
+You can add monitors _on demand_ (i.e. only when you want to track the specified event) by wrapping
+the code in `TestProf::EventProf::CustomEvents.register` method:
+
+```ruby
+TestProf::EventProf::CustomEvents.register("my.work") do
+ TestProf::EventProf.monitor(Work, "my.work", :do_smth)
+end
+
+# Then call `activate_all` with the provided event
+TestProf::EventProf::CustomEvents.activate_all(TestProf::EventProf.config.event)
+```
+
+The block is evaluated only if the specified event is enabled with EventProf.
diff --git a/docs/profilers/factory_doctor.md b/docs/profilers/factory_doctor.md
new file mode 100644
index 0000000..5a23af9
--- /dev/null
+++ b/docs/profilers/factory_doctor.md
@@ -0,0 +1,153 @@
+# Factory Doctor
+
+One common bad pattern that slows our tests down is unnecessary database manipulation. Consider a _bad_ example:
+
+```ruby
+# with FactoryBot/FactoryGirl
+it "validates name presence" do
+ user = create(:user)
+ user.name = ""
+ expect(user).not_to be_valid
+end
+
+# with Fabrication
+it "validates name presence" do
+ user = Fabricate(:user)
+ user.name = ""
+ expect(user).not_to be_valid
+end
+```
+
+Here we create a new user record, run all callbacks and validations and save it to the database. We don't need all these! Here is a _good_ example:
+
+```ruby
+# with FactoryBot/FactoryGirl
+it "validates name presence" do
+ user = build_stubbed(:user)
+ user.name = ""
+ expect(user).not_to be_valid
+end
+
+# with Fabrication
+it "validates name presence" do
+ user = Fabricate.build(:user)
+ user.name = ""
+ expect(user).not_to be_valid
+end
+```
+
+Read more about [`build_stubbed`](https://robots.thoughtbot.com/use-factory-girls-build-stubbed-for-a-faster-test).
+
+FactoryDoctor is a tool that helps you identify such _bad_ tests, i.e. tests that perform unnecessary database queries.
+
+Example output:
+
+```sh
+[TEST PROF INFO] FactoryDoctor report
+
+Total (potentially) bad examples: 2
+Total wasted time: 00:13.165
+
+User (./spec/models/user_spec.rb:3) (3 records created, 00:00.628)
+ validates name (./spec/user_spec.rb:8) – 1 record created, 00:00.114
+ validates email (./spec/user_spec.rb:8) – 2 records created, 00:00.514
+```
+
+**NOTE**: have you noticed the "potentially" word? Unfortunately, FactoryDoctor is not a
+magician (it's still learning) and sometimes it produces false negatives and false positives too.
+
+Please, submit an [issue](https://github.com/test-prof/test-prof/issues) if you found a case which makes FactoryDoctor fail.
+
+You can also tell FactoryDoctor to ignore specific examples/groups. Just add the `:fd_ignore` tag to it:
+
+```ruby
+# won't be reported as offense
+it "is ignored", :fd_ignore do
+ user = create(:user)
+ user.name = ""
+ expect(user).not_to be_valid
+end
+```
+
+## Instructions
+
+FactoryDoctor supports:
+
+- FactoryGirl/FactoryBot
+- Fabrication.
+
+### RSpec
+
+To activate FactoryDoctor use `FDOC` environment variable:
+
+```sh
+FDOC=1 rspec ...
+```
+
+### Using with RSpecStamp
+
+FactoryDoctor can be used with [RSpec Stamp](../recipes/rspec_stamp.md) to automatically mark _bad_ examples with custom tags. For example:
+
+```sh
+FDOC=1 FDOC_STAMP="fdoc:consider" rspec ...
+```
+
+After running the command above all _potentially_ bad examples would be marked with the `fdoc: :consider` tag.
+
+### Minitest
+
+To activate FactoryDoctor use `FDOC` environment variable:
+
+```sh
+FDOC=1 ruby ...
+```
+
+or use CLI option as shown below:
+
+```sh
+ruby ... --factory-doctor
+```
+
+The same option to force Factory Doctor to ignore specific examples is also available for Minitest.
+Just use `fd_ignore` inside your example:
+
+```ruby
+# won't be reported as offense
+it "is ignored" do
+ fd_ignore
+
+ @user.name = ""
+ refute @user.valid?
+end
+```
+
+### Using with Minitest::Reporters
+
+If you're using `Minitest::Reporters` in your project you have to explicitly declare it
+in your test helper file:
+
+```sh
+require 'minitest/reporters'
+Minitest::Reporters.use! [YOUR_FAVORITE_REPORTERS]
+```
+
+**NOTE**: When you have `minitest-reporters` installed as a gem but not declared in your `Gemfile`
+make sure to always prepend your test run command with `bundle exec` (but we sure that you always do it).
+Otherwise, you'll get an error caused by Minitest plugin system, which scans all the entries in the
+`$LOAD_PATH` for any `minitest/*_plugin.rb`, thus initialization of `minitest-reporters` plugin which is
+available in that case doesn't happens correctly.
+
+## Configuration
+
+The following configuration parameters are available (showing defaults):
+
+```ruby
+TestProf::FactoryDoctor.configure do |config|
+ # Which event to track within test example to consider them "DB-dirty"
+ config.event = "sql.active_record"
+ # Consider result "good" if the time in DB is less then the threshold
+ config.threshold = 0.01
+end
+```
+
+You can use the corresponding env variables as well: `FDOC_EVENT` and `FDOC_THRESHOLD`.
diff --git a/docs/profilers/factory_prof.md b/docs/profilers/factory_prof.md
new file mode 100644
index 0000000..42990fc
--- /dev/null
+++ b/docs/profilers/factory_prof.md
@@ -0,0 +1,126 @@
+# FactoryProf
+
+FactoryProf tracks your factories usage statistics, i.e. how often each factory has been used.
+
+Example output:
+
+```sh
+[TEST PROF INFO] Factories usage
+
+Total: 15285
+Total top-level: 10286
+Total time: 04:31.222 (out of 07.16.124)
+Total uniq factories: 119
+
+ total top-level total time time per call top-level time name
+ 6091 2715 115.7671s 0.0426s 50.2517s user
+ 2142 2098 93.3152s 0.0444s 92.1915s post
+ ...
+```
+
+It shows both the total number of the factory runs and the number of _top-level_ runs, i.e. not during another factory invocation (e.g. when using associations.)
+
+It also shows the time spent generating records with factories and the amount of time taken per factory call.
+
+**NOTE**: FactoryProf only tracks the database-persisted factories. In case of FactoryGirl/FactoryBot these are the factories provided by using `create` strategy. In case of Fabrication - objects that created using `create` method.
+
+## Instructions
+
+FactoryProf can be used with FactoryGirl/FactoryBot or Fabrication - application can be bundled with both gems at the same time.
+
+To activate FactoryProf use `FPROF` environment variable:
+
+```sh
+# Simple profiler
+FPROF=1 rspec
+
+# or
+FPROF=1 bundle exec rake test
+```
+
+### [_Nate Heckler_](https://twitter.com/nateberkopec/status/1389945187766456333) mode
+
+To encourage you to fix your factories as soon as possible, we also have a special _Nate heckler_ mode.
+
+Drop this into your `rails_helper.rb` or `test_helper.rb`:
+
+```ruby
+require "test_prof/factory_prof/nate_heckler"
+```
+
+And for every test run see the overall factories usage:
+
+```sh
+[TEST PROF INFO] Time spent in factories: 04:31.222 (54% of total time)
+```
+
+### Exporting profile results to a JSON file
+
+FactoryProf can save profile results as a JSON file.
+
+To use this feature, set the `FPROF` environment variable to `json`:
+
+```sh
+FPROF=json rspec
+
+# or
+FPROF=json bundle exec rake test
+```
+
+Example output:
+
+```
+[TEST PROF INFO] Profile results to JSON: tmp/test_prof/test-prof.result.json
+```
+
+## Factory Flamegraph
+
+The most useful feature of FactoryProf is the _FactoryFlame_ report. That's the special interpretation of Brendan Gregg's [flame graphs](http://www.brendangregg.com/flamegraphs.html) which allows you to identify _factory cascades_.
+
+To generate FactoryFlame report set `FPROF` environment variable to `flamegraph`:
+
+```sh
+FPROF=flamegraph rspec
+
+# or
+FPROF=flamegraph bundle exec rake test
+```
+
+That's how a report looks like:
+
+
+
+How to read this?
+
+Every column represents a _factory stack_ or _cascade_, that is a sequence of recursive `#create` method calls. Consider an example:
+
+```ruby
+factory :comment do
+ answer
+ author
+end
+
+factory :answer do
+ question
+ author
+end
+
+factory :question do
+ author
+end
+
+create(:comment) #=> creates 5 records
+
+# And the corresponding stack is:
+# [:comment, :answer, :question, :author, :author, :author]
+```
+
+The wider column the more often this stack appears.
+
+The `root` cell shows the total number of `create` calls.
+
+## Acknowledgments
+
+- Thanks to [Martin Spier](https://github.com/spiermar) for [d3-flame-graph](https://github.com/spiermar/d3-flame-graph)
+
+- Thanks to [Sam Saffron](https://github.com/SamSaffron) for his [flame graphs implementation](https://github.com/SamSaffron/flamegraph).
diff --git a/docs/profilers/memory_prof.md b/docs/profilers/memory_prof.md
new file mode 100644
index 0000000..f0bd43d
--- /dev/null
+++ b/docs/profilers/memory_prof.md
@@ -0,0 +1,85 @@
+# MemoryProf
+
+MemoryProf tracks memory usage during your test suite run, and can help to detect test examples and groups that cause memory spikes. Memory profiling supports two metrics: RSS and allocations.
+
+Example output:
+
+```sh
+[TEST PROF INFO] MemoryProf results
+
+Final RSS: 673KB
+
+Top 5 groups (by RSS):
+
+AnswersController (./spec/controllers/answers_controller_spec.rb:3) – +80KB (13.50%)
+QuestionsController (./spec/controllers/questions_controller_spec.rb:3) – +32KB (9.08%)
+CommentsController (./spec/controllers/comments_controller_spec.rb:3) – +16KB (3.27%)
+
+Top 5 examples (by RSS):
+
+destroys question (./spec/controllers/questions_controller_spec.rb:38) – +144KB (24.38%)
+change comments count (./spec/controllers/comments_controller_spec.rb:7) – +120KB (20.00%)
+change Votes count (./spec/shared_examples/controllers/voted_examples.rb:23) – +90KB (16.36%)
+change Votes count (./spec/shared_examples/controllers/voted_examples.rb:23) – +64KB (12.86%)
+fails (./spec/shared_examples/controllers/invalid_examples.rb:3) – +32KB (5.00%)
+```
+
+The examples block shows the amount of memory used by each example, and the groups block displays the memory allocated by other code defined in the groups. For example, RSpec groups may include heavy `before(:all)` (or `before_all`) setup blocks, so it is helpful to see which groups use the most amount of memory outside of their examples.
+
+## Instructions
+
+To activate MemoryProf with:
+
+### RSpec
+
+Use `TEST_MEM_PROF` environment variable to set which metric to use:
+
+```sh
+TEST_MEM_PROF='rss' rspec ...
+TEST_MEM_PROF='alloc' rake rspec ...
+```
+
+### Minitest
+
+Use `TEST_MEM_PROF` environment variable to set which metric to use:
+
+```sh
+TEST_MEM_PROF='rss' rake test
+TEST_MEM_PROF='alloc' rspec ...
+```
+
+or use CLI options as well:
+
+```sh
+# Run a specific file using CLI option
+ruby test/my_super_test.rb --mem-prof=rss
+
+# Show the list of possible options:
+ruby test/my_super_test.rb --help
+```
+
+## Configuration
+
+By default, MemoryProf tracks the top 5 examples and groups that use the largest amount of memory.
+You can set how many examples/groups to display with the option:
+
+```sh
+TEST_MEM_PROF='rss' TEST_MEM_PROF_COUNT=10 rspec ...
+```
+
+or with CLI options for Minitest:
+
+```sh
+# Run a specific file using CLI option
+ruby test/my_super_test.rb --mem-prof=rs --mem-prof-top-count=10
+```
+
+## Supported Ruby Engines & OS
+
+Currently the allocation mode is not supported for JRuby.
+
+Since RSS depends on the OS, MemoryProf uses different tools to retrieve it:
+
+* Linux – `/proc/$pid/statm` file,
+* macOS, Solaris, BSD – `ps`,
+* Windows – `Get-Process`, requires PowerShell to be installed.
diff --git a/docs/profilers/rspec_dissect.md b/docs/profilers/rspec_dissect.md
new file mode 100644
index 0000000..d09d7b2
--- /dev/null
+++ b/docs/profilers/rspec_dissect.md
@@ -0,0 +1,82 @@
+# RSpecDissect
+
+Do you know how much time you spend in `before` hooks? Or in memoization helpers such as `let`? Usually, the most of the whole test suite time.
+
+_RSpecDissect_ provides this kind of information and also shows you the worst example groups. The main purpose of RSpecDissect is to identify these slow groups and refactor them using [`before_all`](../recipes/before_all.md) or [`let_it_be`](../recipes/let_it_be.md) recipes.
+
+Example output:
+
+```sh
+[TEST PROF INFO] RSpecDissect enabled
+
+Total time: 25:14.870
+Total `before(:each)` time: 14:36.482
+Total `let` time: 19:20.259
+
+Top 5 slowest suites (by `before(:each)` time):
+
+Webhooks::DispatchTransition (./spec/services/webhooks/dispatch_transition_spec.rb:3) – 00:29.895 of 00:33.706 (327)
+FunnelsController (./spec/controllers/funnels_controller_spec.rb:3) – 00:22.117 of 00:43.649 (133)
+ApplicantsController (./spec/controllers/applicants_controller_spec.rb:3) – 00:21.220 of 00:41.407 (222)
+BookedSlotsController (./spec/controllers/booked_slots_controller_spec.rb:3) – 00:15.729 of 00:27.893 (50)
+Analytics::Wor...rsion::Summary (./spec/services/analytics/workflow_conversion/summary_spec.rb:3) – 00:15.383 of 00:15.914 (12)
+
+
+Top 5 slowest suites (by `let` time):
+
+FunnelsController (./spec/controllers/funnels_controller_spec.rb:3) – 00:38.532 of 00:43.649 (133)
+ ↳ user – 3
+ ↳ funnel – 2
+ApplicantsController (./spec/controllers/applicants_controller_spec.rb:3) – 00:33.252 of 00:41.407 (222)
+ ↳ user – 10
+ ↳ funnel – 5
+ ↳ applicant – 2
+Webhooks::DispatchTransition (./spec/services/webhooks/dispatch_transition_spec.rb:3) – 00:30.320 of 00:33.706 (327)
+ ↳ user – 30
+BookedSlotsController (./spec/controllers/booked_slots_controller_spec.rb:3) – 00:25.710 of 00:27.893 e(50)
+ ↳ user – 21
+ ↳ stage – 14
+AvailableSlotsController (./spec/controllers/available_slots_controller_spec.rb:3) – 00:18.481 of 00:23.366 (85)
+ ↳ user – 15
+ ↳ stage – 10
+```
+
+As you can see, the `let` profiler also tracks the provides the information on how many times each `let` declarations has been used within a group (shows top-3 by default).
+
+## Instructions
+
+RSpecDissect can only be used with RSpec (which is clear from the name).
+
+To activate RSpecDissect use `RD_PROF` environment variable:
+
+```sh
+RD_PROF=1 rspec ...
+```
+
+You can also specify the number of top slow groups through `RD_PROF_TOP` variable:
+
+```sh
+RD_PROF=1 RD_PROF_TOP=10 rspec ...
+```
+
+You can also track only `let` or `before` usage by specifying `RD_PROF=let` and `RD_PROF=before` respectively.
+
+For `let` profiler you can also specify the number of top `let` declarations to print through `RD_PROF_LET_TOP=10` env var.
+
+To disable `let` stats add:
+
+```ruby
+TestProf::RSpecDissect.configure do |config|
+ config.let_stats_enabled = false
+end
+```
+
+## Using with RSpecStamp
+
+RSpecDissect can be used with [RSpec Stamp](../recipes/rspec_stamp.md) to automatically mark _slow_ examples with custom tags. For example:
+
+```sh
+RD_PROF=1 RD_PROF_STAMP="slow" rspec ...
+```
+
+After running the command above the slowest example groups would be marked with the `:slow` tag.
diff --git a/docs/profilers/ruby_prof.md b/docs/profilers/ruby_prof.md
new file mode 100644
index 0000000..013f6b7
--- /dev/null
+++ b/docs/profilers/ruby_prof.md
@@ -0,0 +1,2 @@
+
+Please follow this link.
diff --git a/docs/profilers/ruby_profilers.md b/docs/profilers/ruby_profilers.md new file mode 100644 index 0000000..d52bd6c --- /dev/null +++ b/docs/profilers/ruby_profilers.md @@ -0,0 +1,233 @@ +# Using with Ruby profilers + +Test Prof allows you to use general Ruby profilers to profile test suites without needing to write any profiling code yourself. +Just install the profiler library and run your tests! + +Supported profilers: + +- [StackProf](#stackprof) +- [Vernier](#vernier) +- [RubyProf](#rubyprof) + +## StackProf + +[StackProf][] is a sampling call-stack profiler for Ruby. + +Make sure you have `stackprof` in your dependencies: + +```ruby +# Gemfile +group :development, :test do + gem "stackprof", ">= 0.2.9", require: false +end +``` + +### Profiling the whole test suite with StackProf + +**NOTE:** It's recommended to use [test sampling](../recipes/tests_sampling.md) to generate smaller profiling reports. + +You can activate StackProf profiling by setting the `TEST_STACK_PROF` env variable: + +```sh +TEST_STACK_PROF=1 bundle exec rake test + +# or for RSpec +TEST_STACK_PROF=1 bundle exec rspec ... +``` + +At the end of the test run, you will see the message from Test Prof including paths to generated reports (raw StackProf format and JSON): + +```sh +... + +[TEST PROF INFO] StackProf report generated: tmp/test_prof/stack-prof-report-wall-raw-total.dump +[TEST PROF INFO] StackProf JSON report generated: tmp/test_prof/stack-prof-report-wall-raw-total.json +``` + +We recommend uploading JSON reports to [Speedscope][] and analyze flamegraphs. Otherwise, feel free to use the `stackprof` CLI +to manipulate the raw report. + +### Profiling individual examples with StackProf + +Test Prof provides a built-in shared context for RSpec to profile examples individually: + +```ruby +it "is doing heavy stuff", :sprof do + # ... +end +``` + +**NOTE:** per-example profiling doesn't work when the global (per-suite) profiling is activated. + +### Profiling application boot with StackProf + +The application boot time could also makes testing slower. Try to profile your boot process with StackProf using the following command: + +```sh +# pick some random spec (1 is enough) +$ TEST_STACK_PROF=boot bundle exec rspec ./spec/some_spec.rb + +... +[TEST PROF INFO] StackProf report generated: tmp/test_prof/stack-prof-report-wall-raw-boot.dump +[TEST PROF INFO] StackProf JSON report generated: tmp/test_prof/stack-prof-report-wall-raw-boot.json +``` + +### StackProf configuration + +You can change StackProf mode (which is `wall` by default) through `TEST_STACK_PROF_MODE` env variable. + +You can also change StackProf interval through `TEST_STACK_PROF_INTERVAL` env variable. +For modes `wall` and `cpu`, `TEST_STACK_PROF_INTERVAL` represents microseconds and will default to 1000 as per `stackprof`. +For mode `object`, `TEST_STACK_PROF_INTERVAL` represents allocations and will default to 1 as per `stackprof`. + +You can disable garbage collection frames by setting `TEST_STACK_PROF_IGNORE_GC` env variable. +Garbage collection time will still be present in the profile but not explicitly marked with +its own frame. + +See [stack_prof.rb](https://github.com/test-prof/test-prof/tree/master/lib/test_prof/stack_prof.rb) for all available configuration options and their usage. + +## Vernier + +[Vernier][] is next generation sampling profiler for Ruby. Give it a try and see if it can help in identifying test peformance bottlenecks! + +Make sure you have `vernier` in your dependencies: + +```ruby +# Gemfile +group :development, :test do + gem "vernier", ">= 0.3.0", require: false +end +``` + +### Profiling the whole test suite with Vernier + +**NOTE:** It's recommended to use [test sampling](../recipes/tests_sampling.md) to generate smaller profiling reports. + +You can activate Verner profiling by setting the `TEST_VERNIER` env variable: + +```sh +TEST_VERNIER=1 bundle exec rake test + +# or for RSpec +TEST_VERNIER=1 bundle exec rspec ... +``` + +At the end of the test run, you will see the message from Test Prof including the path to the generated report: + +```sh +... + +[TEST PROF INFO] Vernier report generated: tmp/test_prof/vernier-report-wall-raw-total.json +``` + +Use the [profile-viewer](https://github.com/tenderlove/profiler/tree/ruby) gem or upload your profiles to [profiler.firefox.com](https://profiler.firefox.com). + +### Profiling individual examples with Vernier + +Test Prof provides a built-in shared context for RSpec to profile examples individually: + +```ruby +it "is doing heavy stuff", :vernier do + # ... +end +``` + +**NOTE:** per-example profiling doesn't work when the global (per-suite) profiling is activated. + +### Profiling application boot with Vernier + +You can also profile your application boot process: + +```sh +# pick some random spec (1 is enough) +TEST_VERNIER=boot bundle exec rspec ./spec/some_spec.rb +``` + +## RubyProf + +Easily integrate the power of [ruby-prof](https://github.com/ruby-prof/ruby-prof) into your test suite. + +Make sure `ruby-prof` is installed: + +```ruby +# Gemfile +group :development, :test do + gem "ruby-prof", ">= 1.4.0", require: false +end +``` + +### Profiling the whole test suite with RubyProf + +**NOTE:** It's highly recommended to use [test sampling](../recipes/tests_sampling.md) to generate smaller profiling reports and avoid slow test runs (RubyProf has a signifact overhead). + +You can activate the global profiling using the environment variable `TEST_RUBY_PROF`: + +```sh +TEST_RUBY_PROF=1 bundle exec rake test + +# or for RSpec +TEST_RUBY_PROF=1 bundle exec rspec ... +``` + +At the end of the test run, you will see the message from Test Prof including paths to generated reports: + +```sh +[TEST PROF INFO] RubyProf report generated: tmp/test_prof/ruby-prof-report-flat-wall-total.txt +``` + +### Profiling individual examples with RubyProf + +TestProf provides a built-in shared context for RSpec to profile examples individually: + +```ruby +it "is doing heavy stuff", :rprof do + # ... +end +``` + +**NOTE:** per-example profiling doesn't work when the global profiling is activated. + +### RubyProf configuration + +The most useful configuration option is `printer` – it allows you to specify a RubyProf [printer](https://github.com/ruby-prof/ruby-prof#printers). + +You can specify a printer through environment variable `TEST_RUBY_PROF`: + +```sh +TEST_RUBY_PROF=call_stack bundle exec rake test +``` + +Or in your code: + +```ruby +TestProf::RubyProf.configure do |config| + config.printer = :call_stack +end +``` + +By default, we use `FlatPrinter`. + +**NOTE:** to specify the printer for per-example profiles use `TEST_RUBY_PROF_PRINTER` env variable ('cause using `TEST_RUBY_PROF` activates the global profiling). + +Also, you can specify RubyProf mode (`wall`, `cpu`, etc) through `TEST_RUBY_PROF_MODE` env variable. + +See [ruby_prof.rb](https://github.com/test-prof/test-prof/tree/master/lib/test_prof/ruby_prof.rb) for all available configuration options and their usage. + +It's useful to exclude some methods from the profile to focus only on the application code. + +TestProf uses RubyProf [`exclude_common_methods!`](https://github.com/ruby-prof/ruby-prof/blob/e087b7d7ca11eecf1717d95a5c5fea1e36ea3136/lib/ruby-prof/profile/exclude_common_methods.rb) by default (disable it with `config.exclude_common_methods = false`). + +We exclude some other common methods and RSpec specific internal methods by default. +To disable TestProf-defined exclusions set `config.test_prof_exclusions_enabled = false`. + +You can specify custom exclusions through `config.custom_exclusions`, e.g.: + +```ruby +TestProf::RubyProf.configure do |config| + config.custom_exclusions = {User => %i[save save!]} +end +``` + +[StackProf]: https://github.com/tmm1/stackprof +[Speedscope]: https://www.speedscope.app +[Vernier]: https://github.com/jhawthorn/vernier diff --git a/docs/profilers/stack_prof.md b/docs/profilers/stack_prof.md new file mode 100644 index 0000000..013f6b7 --- /dev/null +++ b/docs/profilers/stack_prof.md @@ -0,0 +1,2 @@ + +Please follow this link.
diff --git a/docs/profilers/tag_prof.md b/docs/profilers/tag_prof.md new file mode 100644 index 0000000..7565958 --- /dev/null +++ b/docs/profilers/tag_prof.md @@ -0,0 +1,114 @@ +# TagProf + +TagProf is a simple profiler which collects examples statistics grouped by a provided tag value. + +That's pretty useful in conjunction with `rspec-rails` built-in feature – `infer_spec_type_from_file_location!` – which automatically adds `type` to examples metadata. + +Example output: + +```sh +[TEST PROF INFO] TagProf report for type + + type time total %total %time avg + + request 00:04.808 42 33.87 54.70 00:00.114 + controller 00:02.855 42 33.87 32.48 00:00.067 + model 00:01.127 40 32.26 12.82 00:00.028 +``` + +It shows both the total number of examples in each group and the total time spent (as long as percentages and average values). + +You can also generate an interactive HTML report: + +```sh +TAG_PROF=type TAG_PROF_FORMAT=html bundle exec rspec +``` + +That's how a report looks like: + + + +## Instructions + +TagProf can be used with both RSpec and Minitest (limited support, see below). + +To activate TagProf use `TAG_PROF` environment variable: + +With Rspec: + +```sh +# Group by type +TAG_PROF=type rspec +``` + +With Minitest: + +```sh +# using pure ruby +TAG_PROF=type ruby + +# using Rails built-in task +TAG_PROF=type bin/rails test +``` + +NB: if another value than "type" is used for TAG_PROF environment variable it will be ignored silently in both Minitest and RSpec. + +### Usage specificity with Minitest + +Minitest does not support the usage of tags by default. TagProf therefore groups statistics by direct subdirectories of the root test directory. It assumes root test directory is named either `spec` or `test`. + +When no root test directory can be found the test statistics will not be grouped with other tests. They will be displayed per test with a significant warning message in the report. + +Example: + +```sh +[TEST PROF INFO] TagProf report for type + + type time sql.active_record total %total %time avg + +__unknown__ 00:04.808 00:01.402 42 33.87 54.70 00:00.114 + controller 00:02.855 00:00.921 42 33.87 32.48 00:00.067 + model 00:01.127 00:00.446 40 32.26 12.82 00:00.028 +``` + +## Profiling events + +You can combine TagProf with [EventProf](./event_prof.md) to track not only the total time spent but also the time spent for the specified activities (through events): + +``` +TAG_PROF=type TAG_PROF_EVENT=sql.active_record rspec +``` + +Example output: + +```sh +[TEST PROF INFO] TagProf report for type + + type time sql.active_record total %total %time avg + + request 00:04.808 00:01.402 42 33.87 54.70 00:00.114 + controller 00:02.855 00:00.921 42 33.87 32.48 00:00.067 + model 00:01.127 00:00.446 40 32.26 12.82 00:00.028 +``` + +Multiple events are also supported (comma-separated). + +## Pro-Tip: More Types + +By default, RSpec only infers types for default Rails app entities (such as controllers, models, mailers, etc.). +Modern Rails applications typically contain other abstractions too (e.g. services, forms, presenters, etc.), but RSpec is not aware of them and doesn't add any metadata. + +That's the quick workaround: + +```ruby +RSpec.configure do |config| + # ... + config.define_derived_metadata(file_path: %r{/spec/}) do |metadata| + # do not overwrite type if it's already set + next if metadata.key?(:type) + + match = metadata[:location].match(%r{/spec/([^/]+)/}) + metadata[:type] = match[1].singularize.to_sym + end +end +``` diff --git a/docs/recipes/active_record_shared_connection.md b/docs/recipes/active_record_shared_connection.md new file mode 100644 index 0000000..f1e3cd2 --- /dev/null +++ b/docs/recipes/active_record_shared_connection.md @@ -0,0 +1,30 @@ +# Active Record Shared Connection + +> 💀 This functionality has been removed in v1.0. + +**NOTE:** a similar functionality has been added to Rails since version 5.1 (see [PR](https://github.com/rails/rails/pull/28083)). You shouldn't use `ActiveRecordSharedConnection` with modern Rails, it could lead to unexpected behaviour (e.g., mutexes deadlocks). + +Active Record creates a connection per thread by default. + +That doesn't allow us to use `transactional_tests` feature in system (with Capybara) tests (since Capybara runs a web server in a separate thread). + +A common approach is to use `database_cleaner` with a non-transactional strategy (`truncation` / `deletion`). But that _cleaning_ phase may affect tests run time (and usually does). + +Sharing the connection between threads would allows us to use transactional tests as always. + +## Instructions + +In your `spec_helper.rb` (or `rails_helper.rb` if any): + +```ruby +require "test_prof/recipes/active_record_shared_connection" +``` + +That automatically enables _shared connection_ mode. + +You can enable/disable it manually: + +```ruby +TestProf::ActiveRecordSharedConnection.enable! +TestProf::ActiveRecordSharedConnection.disable! +``` diff --git a/docs/recipes/any_fixture.md b/docs/recipes/any_fixture.md new file mode 100644 index 0000000..a6c19ff --- /dev/null +++ b/docs/recipes/any_fixture.md @@ -0,0 +1,369 @@ +# AnyFixture + +Fixtures are a great way to increase your test suite performance, but for a large project, they are very hard to maintain. + +We propose a more general approach to lazy-generate the _global_ state for your test suite – AnyFixture. + +With AnyFixture, you can use any block of code for data generation, and it will take care of cleaning it out at the end of the run. + +Consider an example: + +```ruby +# The best way to use AnyFixture is through RSpec shared contexts +RSpec.shared_context "account", account: true do + # You should call AnyFixture outside of transaction to re-use the same + # data between examples + before(:all) do + # The provided name ("account") should be unique. + @account = TestProf::AnyFixture.register(:account) do + # Do anything here, AnyFixture keeps track of affected DB tables + # For example, you can use factories here + FactoryBot.create(:account) + + # or with Fabrication + Fabricate(:account) + + # or with plain old AR + Account.create!(name: "test") + end + end + + # Use .register here to track the usage stats (see below) + let(:account) { TestProf::AnyFixture.register(:account) } + + # Or hard-reload object if there is chance of in-place modification + let(:account) { Account.find(TestProf::AnyFixture.register(:account).id) } +end + +# You can enhance the existing database cleaning. Posts will be deleted before fixtures reset +TestProf::AnyFixture.before_fixtures_reset do + Post.delete_all +end + +# Or after reset +TestProf::AnyFixture.after_fixtures_reset do + Post.delete_all +end + +# Then in your tests + +# Active this fixture using a tag +describe UsersController, :account do + # ... +end + +# This test also uses the same account record, +# no double-creation +describe PostsController, :account do + # ... +end +``` + +See real life [example](http://bit.ly/any-fixture). + +## Instructions + +### RSpec + +In your `spec_helper.rb` (or `rails_helper.rb` if you have one): + +```ruby +require "test_prof/recipes/rspec/any_fixture" +``` + +Now you can use `TestProf::AnyFixture` in your tests. + +### Minitest + +When using AnyFixture with Minitest, you should take care of cleaning the database after each test run by yourself. For example: + +```ruby +# test_helper.rb + +require "test_prof/any_fixture" + +at_exit { TestProf::AnyFixture.clean } +``` + +## DSL + +We provide an optional _syntactic sugar_ (through Refinement) to make it easier to define fixtures and use callbacks: + +```ruby +require "test_prof/any_fixture/dsl" + +# Enable DSL +using TestProf::AnyFixture::DSL + +# and then you can use `fixture` method (which is just an alias for `TestProf::AnyFixture.register`) +before(:all) { fixture(:account) } + +# You can also use it to fetch the record (instead of storing it in instance variable) +let(:account) { fixture(:account) } + +# You can just use `before_fixtures_reset` or `after_fixtures_reset` callbacks +before_fixtures_reset { Post.delete_all } +after_fixtures_reset { Post.delete_all } +``` + +## `ActiveRecord#refind` + +TestProf also provides an extension to _hard-reload_ ActiveRecord objects: + +```ruby +# instead of +let(:account) { Account.find(fixture(:account).id) } + +# load refinement +require "test_prof/ext/active_record_refind" + +using TestProf::Ext::ActiveRecordRefind + +let(:account) { fixture(:account).refind } +``` + +## Temporary disable fixtures + +Some of your tests might rely on _clean database_. Thus running them along with AnyFixture-dependent tests, could produce failures. + +You can disable (or delete) all created fixture while running a specified example or group using the `:with_clean_fixture` shared context: + +```ruby +context "global state", :with_clean_fixture do + # or include explicitly + # include_context "any_fixture:clean" + + specify "table is empty or smth like this" do + # ... + end +end +``` + +How does it work? It wraps the example group into a transaction (using [`before_all`](./before_all.md)) and calls `TestProf::AnyFixture.clean` before running the examples. + +Thus, this context is a little bit _heavy_. Try to avoid such situations and write specs independent of the global state. + +## Usage report + +`AnyFixture` collects the usage information during the test run and could report it at the end: + +```sh +[TEST PROF INFO] AnyFixture usage stats: + + key build time hit count saved time + + user 00:00.004 4 00:00.017 + post 00:00.002 1 00:00.002 + +Total time spent: 00:00.006 +Total time saved: 00:00.019 +Total time wasted: 00:00.000 +``` + +The reporting is off by default, to enable the reporting set `TestProf::AnyFixture.config.reporting_enabled = true` (or you can invoke it manually through `TestProf::AnyFixture.report_stats`). + +You can also enable reporting through the `ANYFIXTURE_REPORT=1` env variable. + +## Using auto-generated SQL dumps + +> @since v1.0, experimental + +AnyFixture is designed to generate data once per a test suite run (and cleanup in the end). It still could be time-consuming (e.g., for system or performance tests); thus, we want to optimize further. + +We provide another way of speeding up test data called `#register_dump`. It works similarly to `#register` for the first run: it accepts a block of code and tracks SQL queries made within it. Then, it generates a plain SQL dump representing the data creating or modified during the call and uses this dump to restore the database state for the subsequent test runs. + +Let's consider an example: + +```ruby +RSpec.shared_context "account", account: true do + # You should call AnyFixture outside of transaction to re-use the same + # data between examples + before(:all) do + # The block is called once per test run (similary to #register) + TestProf::AnyFixture.register_dump("account") do + # Do anything here, AnyFixture keeps track of affected DB tables + # For example, you can use factories here + account = FactoryBot.create(:account, name: "test") + + # or with Fabrication + account = Fabricate(:account, name: "test") + + # or with plain old AR + account = Account.create!(name: "test") + + # updates are also tracked + account.update!(tag: "sql-dump") + end + end + + # Here, we MUST use a custom way to retrieve a record: since we restore the data + # from a plain SQL dump, we have no knowledge of Ruby objects + let(:account) { Account.find_by!(name: "test") } +end +``` + +And that's what happened when we run tests: + +```sh +# first run +$ bundle exec rspec + +# AnyFixture.register_dump is called: +# - is SQL dump present? No +# - run block and write all modifying queries to a new SQL dump +# AnyFixture.clean is called: +# - clean all the affected tables + +# second run +$ bundle exec rspec + +# AnyFixture.register_dump is called: +# - is SQL dump present? Yes +# - restore dump (do not run block) +# AnyFixture.clean is called: +# - clean all the affected tables +``` + +### Requirements + +Currently, only PostgreSQL 12+ and SQLite3 are supported. + +### Dump invalidation + +The generated dump could become out of date for many reasons: database schema changed, fixture block has been updated, etc. +To deal with invalidation, we use file content digests as _cache keys_ (dump file name suffixes). + +By default, AnyFixture _watches_ `db/schema.rb`, `db/structure.sql` and the file that calls `#register_dump`. + +The list of default watch files could be updated by modifying the `default_dump_watch_paths` configuration parameter: + +```ruby +TestProf::AnyFixture.configure do |config| + # you can use exact file paths or globs + config.default_dump_watch_paths << Rails.root.join("spec/factories/**/*") +end +``` + +Also, you add watch files to a specific `#register_dump` call via the `watch` option: + +```ruby +TestProf::AnyFixture.register_dump("account", watch: ["app/models/account.rb", "app/models/account/**/*,rb"]) do + # ... +end +``` + +**NOTE:** When you use the `watch` option, the current file is not added to the watch list. You should use `__FILE__` explicitly for that. + +Finally, if you want to forcefully re-generate a dump, you can use the `ANYFIXTURE_FORCE_DUMP` environment variable: + +- `ANYFIXTURE_FORCE_DUMP=1` will force all dumps regeneration. +- `ANYFIXTURE_FORCE_DUMP=account` will force regeneration only of the matching dumps (i.e., matching `/account/`). + +#### Cache keys + +It's possible to provide custom cache keys to be used as a part of a digest: + +```ruby +# cache_key could be pretty much anything that responds to #to_s +TestProf::AnyFixture.register_dump("account", cache_key: ["str", 1, {key: :val}]) do + # ... +end +``` + +### Hooks + +#### `before` / `after` + +Before hooks are called either before calling a fixture block or before restoring a dump. +One particular use case is to re-create a tenant in a multi-tenant app: + +```ruby +TestProf::AnyFixture.register_dump( + "account", + before: proc do + begin + Apartment::Tenant.create("test") + rescue + nil + end + Apartment::Tenant.create("test") + end +) do + # ... +end +``` + +Similarly, after hooks are called either after calling a fixture block or after restoring a dump. + +You can also specify global before and after hooks: + +```ruby +TestProf::AnyFixture.configure do |config| + config.before_dump do |dump:, import:| + # dump is an object containing information about the dump (e.g., dump.digest) + # import is true if we're restoring a dump and false otherwise + # do something + end + + config.after_dump do |dump:, import:| + # ... + end +end +``` + +**NOTE**: after callbacks are always executed, even if dump creation failed. You can use the `dump.success?` method to determine whether data generation succeeds or not. + +#### `skip_if` + +This callback is available only as of the `#register_dump` option and could be used to ignore the fixture completely. This is useful when you want to preserve the database state between test runs (i.e., do not clean the DB). + +Here is a complete example: + +```ruby +TestProf::AnyFixture.register_dump( + "account", + # do not track tables for AnyFixture.clean (though other fixtures could affect this) + clean: false, + skip_if: proc do |dump:| + Apartment::Tenant.switch!("test") + # if the current account has matching meta — the database is in actual state + Account.find_by!(name: "test").meta["dump-version"] == dump.digest + end, + before: proc do + begin + Apartment::Tenant.create("test") + rescue + nil + end + Apartment::Tenant.create("test") + end, + after: proc do |dump:, import:| + # do not persist dump version if dump failed or we're restoring data + next if import || !dump.success? + + Account.find_by!(name: "test").then do |account| + account.meta["dump-version"] = dump.digest + account.save! + end + end +) do + # ... +end +``` + +### Configuration + +There a few more configuration options available: + +```ruby +TestProf::AnyFixture.configure do |config| + # Where to store dumps (by default, TestProf.artifact_path + '/any_dumps') + config.dumps_dir = "any_dumps" + # Include mathing queries into a dump (in addition to INSERT/UPDATE/DELETE queries) + config.dump_matching_queries = /^$/ + # Whether to try using CLI tools such as psql or sqlite3 to restore dumps or not (and use ActiveRecord instead) + config.import_dump_via_cli = false +end +``` + +**NOTE:** When using CLI tools to restore dumps, it's not possible to track affected tables and thus clean them via `AnyFixture.clean`. diff --git a/docs/recipes/before_all.md b/docs/recipes/before_all.md new file mode 100644 index 0000000..e91f6bb --- /dev/null +++ b/docs/recipes/before_all.md @@ -0,0 +1,283 @@ +# Before All + +Rails has a great feature – `transactional_tests`, i.e. running each example within a transaction which is roll-backed in the end. + +Thus no example pollutes global database state. + +But what if have a lot of examples with a common setup? + +Of course, we can do something like this: + +```ruby +describe BeatleWeightedSearchQuery do + before(:each) do + @paul = create(:beatle, name: "Paul") + @ringo = create(:beatle, name: "Ringo") + @george = create(:beatle, name: "George") + @john = create(:beatle, name: "John") + end + + # and about 15 examples here +end +``` + +Or you can try `before(:all)`: + +```ruby +describe BeatleWeightedSearchQuery do + before(:all) do + @paul = create(:beatle, name: "Paul") + # ... + end + + # ... +end +``` + +But then you have to deal with database cleaning, which can be either tricky or slow. + +There is a better option: we can wrap the whole example group into a transaction. +And that's how `before_all` works: + +```ruby +describe BeatleWeightedSearchQuery do + before_all do + @paul = create(:beatle, name: "Paul") + # ... + end + + # ... +end +``` + +That's all! + +**NOTE**: requires RSpec >= 3.3.0. + +**NOTE**: Great superpower that `before_all` provides comes with a great responsibility. +Make sure to check the [Caveats section](#caveats) of this document for details. + +## Instructions + +### Multiple database support + +The ActiveRecord BeforeAll adapter will only start a transaction using ActiveRecord::Base connection. +If you want to ensure `before_all` can use multiple connections, you need to ensure the connection +classes are loaded before using `before_all`. + +For example, imagine you have `ApplicationRecord` and a separate database for user accounts: + +```ruby +class Users < AccountsRecord + # ... +end + +class Articles < ApplicationRecord + # ... +end +``` + +Then those two Connection Classes do need to be loaded before the tests are run: + +```ruby + +# Ensure connection classes are loaded +ApplicationRecord +AccountsRecord +``` + +This code can be added to `rails_helper.rb` or the rake tasks that runs minitests. + +### RSpec + +In your `rails_helper.rb` (or `spec_helper.rb` after *ActiveRecord* has been loaded): + +```ruby +require "test_prof/recipes/rspec/before_all" +``` + +**NOTE**: `before_all` (and `let_it_be` that depends on it), does not wrap individual +tests in a database transaction of its own. Use Rails' native `use_transactional_tests` +(`use_transactional_fixtures` in Rails < 5.1), RSpec Rails' `use_transactional_fixtures`, +DatabaseCleaner, or custom code that begins a transaction before each test and rolls it +back after. + +### Minitest + +It is possible to use `before_all` with Minitest too: + +```ruby +require "test_prof/recipes/minitest/before_all" + +class MyBeatlesTest < Minitest::Test + include TestProf::BeforeAll::Minitest + + before_all do + @paul = create(:beatle, name: "Paul") + @ringo = create(:beatle, name: "Ringo") + @george = create(:beatle, name: "George") + @john = create(:beatle, name: "John") + end + + # define tests which could access the object defined within `before_all` +end +``` + +In addition to `before_all`, TestProf also provides a `after_all` callback, which is called right before the transaction open by `before_all` is closed, i.e., after the last example from the test class completes. + +## Database adapters + +You can use `before_all` not only with ActiveRecord (which is supported out-of-the-box) but with other database tools too. + +All you need is to build a custom adapter and configure `before_all` to use it: + +```ruby +class MyDBAdapter + # before_all adapters must implement two methods: + # - begin_transaction + # - rollback_transaction + def begin_transaction + # ... + end + + def rollback_transaction + # ... + end +end + +# And then set adapter for `BeforeAll` module +TestProf::BeforeAll.adapter = MyDBAdapter.new +``` + +## Hooks + +You can register callbacks to run before/after `before_all` opens and rollbacks a transaction: + +```ruby +TestProf::BeforeAll.configure do |config| + config.before(:begin) do + # do something before transaction opens + end + # after(:begin) is also available + + config.after(:rollback) do + # do something after transaction closes + end + # before(:rollback) is also available +end +``` + +See the example in [Discourse](https://github.com/discourse/discourse/blob/4a1755b78092d198680c2fe8f402f236f476e132/spec/rails_helper.rb#L81-L141). + +## Caveats + +### Database is rolled back to a pristine state, but the objects are not + +If you modify objects generated within a `before_all` block in your examples, you maybe have to re-initiate them: + +```ruby +before_all do + @user = create(:user) +end + +let(:user) { @user } + +it "when user is admin" do + # we modified our object in-place! + user.update!(role: 1) + expect(user).to be_admin +end + +it "when user is regular" do + # now @user's state depends on the order of specs! + expect(user).not_to be_admin +end +``` + +The easiest way to solve this is to reload record for every example (it's still much faster than creating a new one): + +```ruby +before_all do + @user = create(:user) +end + +# Note, that @user.reload may not be enough, +# 'cause it doesn't reset associations +let(:user) { User.find(@user.id) } + +# or with Minitest +def setup + @user = User.find(@user.id) +end +``` + +### Database is not rolled back between tests + +Database is not rolled back between RSpec examples, only between example groups. +We don't want to reinvent the wheel and encourage you to use other tools that +provide this out of the box. + +If you're using RSpec Rails, turn on `RSpec.configuration.use_transactional_fixtures` in your `spec/rails_helper.rb`: + +```ruby +RSpec.configure do |config| + config.use_transactional_fixtures = true # RSpec takes care to use `use_transactional_tests` or `use_transactional_fixtures` depending on the Rails version used +end +``` + +Make sure to set `use_transactional_tests` (`use_transactional_fixtures` in Rails < 5.1) to `true` if you're using Minitest. + +If you're using DatabaseCleaner, make sure it rolls back the database between tests. + +## Usage with Isolator + +[Isolator](https://github.com/palkan/isolator) is a runtime detector of potential atomicity breaches within DB transactions (e.g. making HTTP calls or enqueuing background jobs). + +TestProf recognizes Isolator out-of-the-box and make it ignore `before_all` transactions. + +You just need to make sure that you require `isolator` before loading `before_all` (or `let_it_be`). + +Alternatively, you can load the patch explicitly: + +```ruby +# after loading before_all or/and let_it_be +require "test_prof/before_all/isolator" +``` + +## Using Rails fixtures (_experimental_) + +If you want to use fixtures within a `before_all` hook, you must explicitly opt-in via `setup_fixture:` option: + +```ruby +before_all(setup_fixtures: true) do + @user = users(:john) + @post = create(:post, user: user) +end +``` + +Works for both Minitest and RSpec. + +You can also enable fixtures globally (i.e., for all `before_all` hooks): + +```ruby +TestProf::BeforeAll.configure do |config| + config.setup_fixtures = true +end +``` + +## Global Tags + +You can register callbacks for specific RSpec Example Groups using tags: + +```ruby +TestProf::BeforeAll.configure do |config| + config.before(:begin, reset_sequences: true, foo: :bar) do + warn <<~MESSAGE + Do NOT create objects outside of transaction + because all db sequences will be reset to 1 + in every single example, so that IDs of new objects + can get into conflict with the long-living ones. + MESSAGE + end +end +``` diff --git a/docs/recipes/factory_all_stub.md b/docs/recipes/factory_all_stub.md new file mode 100644 index 0000000..13d7d4a --- /dev/null +++ b/docs/recipes/factory_all_stub.md @@ -0,0 +1,58 @@ +# FactoryAllStub + +_Factory All Stub_ is a spell to force FactoryBot/FactoryGirl use only `build_stubbed` strategy (even if you call `create` or `build`). + +The idea behind it is to quickly fix [Factory Doctor](../profilers/factory_doctor.md) offenses (and even do that automatically). + +**NOTE**. Only works with FactoryGirl/FactoryBot. Should be considered only as a temporary specs fix. + +## Instructions + +First, you have to initialize `FactoryAllStub`: + +```ruby +TestProf::FactoryAllStub.init +``` + +The initialization process injects custom logic into FactoryBot generator. + +To enable _all-stub_ mode: + +```ruby +TestProf::FactoryAllStub.enable! +``` + +To disable _all-stub_ mode and use factories as always: + +```ruby +TestProf::FactoryAllStub.disable! +``` + +## RSpec + +In your `spec_helper.rb` (or `rails_helper.rb` if any): + +```ruby +require "test_prof/recipes/rspec/factory_all_stub" +``` + +That would automatically initialize `FactoryAllStub` (no need to call `.init`) and provide +`"factory:stub"` shared context with enables it for the marked examples or example groups: + +```ruby +describe "User" do + let(:user) { create(:user) } + + it "is valid", factory: :stub do + # use `build_stubbed` instead of `create` + expect(user).to be_valid + end +end +``` + +`FactoryAllStub` was designed to be used with `FactoryDoctor` the following way: + +```sh +# Run FactoryDoctor and mark all offensive examples with factory:stub +FDOC=1 FDOC_STAMP=factory:stub rspec ./spec/models +``` diff --git a/docs/recipes/factory_default.md b/docs/recipes/factory_default.md new file mode 100644 index 0000000..1310a9f --- /dev/null +++ b/docs/recipes/factory_default.md @@ -0,0 +1,254 @@ +# FactoryDefault + +_FactoryDefault_ aims to help you cope with _factory cascades_ (see [FactoryProf](../profilers/factory_prof.md)) by reusing associated records. + +**NOTE**. Only works with FactoryGirl/FactoryBot. + +It can be very useful when you're working on a typical SaaS application (or other hierarchical data). + +Consider an example. Assume we have the following factories: + +```ruby +factory :account do +end + +factory :user do + account +end + +factory :project do + account + user +end + +factory :task do + account + project + user +end +``` + +And we want to test the `Task` model: + +```ruby +describe "PATCH #update" do + let(:task) { create(:task) } + + it "works" do + patch :update, id: task.id, task: {completed: "t"} + expect(response).to be_success + end + + # ... +end +``` + +How many users and accounts are created per example? Two and four respectively. + +And it breaks our logic (every object should belong to the same account). + +Typical workaround: + +```ruby +describe "PATCH #update" do + let(:account) { create(:account) } + let(:project) { create(:project, account: account) } + let(:task) { create(:task, project: project, account: account) } + + it "works" do + patch :update, id: task.id, task: {completed: "t"} + expect(response).to be_success + end +end +``` + +That works. And there are some cons: it's a little bit verbose and error-prone (easy to forget something). + +Here is how we can deal with it using FactoryDefault: + +```ruby +describe "PATCH #update" do + let(:account) { create_default(:account) } + let(:project) { create_default(:project) } + let(:task) { create(:task) } + + # and if we need more projects, users, tasks with the same parent record, + # we just write + let(:another_project) { create(:project) } # uses the same account + let(:another_task) { create(:task) } # uses the same account + + it "works" do + patch :update, id: task.id, task: {completed: "t"} + expect(response).to be_success + end +end +``` + +**NOTE**. This feature introduces a bit of _magic_ to your tests, so use it with caution ('cause tests should be human-readable first). Good idea is to use defaults for top-level entities only (such as tenants in multi-tenancy apps). + +## Instructions + +In your `spec_helper.rb`: + +```ruby +require "test_prof/recipes/rspec/factory_default" +``` + +This adds two new methods to FactoryBot: + +- `FactoryBot#set_factory_default(factory, object)` – use the `object` as default for associations built with `factory` + +Example: + +```ruby +let(:user) { create(:user) } + +before { FactoryBot.set_factory_default(:user, user) } +``` + +- `FactoryBot#create_default(factory, *args)` – is a shortcut for `create` + `set_factory_default`. + +**IMPORTANT:** Defaults are **cleaned up after each example** by default (i.e., when using `test_prof/recipes/rspec/factory_default`). + +### Using with `before_all` / `let_it_be` + +Defaults created within `before_all` and `let_it_be` are not reset after each example, but only at the end of the corresponding example group. So, it's possible to call `create_default` within `let_it_be` without any additional configuration. **RSpec only** + +**IMPORTANT:** You must load FactoryDefault after loading BeforeAll to make this feature work. + +**NOTE**. Regular `before(:all)` callbacks are not supported. + +### Working with traits + +You can use traits in your associations, for example: + +```ruby +factory :comment do + user +end + +factory :post do + association :user, factory: %i[user able_to_post] +end + +factory :view do + association :user, factory: %i[user unable_to_post_only_view] +end +``` + +If there is a default value for the `user` factory, it's gonna be used independently of traits. This may break your logic. + +To prevent this, configure FactoryDefault to preserve traits: + +```ruby +# Globally +TestProf::FactoryDefault.configure do |config| + config.preserve_traits = true +end + +# or in-place +create_default(:user, preserve_traits: true) +``` + +Creating a default with trait works as follows: + +```ruby +# Create a default with trait +user = create_default(:user_poster, :able_to_post) + +# When an association has no traits specified, the default with trait is used +create(:comment).user == user #=> true +# When an association has the matching trait specified, the default is used, too +create(:post).user == user #=> true +# When the association's trait differs, default is skipped +create(:view).user == user #=> false +``` + +### Handling attribute overrides + +It's possible to define attribute overrides for associations: + +```ruby +factory :post do + association :user, name: "Poster" +end + +factory :view do + association :user, name: "Viewer" +end +``` + +FactoryDefault ignores such overrides and still returns a default `user` record (if created). You can turn the attribute awareness feature on to skip the default record if overrides don't match the default object attributes: + +```ruby +# Globally +TestProf::FactoryDefault.configure do |config| + config.preserve_attributes = true +end + +# or in-place +create_default :user, preserve_attributes: true +``` + +**NOTE:** In the future versions of Test Prof, both `preserve_traits` and `preserve_attributes` will default to true. We recommend settings them to true if you just starting using this feature. + +### Ignoring default factories + +You can temporary disable the defaults usage by wrapping a code with the `skip_factory_default` method: + +```ruby +account = create_default(:account) +another_account = skip_factory_default { create(:account) } + +expect(another_account).not_to eq(account) +``` + +### Showing usage stats + +You can display the FactoryDefault usage stats by setting the `FACTORY_DEFAULT_SUMMARY=1` or `FACTORY_DEFAULT_STATS=1` env vars or by setting the configuration values: + +```ruby +TestProf::FactoryDefault.configure do |config| + config.report_summary = true + # Report stats prints the detailed usage information (including summary) + config.report_stats = true +end +``` + +For example: + +```sh +$ FACTORY_DEFAULT_SUMMARY=1 bundle exec rspec + +FactoryDefault summary: hit=11 miss=3 +``` + +Where `hit` indicates the number of times the default factory value was used instead of a new one when an association was created; `miss` indicates the number of time the default value was ignored due to traits or attributes mismatch. + +## Factory Default profiling, or when to use defaults + +Factory Default ships with the profiler, which can help you to see how associations are being used in your test suite, so you can decide on using `create_default` or not. + +To enable profiling, run your tests with the `FACTORY_DEFAULT_PROF=1` set: + +```sh +$ FACTORY_DEFAULT_PROF=1 bundle exec rspec spec/some/file_spec.rb + +..... + +[TEST PROF INFO] Factory associations usage: + + factory count total time + + user 17 00:42.010 + user[traited] 15 00:31.560 + user{tag:"some tag"} 1 00:00.205 + +Total associations created: 33 +Total uniq associations created: 3 +Total time spent: 01:13.775 +``` + +Since default factories are usually registered per an example group (or test class), we recommend running this profiler against a particular file, so you can quickly identify the possibility of adding `create_default` and improve the tests speed. + +**NOTE:** You can also use the profiler to measure the effect of adding `create_default`; for that, compare the results of running the profiler with FactoryDefault enabled and disabled (you can do that by passing the `FACTORY_DEFAULT_DISABLED=1` env var). diff --git a/docs/recipes/let_it_be.md b/docs/recipes/let_it_be.md new file mode 100644 index 0000000..7eaad8c --- /dev/null +++ b/docs/recipes/let_it_be.md @@ -0,0 +1,268 @@ +# Let It Be + +Let's bring a little bit of magic and introduce a new way to set up a _shared_ test data. + +Suppose you have the following setup: + +```ruby +describe BeatleWeightedSearchQuery do + let!(:paul) { create(:beatle, name: "Paul") } + let!(:ringo) { create(:beatle, name: "Ringo") } + let!(:george) { create(:beatle, name: "George") } + let!(:john) { create(:beatle, name: "John") } + + specify { expect(subject.call("john")).to contain_exactly(john) } + + # and more examples here +end +``` + +We don't need to re-create the Fab Four for every example, do we? + +We already have [`before_all`](./before_all.md) to solve the problem of _repeatable_ data: + +```ruby +describe BeatleWeightedSearchQuery do + before_all do + @paul = create(:beatle, name: "Paul") + # ... + end + + specify { expect(subject.call("joh")).to contain_exactly(@john) } + + # ... +end +``` + +That technique works pretty good but requires us to use instance variables and define everything at once. Thus it's not easy to refactor existing tests which use `let/let!` instead. + +With `let_it_be` you can do the following: + +```ruby +describe BeatleWeightedSearchQuery do + let_it_be(:paul) { create(:beatle, name: "Paul") } + let_it_be(:ringo) { create(:beatle, name: "Ringo") } + let_it_be(:george) { create(:beatle, name: "George") } + let_it_be(:john) { create(:beatle, name: "John") } + + specify { expect(subject.call("john")).to contain_exactly(john) } + + # and more examples here +end +``` + +That's it! Just replace `let!` with `let_it_be`. That's equal to the `before_all` approach but requires less refactoring. + +**NOTE**: Great superpower that `before_all` provides comes with a great responsibility. +Make sure to check the [Caveats section](#caveats) of this document for details. + +## Instructions + +In your `rails_helper.rb` or `spec_helper.rb`: + +```ruby +require "test_prof/recipes/rspec/let_it_be" +``` + +In your tests: + +```ruby +describe MySuperDryService do + let_it_be(:user) { create(:user) } + + # ... +end +``` + +`let_it_be` won't automatically bring the database to its previous state between +the examples, it only does that between example groups. +Use Rails' native `use_transactional_tests` (`use_transactional_fixtures` in Rails < 5.1), +RSpec Rails' `use_transactional_fixtures`, DatabaseCleaner, or custom code that +begins a transaction before each test and rolls it back after. + +## Caveats + +### Database is rolled back to a pristine state, but the objects are not + +If you modify objects generated within a `let_it_be` block in your examples, you maybe have to re-initiate them. +We have a built-in _modifiers_ support for that. + +### Database is not rolled back between tests + +Database is not rolled back between RSpec examples, only between example groups. +We don't want to reinvent the wheel and encourage you to use other tools that +provide this out of the box. + +If you're using RSpec Rails, turn on `RSpec.configuration.use_transactional_fixtures` in your `spec/rails_helper.rb`: + +```ruby +RSpec.configure do |config| + config.use_transactional_fixtures = true # RSpec takes care to use `use_transactional_tests` or `use_transactional_fixtures` depending on the Rails version used +end +``` + +Make sure to set `use_transactional_tests` (`use_transactional_fixtures` in Rails < 5.1) to `true` if you're using Minitest. + +If you're using DatabaseCleaner, make sure it rolls back the database between tests. + +## Aliases + +Naming is hard. Handling edge cases (the ones described above) is also tricky. + +To solve this we provide a way to define `let_it_be` aliases with the predefined options: + +```ruby +# rails_helper.rb +TestProf::LetItBe.configure do |config| + # define an alias with `refind: true` by default + config.alias_to :let_it_be_with_refind, refind: true +end + +# then use it in your tests +describe "smth" do + let_it_be_with_refind(:foo) { Foo.create } + + # refind can still be overridden + let_it_be_with_refind(:bar, refind: false) { Bar.create } +end +``` + +## Modifiers + +If you modify objects generated within a `let_it_be` block in your examples, you maybe have to re-initiate them to avoid state leakage between the examples. +Keep in mind that even though the database is rolled back to its pristine state, models themselves are not. + +We have a built-in _modifiers_ support for getting models to their pristine state: + +```ruby +# Use reload: true option to reload user object (assuming it's an instance of ActiveRecord) +# for every example +let_it_be(:user, reload: true) { create(:user) } + +# it is almost equal to +before_all { @user = create(:user) } +let(:user) { @user.reload } + +# You can also specify refind: true option to hard-reload the record +let_it_be(:user, refind: true) { create(:user) } + +# it is almost equal to +before_all { @user = create(:user) } +let(:user) { User.find(@user.id) } +``` + +**NOTE:** make sure that you require `let_it_be` after `active_record` is loaded (e.g., in `rails_helper.rb` **after** requiring the Rails app); otherwise the `refind` and `reload` modifiers are not activated. + +You can also use modifiers with array values, e.g. `create_list`: + +```ruby +let_it_be(:posts, reload: true) { create_list(:post, 3) } + +# it's the same as +before_all { @posts = create_list(:post, 3) } +let(:posts) { @posts.map(&:reload) } +``` + +### Custom Modifiers + +If `reload` and `refind` is not enough, you can add your custom modifier: + +```ruby +# rails_helper.rb +TestProf::LetItBe.configure do |config| + # Define a block which will be called when you access a record first within an example. + # The first argument is the pre-initialized record, + # the second is the value of the modifier. + # + # This is how `reload` modifier is defined + config.register_modifier :reload do |record, val| + # ignore when `reload: false` + next record unless val + # ignore non-ActiveRecord objects + next record unless record.is_a?(::ActiveRecord::Base) + record.reload + end +end +``` + +### Default Modifiers + +It's possible to configure the default modifiers used for all `let_it_be` calls: + +- Globally: + +```ruby +TestProf::LetItBe.configure do |config| + # Make refind activated by default + config.default_modifiers[:refind] = true +end +``` + +- For specific contexts using tags: + +```ruby +context "with let_it_be reload", let_it_be_modifiers: {reload: true} do + # examples +end +``` + +**NOTE:** Nested contexts tags are overwritten not merged: + +```ruby +TestProf::LetItBe.configure do |config| + config.default_modifiers[:freeze] = false +end + +context "with reload", let_it_be_modifiers: {reload: true} do + # uses freeze: false, reload: true here + + context "with freeze", let_it_be_modifiers: {freeze: true} do + # uses only freeze: true (reload: true is overwritten by new metadata) + end +end +``` + +## State Leakage Detection + +From [`rspec-rails` docs](https://relishapp.com/rspec/rspec-rails/v/3-9/docs/transactions) on transactions and `before(:context)`: + +> Even though database updates in each example will be rolled back, the object won't know about those rollbacks so the object and its backing data can easily get out of sync. + +Since `let_it_be` initialize objects in `before(:context)` hooks under the hood, it's affected by this problem: the code might modify models shared between examples (thus causing _shared state leaks_). That could happen unwillingly: when the underlying code under test modifies models, e.g. modifies `updated_at` attribute; or deliberately: when models are updated in `before` hooks or examples themselves instead of creating models in a proper state initially. + +This state leakage comes with potentially harmful side effects on the other examples, such as implicit dependencies and execution order dependency. + +With many shared models between many examples, it's hard to track down the example and exact place in the code that modifies the model. + +To detect modifications, objects that are passed to `let_it_be` are frozen (with `#freeze`), and `FrozenError` is raised: + +```ruby +# use freeze: true modifier to enable this feature +let_it_be(:user, freeze: true) { create(:user) } + +# it is almost equal to +before_all { @user = create(:user).freeze } +let(:user) { @user } +``` + +To fix the `FrozenError`: + +- Add `reload: true`/`refind: true`, it pacifies leakage detection and prevents leakage itself. Typically it's significantly faster to reload the model than to re-create it from scratch before each example (two or even three orders of magnitude faster in some cases). +- Rewrite the problematic test code. + +This feature is opt-in, since it may find a significant number of leakages in specs that may be a significant burden to fix all at once. +It's possible to gradually turn it on for parts of specs (e.g., only models) by using: + +```ruby +# spec/spec_helper.rb +RSpec.configure do |config| + # ... + config.define_derived_metadata(let_it_be_frost: true) do |metadata| + metadata[:let_it_be_modifiers] ||= {freeze: true} + end +end +``` + +And then tag contexts/examples with `:let_it_be_frost` to enable this feature. + +Alternatively, you can specify `freeze` modifier explicitly (`let_it_be(freeze: true)`) or configure an alias. diff --git a/docs/recipes/logging.md b/docs/recipes/logging.md new file mode 100644 index 0000000..7904933 --- /dev/null +++ b/docs/recipes/logging.md @@ -0,0 +1,81 @@ +# Verbose Logging + +Sometimes digging through logs is the best way to figure out what's going on. + +When you run your test suite, logs are not printed out by default (although written to `test.log` – who cares?). + +We provide a recipe to turn verbose logging for a specific example/group. + +**NOTE:** Rails only. + +## Instructions + +Drop this line to your `rails_helper.rb` / `spec_helper.rb` / `test_helper.rb` / whatever: + +```ruby +require "test_prof/recipes/logging" +``` + +### Log everything + +To turn on logging globally use `LOG` env variable: + +```sh +# log everything to stdout +LOG=all rspec ... + +# or +LOG=all rake test + +# log only Active Record statements +LOG=ar rspec ... +``` + +### Per-example logging + +**NOTE:** RSpec only. + +Activate logging by adding special tag – `:log`: + +```ruby +# Add the tag and you will see a lot of interesting stuff in your console +it "does smthng weird", :log do + # ... +end + +# or for the group +describe "GET #index", :log do + # ... +end +``` + +To enable only Active Record log use `log: :ar` tag: + +```ruby +describe "GET #index", log: :ar do + # ... +end +``` + +### Logging helpers + +For more granular control you can use `with_logging` (log everything) and +`with_ar_logging` (log Active Record) helpers: + +```ruby +it "does somthing" do + do_smth + # show logs only for the code within the block + with_logging do + # ... + end +end +``` + +**NOTE:** in order to use this helpers with Minitest you should include the `TestProf::Rails::LoggingHelpers` module manually: + +```ruby +class MyLoggingTest < Minitest::Test + include TestProf::Rails::LoggingHelpers +end +``` diff --git a/docs/recipes/rspec_stamp.md b/docs/recipes/rspec_stamp.md new file mode 100644 index 0000000..4ff8e16 --- /dev/null +++ b/docs/recipes/rspec_stamp.md @@ -0,0 +1,52 @@ +# RSpecStamp + +RSpecStamp is a tool to automatically _tag_ failed examples with custom tags. + +It _literally_ adds tags to your examples (i.e. rewrites them). + +The main purpose of RSpecStamp is to make testing codebase refactoring easy. Changing global configuration may cause a lot of failures. You can patch failing spec by adding a shared context. And here comes RSpecStamp. + +## Example Use Case: Sidekiq Inline + +Using `Sidekiq::Testing.inline!` may be considered a _bad practice_ (see [here](https://github.com/mperham/sidekiq/issues/3495)) due to its negative performance impact. But it's still widely used. + +How to migrate from `inline!` to `fake!`? + +Step 0. Make sure that all your tests pass. + +Step 1. Create a shared context to conditionally turn on `inline!` mode: + +```ruby +shared_context "sidekiq:inline", sidekiq: :inline do + around(:each) { |ex| Sidekiq::Testing.inline!(&ex) } +end +``` + +Step 2. Turn on `fake!` mode globally. + +Step 3. Run `RSTAMP=sidekiq:inline rspec`. + +The output of the command above contains information about the _stamping_ process: + +- How many files have been affected? + +- How many patches were made? + +- How many patches failed? + +- How many files have been ignored? + +Now all (or almost all) failing specs are tagged with `sidekiq: :inline`. Run the whole suite again and check it there are any failures left. + +There is also a `dry-run` mode (activated by `RSTAMP_DRY_RUN=1` env variable) which prints out patches instead of re-writing files. + +## Configuration + +By default, RSpecStamp ignores examples located in `spec/support` directory (typical place to put shared examples in). +You can add more _ignore_ patterns: + +```ruby +TestProf::RSpecStamp.configure do |config| + config.ignore_files << %r{spec/my_directory} +end +``` diff --git a/docs/recipes/tests_sampling.md b/docs/recipes/tests_sampling.md new file mode 100644 index 0000000..1fe5580 --- /dev/null +++ b/docs/recipes/tests_sampling.md @@ -0,0 +1,36 @@ +# Tests Sampling + +Sometimes it's useful to run profilers against randomly chosen tests. Unfortunately, test frameworks don't support such functionality. That's why we've included small patches for RSpec and Minitest in TestProf. + +## Instructions + +Require the corresponding patch: + +```ruby +# For RSpec in your spec_helper.rb +require "test_prof/recipes/rspec/sample" + +# For Minitest in your test_helper.rb +require "test_prof/recipes/minitest/sample" +``` + +And then just add `SAMPLE` env variable with the number examples you want to run: + +```sh +SAMPLE=10 rspec +``` + +You can also run random set of example groups (or suites) using `SAMPLE_GROUPS` variable: + +```sh +SAMPLE_GROUPS=10 rspec +``` + +Note that you can use tests sampling with RSpec filters: + +```sh +SAMPLE=10 rspec --tag=api +SAMPLE_GROUPS=10 rspec -e api +``` + +That's it. Enjoy!