Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Reporter for RubyMine IDE / TeamCity CI Server #5

Merged
merged 4 commits into from

5 participants

@iromeo

Hi,

At the moment RubyMine IDE and TeamCity CI Server doesn't provide integration with mintiest framework. I think minitest-repoters gem is a best place for RubyMine/TeamCity custom test results reporter.

RubyMine and TeamCity expects stdout marked in the same format. On run-time they inject several helper ruby scripts in ruby process loadpath, my reporter adds run-time dependency on these scripts, but due to several reasons I cannot move scripts to this gem.

@conradwt

Hi, can you tell me the status of your commits for Rubymine? At this time, I'm playing with them with Rubymine 4. Are there any additional configuration that one needs to perform in regards to Rubymine?

@iromeo

Hi Conrad,
Most likely today we will publish next RubyMine 4 EAP. It provides sets RubyMine specific 'RM_INFO' env variable, so you can use in test_helper.rb smth like

if ENV["RM_INFO"] || ENV["TEAMCITY_VERSION"]
  MiniTest::Unit.runner.reporters << MiniTest::Reporters::RubyMineReporter.new
else
  ...
end

and single test/spec example launching will be also supported in coming EAP build

@kern kern merged commit d73c369 into kern:develop
@kern
Owner

Wonderful! :)

@iromeo

Thanks!

@bensomers

Hello!

I'm in the process of switching several very large apps from test-unit to minitest. I'm very much liking minitest-reporters so far, BUT: the RubyMineReporter is giving me severe performance problems. I'm using it for my TeamCity server, and it appears to be giving me about a 6x increase in test run times. I tested it by switching TeamCity to use the default reporter instead, and I get ~10 minute runs, the same as I had with test-unit. I actually get worse slowdowns using RubyMine locally (9x and counting, don't have time to let it finish).

I will be continuing to debug this, but has this issue surfaced before? I've got a very large test suite, which may be why I'm noticing the slowdown if other users haven't.

@VladRassokhin

Hello @bensomers
Could you please create issue on tracker or create discussion at our forum ?

@bensomers

I don't really know whether the issue is with TeamCity, or with the RubyMineReporter, or with the Teamcity hooks it's using. I'm also not much help at debugging this; I didn't see anything obvious in the reporter code, and didn't have convenient access to the APIs it was calling. I've since (written and) switched to the included JUnitReporter, which works just fine.

@VladRassokhin

@bensomers
How much tests do you have? There may be slowdown because RubyMineReporter write into $stdout for each test...

@bensomers

I was mostly testing with one suite of unit tests, containing 40072 assertions across 10398 tests across 456 files. I also tested against a few smaller functional suites, ranging from a few hundred to about 2000 tests.

I don't think I specifically checked the output-printing as a source for the problem, except by comparison with the DefaultReporter (which obviously prints less, though still quite a bit).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
This page is out of date. Refresh to see the latest.
View
11 README.md
@@ -24,7 +24,8 @@ The following reporters are provided:
MiniTest::Reporters::DefaultReporter # => Identical to the standard MiniTest reporter
MiniTest::Reporters::SpecReporter # => Turn-like output that reads like a spec
MiniTest::Reporters::ProgressReporter # => Fuubar-like output with a progress bar
- MiniTest::Reporters::RubyMateReporter # => Simple reporter designed for RubyMate; see below
+ MiniTest::Reporters::RubyMateReporter # => Simple reporter designed for RubyMate
+ MiniTest::Reporters::RubyMineReporter # => Reporter designed for RubyMine IDE and TeamCity CI server; see below
I really like `ProgressReporter` for my everyday terminal usage, but I like
using `RubyMateReporter` when I'm executing test suites from TextMate. My usual
@@ -38,6 +39,14 @@ set up looks like this:
MiniTest::Unit.runner.reporters << MiniTest::Reporters::ProgressReporter.new
end
+If you prefer integration with RubyMine test runner or TeamCity CI server you'll need:
+
+ if ENV["RM_INFO"] || ENV["TEAMCITY_VERSION"]
+ MiniTest::Unit.runner.reporters << MiniTest::Reporters::RubyMineReporter.new
+ else
+ ...
+ end
+
## TODO ##
* Make the boilerplate code look prettier. Something like a one-line require for the general use-case would be nice.
View
1  lib/minitest/reporters.rb
@@ -12,5 +12,6 @@ module Reporters
autoload :SpecReporter, 'minitest/reporters/spec_reporter'
autoload :ProgressReporter, 'minitest/reporters/progress_reporter'
autoload :RubyMateReporter, 'minitest/reporters/ruby_mate_reporter'
+ autoload :RubyMineReporter, 'minitest/reporters/rubymine_reporter'
end
end
View
163 lib/minitest/reporters/rubymine_reporter.rb
@@ -0,0 +1,163 @@
+# Test results reporter for RubyMine IDE (http://www.jetbrains.com/ruby/) and
+# TeamCity(http://www.jetbrains.com/teamcity/) Continuous Integration Server
+#
+# Usage:
+# # in test_helper.rb
+# ...
+# if ENV["RM_INFO"] || ENV["TEAMCITY_VERSION"]
+# MiniTest::Unit.runner.reporters << MiniTest::Reporters::RubyMineReporter.new
+# else
+# MiniTest::Unit.runner.reporters << MiniTest::Reporters::DefaultReporter.new
+# end
+
+require "ansi"
+begin
+ require 'teamcity/runner_common'
+ require 'teamcity/utils/service_message_factory'
+ require 'teamcity/utils/runner_utils'
+ require 'teamcity/utils/url_formatter'
+rescue LoadError
+ MiniTest::Unit.runner.output.puts("====================================================================================================\n")
+ MiniTest::Unit.runner.output.puts("RubyMine reporter works only if it test was launched using RubyMine IDE or TeamCity CI server !!!\n")
+ MiniTest::Unit.runner.output.puts("====================================================================================================\n")
+ MiniTest::Unit.runner.output.puts("Using default results reporter...\n")
+
+ require "minitest/reporters/default_reporter"
+
+ # delegate to default reporter
+ module MiniTest
+ module Reporters
+ class RubyMineReporter < MiniTest::Reporters::DefaultReporter
+ end
+ end
+ end
+else
+ module MiniTest
+ module Reporters
+ class RubyMineReporter
+ include MiniTest::Reporter
+ include ANSI::Code
+
+ include ::Rake::TeamCity::RunnerCommon
+ include ::Rake::TeamCity::RunnerUtils
+ include ::Rake::TeamCity::Utils::UrlFormatter
+
+ def runner
+ MiniTest::Unit.runner
+ end
+
+ def output
+ runner.output
+ end
+
+ def verbose?
+ runner.verbose
+ end
+
+ def print(*args)
+ runner.output.print(*args)
+ end
+
+ def puts(*args)
+ runner.output.puts(*args)
+ end
+
+ def before_suites(suites, type)
+ puts 'Started'
+ puts
+
+ # Setup test runner's MessageFactory
+ set_message_factory(Rake::TeamCity::MessageFactory)
+ log_test_reporter_attached()
+
+ # Report tests count:
+ test_count = runner.test_count
+ if ::Rake::TeamCity.is_in_idea_mode
+ log(@message_factory.create_tests_count(test_count))
+ elsif ::Rake::TeamCity.is_in_buildserver_mode
+ log(@message_factory.create_progress_message("Starting.. (#{test_count} tests)"))
+ end
+
+ end
+
+ def after_suites(suites, type)
+ total_time = Time.now - runner.start_time
+
+ puts('Finished in %.5fs' % total_time)
+ print('%d tests, %d assertions, ' % [runner.test_count, runner.assertion_count])
+ print(red { '%d failures, %d errors, ' } % [runner.failures, runner.errors])
+ print(yellow { '%d skips' } % runner.skips)
+ puts
+ end
+
+ def before_suite(suite)
+ fqn = suite.name
+ log(@message_factory.create_suite_started(suite.name, location_from_ruby_qualified_name(fqn)))
+ end
+
+ def after_suite(suite)
+ log(@message_factory.create_suite_finished(suite.name))
+ end
+
+ def before_test(suite, test)
+ fqn = "#{suite.name}.#{test.to_s}"
+ log(@message_factory.create_test_started(test, minitest_test_location(fqn)))
+ end
+
+ def pass(suite, test, test_runner)
+ test_finished(test, test_runner)
+ end
+
+ def skip(suite, test, test_runner)
+ test_finished(test, test_runner) do |exception_msg, backtrace|
+ log(@message_factory.create_test_ignored(test, exception_msg, backtrace))
+ end
+ end
+
+ def failure(suite, test, test_runner)
+ test_finished(test, test_runner) do |exception_msg, backtrace|
+ log(@message_factory.create_test_failed(test, exception_msg, backtrace))
+ end
+ end
+
+ def error(suite, test, test_runner)
+ test_finished(test, test_runner) do |exception_msg, backtrace|
+ log(@message_factory.create_test_error(test, exception_msg, backtrace))
+ end
+ end
+
+ #########
+ def log(msg)
+ output.flush
+ output.puts("\n#{msg}")
+ output.flush
+
+ # returns:
+ msg
+ end
+
+ def minitest_test_location(fqn)
+ return nil if (fqn.nil?)
+ "ruby_minitest_qn://#{fqn}"
+ end
+
+ def test_finished(test, test_runner)
+ duration_ms = get_current_time_in_ms() - get_time_in_ms(runner.test_start_time)
+
+ begin
+ if block_given?
+ exception = test_runner.exception
+ msg = exception.nil? ? "" : "#{exception.class.name}: #{exception.message}"
+ backtrace = exception.nil? ? "" : format_backtrace(exception.backtrace)
+
+ yield(msg, backtrace)
+ end
+ ensure
+ log(@message_factory.create_test_finished(test, duration_ms.nil? ? 0 : duration_ms))
+ end
+ end
+ end
+ end
+ end
+end
+
View
2  test/test_helper.rb
@@ -29,6 +29,8 @@ module Fixtures
# when you commit.
if ENV['TM_PID']
MiniTest::Unit.runner.reporters << MiniTest::Reporters::RubyMateReporter.new
+elsif ENV["RM_INFO"]
+ MiniTest::Unit.runner.reporters << MiniTest::Reporters::RubyMineReporter.new
else
# MiniTest::Unit.runner.reporters << MiniTest::Reporters::DefaultReporter.new
# MiniTest::Unit.runner.reporters << MiniTest::Reporters::SpecReporter.new
Something went wrong with that request. Please try again.