OutputToStdout and OutputToStderr matchers #410

Merged
merged 25 commits into from Jan 16, 2014

Conversation

Projects
None yet
4 participants
Contributor

lucapette commented Jan 5, 2014

This pull-request is a very raw draft of how I would like to implement two new built-in matchers that have been discussed in #399, since @matthias-guenther said he wouldn't have the time to move on with it I thought it would have been great to try to contribute to my favourite Ruby open-source project.

What I did so far is implementing the matchers following the specs suggested by @myronmarston to a point all the specs I have are green on MRI 2.1.0 (not sure about other implementations but travis-ci will answer this question soon).

Since it's the first time I try to contribute to RSpec please bare with me because I have some questions:

  • If I understood it correctly we need alias methods (and tests for them) for output_to_stdout and output_to_stderr like output_from_stdout in order to support composition of the matchers.
  • We need cukes because all the matchers have them, for documentation reasons if I understood it.
  • I'm not sure you like the fact we have one OutputToStream class handling both $stdout and stderr. I did it in order to fight a clear duplication. What I don't like about this approach is the way we find out which stream is requested (@stream == $stdout for example).
  • I can't say I like the naming of all the things. But this is a problem I always have, I'm never satisfied with names. Never. It's always a trade-off for me and I'd like to know if you like the one we currently have.
  • In the current implementation we have the shared examples for both matchers in the existing file we have for a more general shared example. I can live with this but I can't say if it's OK with you.
  • A problem connected with the previous is that now we have two different spec files for the matchers but we have one file (with a different name) with the actual implementation.
  • I think we're covering all the basic cases but since it's the first time I'm trying to test a testing framework (that's so nicely meta btw) that I can't say the coverage we have it's enough. I added some tests to the one suggests in #399 and changed the wording a bit.

Sorry for asking so many questions :)

@JonRowe JonRowe and 2 others commented on an outdated diff Jan 5, 2014

lib/rspec/matchers/built_in/output_to_stream.rb
+ if @expected
+ "expected block to output #{@expected.inspect} to #{formatted_stream}, but output #{formatted_actual}"
+ else
+ "expected block to output to #{formatted_stream}, but did not"
+ end
+ end
+
+ def failure_message_when_negated
+ if @expected
+ "expected block to not output #{@expected.inspect} to #{formatted_stream}, but did"
+ else
+ "expected block to not output to #{formatted_stream}, but did"
+ end
+ end
+
+ private
@JonRowe

JonRowe Jan 5, 2014

Owner

We like to indent private / public inline with class/end so this should be dedented two spaces.

@lucapette

lucapette Jan 6, 2014

Contributor

Will do. Would you mind telling me the reasoning you have for it? Fortunately vim-ruby has a nice feature to help not getting wrong it again.

@myronmarston

myronmarston Jan 6, 2014

Owner

Semantically, private affects method definitions below it, so it's nice to have the method defs indented below them. However, I find the rails convention to be ugly:

class Foo
  def public_method
  end

  private

    def private_method
    end
end

We like putting private at the same indentation level as the class to be the best of both worlds: it keeps all method defs at the same level of indentation, while visually showing that the methods below private are affected by it:

class Foo
  def public_method
  end

private

  def private_method
  end
end
@lucapette

lucapette Jan 6, 2014

Contributor

I see, pretty interesting position about this. Thank you for the explanation.

@myronmarston

myronmarston Jan 6, 2014

Owner

Our of curiosity, what is your preferred convention?

@lucapette

lucapette Jan 6, 2014

Contributor

I don't really have a favourite one. But I do dislike Rails convention. The extra-level of indentation drives me crazy :)

@JonRowe JonRowe and 2 others commented on an outdated diff Jan 5, 2014

lib/rspec/matchers/built_in/output_to_stream.rb
+ $stderr = captured_stderr
+
+ block.call
+
+ orig_stdout == @stream ? captured_stdout.string : captured_stderr.string
+ ensure
+ $stdout = orig_stdout
+ $stderr = orig_stderr
+ end
+
+ def captured?
+ @actual.length > 0
+ end
+
+ def formatted_stream
+ @stream == $stdout ? "stdout" : "stderr"
@JonRowe

JonRowe Jan 5, 2014

Owner

This doesn't cater for when @stream is neither $stdout or $stderr. How about using a case statement with the default just being inspect?

@myronmarston

myronmarston Jan 6, 2014

Owner

I'd rather this just be a stream_name attribute that is initialized by a value passed from output_to_stdout and output_to_stderr:

def output_to_stdout(expected=nil)
  BuiltIn::OutputToStream.new($stdout, :stdout, expected)
end

def output_to_stderr(expected=nil)
  BuiltIn::OutputToStream.new($stderr, :stderr, expected)
end

Then there's no need for this implicit inference logic.

@myronmarston

myronmarston Jan 6, 2014

Owner

Nevermind on that idea: I came up with a better one (see my comment below capture_stream...): make a base class and two subclasses -- one for stdout, one for stderr.

@lucapette

lucapette Jan 6, 2014

Contributor

I thought of having two sub-classes at first but then I went for a simpler (raw) version because I wasn't convinced enough you would like it. Now I am.

@JonRowe JonRowe and 1 other commented on an outdated diff Jan 5, 2014

lib/rspec/matchers/built_in/output_to_stream.rb
+
+ def failure_message
+ if @expected
+ "expected block to output #{@expected.inspect} to #{formatted_stream}, but output #{formatted_actual}"
+ else
+ "expected block to output to #{formatted_stream}, but did not"
+ end
+ end
+
+ def failure_message_when_negated
+ if @expected
+ "expected block to not output #{@expected.inspect} to #{formatted_stream}, but did"
+ else
+ "expected block to not output to #{formatted_stream}, but did"
+ end
+ end
@JonRowe

JonRowe Jan 5, 2014

Owner

It seems like failuare_message, failure_message_when_negated could be DRYed up a bit, mostly thinking the conditional, how about wrapping the conditional into a method:

"... #{formatted_expected} ... to #{formatted_stream}"

def formatted_expected
  "#{@expected.inspect} " unless @expected
end
@lucapette

lucapette Jan 6, 2014

Contributor

👍

@JonRowe JonRowe and 1 other commented on an outdated diff Jan 5, 2014

spec/rspec/matchers/built_in/output_to_stderr_spec.rb
@@ -0,0 +1,11 @@
+require 'spec_helper'
+
+module RSpec
+ module Matchers
+ describe "output_to_stderr matcher" do
+ include_examples "output_to_stream", :stderr do
+ let(:stream) { $stderr }
+ end
+ end
+ end
+end
@JonRowe

JonRowe Jan 5, 2014

Owner

For clarity it might be better to combine the two spec files into output_to_stream_spec.rb and include the shared examples there...

@myronmarston

myronmarston Jan 6, 2014

Owner

Yep, that would improve things, I think, given there's one common implementation file. We've done that for others (e.g. yield_spec.rb).

Owner

JonRowe commented Jan 5, 2014

Hey thanks! This is a good start, I left a few minor comments :)

@myronmarston myronmarston and 1 other commented on an outdated diff Jan 6, 2014

lib/rspec/matchers/built_in/output_to_stream.rb
@@ -0,0 +1,64 @@
+module RSpec
@myronmarston

myronmarston Jan 6, 2014

Owner

This file needs a require 'stringio' since it relies upon that piece of stdlib.

@lucapette

lucapette Jan 6, 2014

Contributor

👍

@myronmarston myronmarston and 1 other commented on an outdated diff Jan 6, 2014

lib/rspec/matchers/built_in/output_to_stream.rb
+ orig_stdout == @stream ? captured_stdout.string : captured_stderr.string
+ ensure
+ $stdout = orig_stdout
+ $stderr = orig_stderr
+ end
+
+ def captured?
+ @actual.length > 0
+ end
+
+ def formatted_stream
+ @stream == $stdout ? "stdout" : "stderr"
+ end
+
+ def formatted_actual
+ captured? ? @actual.inspect : "nothing"
@myronmarston

myronmarston Jan 6, 2014

Owner

This is formatted funny: two spaces of indention, please.

@lucapette

lucapette Jan 6, 2014

Contributor

it's funny indeed. I'll be more careful.

@myronmarston myronmarston commented on an outdated diff Jan 6, 2014

spec/support/shared_examples.rb
+
+ context "expect { ... }.not_to #{matcher_method} (with no arg)" do
+ it "passes if the block does not output to #{stream_name}" do
+ expect { }.not_to matcher
+ end
+
+ it "fails if the block outputs to #{stream_name}" do
+ expect {
+ expect { stream.puts 'foo' }.not_to matcher
+ }.to fail_with("expected block to not output to #{stream_name}, but did")
+ end
+ end
+
+ context "expect { ... }.to #{matcher_method}('string')" do
+ it "passes if the block outputs that string to #{stream_name}" do
+ expect { stream.puts 'foo' }.to matcher("foo\n")
@myronmarston

myronmarston Jan 6, 2014

Owner

The extra \n is a little jarring at first but I see why it's needed. What do you think about changing stream.puts to stream.print? Then it's not needed.

@myronmarston myronmarston commented on an outdated diff Jan 6, 2014

spec/support/shared_examples.rb
+ expect { }.not_to matcher
+ end
+
+ it "fails if the block outputs to #{stream_name}" do
+ expect {
+ expect { stream.puts 'foo' }.not_to matcher
+ }.to fail_with("expected block to not output to #{stream_name}, but did")
+ end
+ end
+
+ context "expect { ... }.to #{matcher_method}('string')" do
+ it "passes if the block outputs that string to #{stream_name}" do
+ expect { stream.puts 'foo' }.to matcher("foo\n")
+ end
+
+ it "fails if the block does not outputs to #{stream_name}" do
@myronmarston

myronmarston Jan 6, 2014

Owner

s/outputs/output/

@myronmarston myronmarston and 1 other commented on an outdated diff Jan 6, 2014

lib/rspec/matchers/built_in/output_to_stream.rb
+
+ def capture_stream(block)
+ captured_stdout, captured_stderr = StringIO.new, StringIO.new
+
+ orig_stdout = $stdout
+ orig_stderr = $stderr
+ $stdout = captured_stdout
+ $stderr = captured_stderr
+
+ block.call
+
+ orig_stdout == @stream ? captured_stdout.string : captured_stderr.string
+ ensure
+ $stdout = orig_stdout
+ $stderr = orig_stderr
+ end
@myronmarston

myronmarston Jan 6, 2014

Owner

Your implementation here always captures both stdout and stderr. Besides the extra work this does, it feels like a bug to me that in a spec like this:

expect {
  puts "calling the method"
  the_method
}.to output_to_stderr("foo")

...my debugging output of "calling the method" is silenced. I've only told RSpec only to look at stderr, so why should it mess with stdout?

So, if you make a base class and two subclasses (one for stdout, one for stderr), you can implement the capture_stream method to capture only the appropriate stream.

@JonRowe

JonRowe Jan 6, 2014

Owner

I had this concern initially too, I'd like to see this work for any stream, and then be setup for the individual $stdout / $stderr.

@myronmarston myronmarston commented on an outdated diff Jan 6, 2014

spec/support/shared_examples.rb
+ context "expect { ... }.to_not #{matcher_method}('string')" do
+ it "passes if the block outputs a different string to #{stream_name}" do
+ expect { stream.puts 'food' }.to_not matcher("foo\n")
+ end
+
+ it "passes if the block does not output to #{stream_name}" do
+ expect { }.to_not matcher('foo')
+ end
+
+ it "fails if the block outputs the same string to #{stream_name}" do
+ expect {
+ expect { stream.puts 'foo' }.to_not matcher("foo\n")
+ }.to fail_with("expected block to not output \"foo\\n\" to #{stream_name}, but did")
+ end
+ end
+end
@myronmarston

myronmarston Jan 6, 2014

Owner

Some additional specs I'd like to see passing (I'll leave the implementation as an exercise to you!):

  context "expect { ... }.to #{matcher_method}(/regex/)" do
    it "passes if the block outputs a string to #{stream_name} that matches the regex" do
      expect { stream.puts 'food' }.to matcher(/foo/)
    end
    it "fails if the block does not output to #{stream_name}"
    it "fails if the block outputs a string to #{stream_name} that does not match"
  end

  context "expect { ... }.to #{matcher_method}(/regex/)" do
    # similar specs needed
  end

  context "expect { ... }.to #{matcher_method}(matcher)" do
    it "passes if the block outputs a string to #{stream_name} that passes the given matcher" do
      expect { stream.puts 'foo' }.to matcher(a_string_starting_with("f"))
    end
    # more specs needed (including failing examples)
  end

  context "expect { ... }.not_to #{matcher_method}(matcher)" do
    # similar specs needed
  end

To get these to pass you'll need to leverage the Composable mixin (already included in BaseMatcher):

  • Use values_match? to do the string matching (it'll match exact strings, against a regex or against any matcher).
  • Use description_of in failure messages to get good failure output when matchers are passed.
Owner

myronmarston commented Jan 6, 2014

If I understood it correctly we need alias methods (and tests for them) for output_to_stdout and output_to_stderr like output_from_stdout in order to support composition of the matchers.

To be consistent with the other matcher aliases, I think the alias we want is a_block_outputting_to_stdout and a_block_outputting_to_stderr. That will allow it to be used in matcher expressions like:

expect(manager.callback_blocks).to include( a_block_outputting_to_stdout("foo") )

The matcher also needs to be able to accept a matchers as an argument (rather than just a string) -- I left some more detail about that in an inline comment.

This also needs the it_behaves_like "an RSpec matcher" shared example group applied to these two matchers -- we use that on all built in matchers to ensure some consistent things about all matchers.

BTW, I think that there's some potential gotchas around usage of these matchers. There are ways to output to stdout and stderr that this matcher won't be able to intercept:

  • Using STDOUT.puts or STDERR.puts.
  • Storing a reference to $stdout and $stderr before the matcher is used:
class MyClass
  def initialize(log_to=$stdout)
    @log_to = log_to
  end

  def do_it
    @log_to.print "I did it"
  end
end

describe MyClass do
  it "logs to stdout" do
    instance = MyClass.new
    expect { instance.do_it }.to output_to_stdout("I did it")
  end
end

I'm not sure that we can or should do anything about these gotchas...but it would be good to document them with specs (e.g. demonstrating known, documented cases where this matcher cannot work properly) and by adding a @note to the YARD docs for the matcher methods and the cuke file. (Speaking of which, yes, please do add that cuke!).

Thanks for your hard work on this @lucapette -- this is going to be a nice addition to rspec-expectations :).

Contributor

lucapette commented Jan 6, 2014

Thank you the fantastic feedback! I'll happily do the changes you asked for. Everything makes sense to me. I'll come back as soon as possible with code/questions!

Thanks to @lucapette, @myronmarston, and @JonRowe, it's such a pleasure to read about your interesting discussions and thoughts about how to solve this problem.

Contributor

lucapette commented Jan 12, 2014

@myronmarston @JonRowe I finally found some hours to add more code/docs based on your feedback. The only thing I'm not sure is my English in documentation and in the cukes, being a non-native speaker doesn't help. I tried to clean up the history of the branch a bit but I'm not sure you want all those commits or you prefer me to squash them in one commit, please let me know what you prefer.

So far I like the output but I honestly can't say if the branch can be considered ready. I'll gladly follow your feedback!

@myronmarston myronmarston commented on an outdated diff Jan 13, 2014

spec/rspec/matchers/built_in/output_to_stream_spec.rb
+ it "passes if the block does not output a string to #{stream_name} that passes the given matcher" do
+ expect { stream.print 'foo' }.to_not matcher(a_string_starting_with("b"))
+ end
+
+ it "fails if the block outputs a string to #{stream_name} that passes the given matcher" do
+ expect {
+ expect { stream.print 'foo' }.to_not matcher(a_string_starting_with("f"))
+ }.to fail_with("expected block to not output a string starting with \"f\" to #{stream_name}, but did")
+ end
+ end
+end
+
+module RSpec
+ module Matchers
+ describe "output_to_stderr matcher" do
+ it_behaves_like "an RSpec matcher", :valid_value => lambda { $stderr.print 'foo' }, :invalid_value => lambda {} do
@myronmarston

myronmarston Jan 13, 2014

Owner

This line is apparently a syntax error on 1.8 (see the failing travis build). It's not obvious to me what about it causes the failure; maybe it's the lambda { } do bit at the end (as that could be interpretted as passing two blocks to lambda). Maybe you need to use parens (e.g. ( after it_behaves_like and ) before do)?

@myronmarston myronmarston commented on an outdated diff Jan 13, 2014

spec/rspec/matchers/built_in/output_to_stream_spec.rb
+end
+
+module RSpec
+ module Matchers
+ describe "output_to_stderr matcher" do
+ it_behaves_like "an RSpec matcher", :valid_value => lambda { $stderr.print 'foo' }, :invalid_value => lambda {} do
+ let(:matcher) { output_to_stderr }
+ end
+
+ include_examples "output_to_stream", :stderr do
+ let(:stream) { $stderr }
+ end
+ end
+
+ describe "output_to_stdout matcher" do
+ it_behaves_like "an RSpec matcher", :valid_value => lambda { $stdout.print 'foo' }, :invalid_value => lambda {} do
@myronmarston

myronmarston Jan 13, 2014

Owner

Ditto here.

@myronmarston myronmarston commented on an outdated diff Jan 13, 2014

features/built_in_matchers/output_to_stream.feature
+ """ruby
+
+ describe "output_to_stdout matcher" do
+ specify { expect { $stdout.print('foo') }.to output_to_stdout }
+ specify { expect { $stdout.print('foo') }.to output_to_stdout('foo') }
+ specify { expect { $stdout.print('foo') }.to output_to_stdout(/foo/) }
+ specify { expect { }.to_not output_to_stdout }
+ specify { expect { $stdout.print('foo') }.to_not output_to_stdout('bar') }
+ specify { expect { $stdout.print('foo') }.to_not output_to_stdout(/bar/) }
+
+ # deliberate failures
+ specify { expect { }.to output_to_stdout }
+ specify { expect { }.to output_to_stdout('foo') }
+ specify { expect { $stdout.print('foo') }.to_not output_to_stdout }
+ specify { expect { $stdout.print('foo') }.to output_to_stdout('bar') }
+ specify { expect { $stdout.print('foo') }.to output_to_stdout(/bar/) }
@myronmarston

myronmarston Jan 13, 2014

Owner

I think I'd drop the $stdout. bit and just use a bare print -- which goes to $stdout automatically. I think most people use puts/print as bare keywords so you might as well do the same, especially because it's shorter :).

@myronmarston myronmarston commented on an outdated diff Jan 13, 2014

features/built_in_matchers/output_to_stream.feature
+ """ruby
+
+ describe "output_to_stderr matcher" do
+ specify { expect { $stderr.print('foo') }.to output_to_stderr }
+ specify { expect { $stderr.print('foo') }.to output_to_stderr('foo') }
+ specify { expect { $stderr.print('foo') }.to output_to_stderr(/foo/) }
+ specify { expect { }.to_not output_to_stderr }
+ specify { expect { $stderr.print('foo') }.to_not output_to_stderr('bar') }
+ specify { expect { $stderr.print('foo') }.to_not output_to_stderr(/bar/) }
+
+ # deliberate failures
+ specify { expect { }.to output_to_stderr }
+ specify { expect { }.to output_to_stderr('foo') }
+ specify { expect { $stderr.print('foo') }.to_not output_to_stderr }
+ specify { expect { $stderr.print('foo') }.to output_to_stderr('bar') }
+ specify { expect { $stderr.print('foo') }.to output_to_stderr(/bar/) }
@myronmarston

myronmarston Jan 13, 2014

Owner

Similarly here, you might use bare warn as that will go to stderr automatically, and is the most common way people write to stderr, I think.

@myronmarston myronmarston commented on an outdated diff Jan 13, 2014

lib/rspec/matchers.rb
@@ -598,6 +598,50 @@ def match_array(items)
contain_exactly(*items)
end
+ # With no args, passes if the block outputs to $stdout.
+ # With a string, passes if the blocks outputs that specific string to $stdout.
+ # With a regexp, passes if the blocks outputs a string to $stdout that matches.
+ #
+ # @example
+ #
+ # expect { $stdout.print 'foo' }.to output_to_stdout
+ # expect { $stdout.print 'foo' }.to output_to_stdout('foo')
+ # expect { $stdout.print 'foo' }.to output_to_stdout(/foo/)
+ #
+ # expect { do_something }.to_not output_to_stdout
+ #
+ # @note This matcher won't be able to intercept output to STDOUT when the
+ # explicit STDOUT.puts 'foo' is used or in case a reference to STDOUT is
+ # stored before the matcher is used.
@myronmarston

myronmarston Jan 13, 2014

Owner
  • Please wrap code elements in backticks as it renders better (e.g. around STDOUT and STDOUT.puts ...)..
  • The second case is when a reference to $stderr is stored before, not when a reference to STDOUT is stored (STDOUT will never work, regardless of if the ref is stored before or after...$stdout works as long as the user isn't using a prior reference to it).

@myronmarston myronmarston commented on an outdated diff Jan 13, 2014

lib/rspec/matchers.rb
@@ -598,6 +598,50 @@ def match_array(items)
contain_exactly(*items)
end
+ # With no args, passes if the block outputs to $stdout.
+ # With a string, passes if the blocks outputs that specific string to $stdout.
+ # With a regexp, passes if the blocks outputs a string to $stdout that matches.
+ #
+ # @example
+ #
+ # expect { $stdout.print 'foo' }.to output_to_stdout
+ # expect { $stdout.print 'foo' }.to output_to_stdout('foo')
+ # expect { $stdout.print 'foo' }.to output_to_stdout(/foo/)
@myronmarston

myronmarston Jan 13, 2014

Owner

Same suggestion as in the cuke: I'd drop the $stdout. prefix to print since print goes to stdout automatically.

@myronmarston myronmarston and 1 other commented on an outdated diff Jan 13, 2014

lib/rspec/matchers.rb
@@ -598,6 +598,50 @@ def match_array(items)
contain_exactly(*items)
end
+ # With no args, passes if the block outputs to $stdout.
+ # With a string, passes if the blocks outputs that specific string to $stdout.
+ # With a regexp, passes if the blocks outputs a string to $stdout that matches.
@myronmarston

myronmarston Jan 13, 2014

Owner

I'd change this last line to

# With a regexp or matcher, passes if the blocks outputs a string to $stdout that matches.
@JonRowe

JonRowe Jan 13, 2014

Owner

Does using backticks make this more readable in YARD?

@myronmarston myronmarston commented on an outdated diff Jan 13, 2014

lib/rspec/matchers.rb
+ #
+ # expect { do_something }.to_not output_to_stdout
+ #
+ # @note This matcher won't be able to intercept output to STDOUT when the
+ # explicit STDOUT.puts 'foo' is used or in case a reference to STDOUT is
+ # stored before the matcher is used.
+ def output_to_stdout(expected=nil)
+ BuiltIn::OutputToStdout.new(expected)
+ end
+ alias_matcher :a_block_outputting_to_stdout, :output_to_stdout do |desc|
+ desc.sub('output', 'a block outputting')
+ end
+
+ # With no args, passes if the block outputs to $stderr.
+ # With a string, passes if the blocks outputs that specific string to $stderr.
+ # With a regexp, passes if the blocks outputs a string to $stderr that matches.
@myronmarston

myronmarston Jan 13, 2014

Owner

Same here: please mention regexp or matcher.

@myronmarston myronmarston commented on an outdated diff Jan 13, 2014

lib/rspec/matchers.rb
+ def output_to_stdout(expected=nil)
+ BuiltIn::OutputToStdout.new(expected)
+ end
+ alias_matcher :a_block_outputting_to_stdout, :output_to_stdout do |desc|
+ desc.sub('output', 'a block outputting')
+ end
+
+ # With no args, passes if the block outputs to $stderr.
+ # With a string, passes if the blocks outputs that specific string to $stderr.
+ # With a regexp, passes if the blocks outputs a string to $stderr that matches.
+ #
+ # @example
+ #
+ # expect { $stderr.print 'foo' }.to output_to_stderr
+ # expect { $stderr.print 'foo' }.to output_to_stderr('foo')
+ # expect { $stderr.print 'foo' }.to output_to_stderr(/foo/)
@myronmarston

myronmarston Jan 13, 2014

Owner

And again here warn can be used, I think.

@myronmarston myronmarston commented on an outdated diff Jan 13, 2014

lib/rspec/matchers.rb
+
+ # With no args, passes if the block outputs to $stderr.
+ # With a string, passes if the blocks outputs that specific string to $stderr.
+ # With a regexp, passes if the blocks outputs a string to $stderr that matches.
+ #
+ # @example
+ #
+ # expect { $stderr.print 'foo' }.to output_to_stderr
+ # expect { $stderr.print 'foo' }.to output_to_stderr('foo')
+ # expect { $stderr.print 'foo' }.to output_to_stderr(/foo/)
+ #
+ # expect { do_something }.to_not output_to_stderr
+ #
+ # @note This matcher won't be able to intercept output to STDERR when the
+ # explicit STDERR.puts 'foo' is used or in case a reference to STDERR is
+ # stored before the matcher is used.
@myronmarston

myronmarston Jan 13, 2014

Owner

And same feedback here as well :).

@myronmarston myronmarston and 3 others commented on an outdated diff Jan 13, 2014

lib/rspec/matchers/built_in/output_to_stream.rb
+ "expected block to #{description}, #{actual_description}"
+ end
+
+ def failure_message_when_negated
+ "expected block to not #{description}, but did"
+ end
+
+ def description
+ expected = case @expected
+ when Regexp then "a string matching #{@expected.inspect}"
+ when AliasedMatcher then description_of(@expected)
+ else
+ @expected.inspect
+ end
+ @expected ? "output #{expected} to #{@stream_name}" : "output to #{@stream_name}"
+ end
@myronmarston

myronmarston Jan 13, 2014

Owner

I don't think I'd special case Regexp in the description. As far as I know, we don't do that in any of the other matchers. If the user wants the description/failure message to say a string matching..., they can pass the a_string_matching(/regexp/) matcher rather than just using /regexp/). As a full rspec matcher, it has an appropriate description. OTOH, some users may find the a string matching bit to be unnecessary noise in the failure message, and they can use regexp rather than the matcher to get the slightly more terse failure message. I think it still reads fine as a raw regexp.

Anyhow, once you do that, you can just use "output #{description_of @expected}..." -- no need for the case statement at all because description_of is smart about using the description vs. inspect. In fact, I just realized a concrete bug with your case statement: non-aliased matchers will get their inspect output included in the description rather than their description. User may define their own custom matchers that are already phrased appropriately to be passed as an arg, with no need to alias it.

@myronmarston

myronmarston Jan 13, 2014

Owner

Seeing your description (e.g. output "foo" to stdout) makes me think that maybe our matcher methods are sub-optimal. What do you think about changing to this:

expect { }.to output("foo").to_stdout
expect { }.to output(/bar/).to_stderr

...rather than:

expect { }.to output_to_stdout("foo")
expect { }.to output_to_stderr(/bar/)

The former reads more closely to how we actually say it (and how the description reads), which is nice. Also, it opens up new possibilities like this:

expect { }.to output("foo").to(some_alternate_stream)

(I don't think we necessarily should support that yet, and that's out of scope for this PR, but I'm just thinking it allows a bit more flexibility in the future to support things like that).

Anyhow, don't change to this new form just yet. I'm really just thinking out loud here.

What do you think @lucapette? @matthias-guenther? @JonRowe? @xaviershay? @samphippen? @alindeman? @soulcutter?

@wikimatze

wikimatze Jan 13, 2014

Hi @myronmarston you are right, in my eyes

expect { }.to output_to_stdout("foo")

sounds to long and is hard to remember. What about the shortform:

expect { }.to stdout("foo")
@myronmarston

myronmarston Jan 13, 2014

Owner

I don't think that reads well at all. Matchers should be verbs, and stdout is not a verb.

@JonRowe

JonRowe Jan 13, 2014

Owner

I like @myronmarston's idea here

expect { }.to output("foo").to_stdout
expect { }.to output(/bar/).to_stderr
@lucapette

lucapette Jan 14, 2014

Contributor

I thought the very same when writing the specs.

expect { }.to output("foo").to_stdout

reads better that what we have now. I would probably do it in a separate PR if you're OK with it. Actually, having the opportunity to do something like

expect { }.to output("foo").to(some_alternate_stream)

looks pretty awesome to me even though the implementation can be tricky I think. And we can do it as a last step of a new pull request (this way I'll get your feedback on the way to the implementation :))

Owner

myronmarston commented Jan 13, 2014

@lucapette -- this is looking very good :). One other suggestion (besides the feedback left above): can you add a scenario to the composing_matchers.feature cuke (and update the narrative to mention these matchers)? We want that to list all the matchers that can receive matcher args. (Of course you may want to hold off on adding that until we settle on the matcher method names given my idea above).

Owner

myronmarston commented Jan 13, 2014

Oh yeah, one other thing I just remembered: I think this matcher should be diffable. That'll come in handy when there are multiline strings.

@JonRowe JonRowe commented on an outdated diff Jan 13, 2014

features/built_in_matchers/output_to_stream.feature
@@ -0,0 +1,76 @@
+Feature: output to stream matchers
+
+ There are two related matchers that allow you to specify whether a block
+ outputs to a stream. With no args, the matcher passes whenever the
+ block-under-test outputs. With a string argument, it passes whenever
+ block-under-test outputs a string that `==` the given string. With a regex,
+ it passes whenever the block-under-test outputs a string that matches the
+ given string.
@JonRowe

JonRowe Jan 13, 2014

Owner

I'd reword this... how about something like:

The `output_to_stdout` and `output_to_stderr` matcher(s) provides a way to assert that
the supplied block has emitted content to either `stdout` or `stderr`.

With no argument the matcher asserts that there has been output, when you pass a
string then it asserts the output is equal (`==`) to that string and if you pass a regular
expression then it asserts the output matches it.

For example:

```Ruby
expect { puts "Some content" }.to output_to_stdout
expect { warn "Specifc content" }.to output_to_stderr('Specific content')
expect { warn "Specifc content" }.to output_to_stderr(/content/)
```.

Contributor

lucapette commented Jan 13, 2014

the feedback is great. Thank you so much for being so patient. I really didn't have the time today to play with it. I'll get back with news as soon as I can.

Owner

myronmarston commented Jan 13, 2014

Thank you so much for being so patient.

Thanks for doing this, and working with our feedback :). RSpec always gets better when new features go through this refining process of the PR/code review cycle!

Contributor

lucapette commented Jan 14, 2014

OK, I found some time during the week, that's rare. I think I applied all the changes you requested. I love this PR/code review cycle.

Owner

myronmarston commented Jan 14, 2014

Thanks, @lucapette!

I would probably do it in a separate PR if you're OK with it.

Normally that would be fine, but we're trying to get 3.0.0.beta2 out real soon, and I don't want to merge this and release with an API we plan to change before 3.0 final. So I'll leave this PR as-is for now, and if we release before you get around to changing it, I can merge this as-is; otherwise, if you start working on changing it before we release you can keep adding it to this PR. Does that sound fine?

Contributor

lucapette commented Jan 14, 2014

It sounds very reasonable to me. So to move forward I have a couple of questions.

Just to be sure we move in the right direction, what we want is something like:

        specify { expect { print('foo') }.to output_to_stdout }
        specify { expect { print('foo') }.to output('foo').to_stdout }
        specify { expect { print('foo') }.to output(/foo/).to_stdout }
        specify { expect { }.to_not output_to_stdout }
        specify { expect { print('foo') }.to_not output('bar').to_stdout }
        specify { expect { print('foo') }.to_not output(/bar/).to_stdout }

Do you have a rough idea of when the release is coming? Just curious, maybe I can try to find a bit more time to work on this.

Owner

myronmarston commented Jan 14, 2014

It's a little weird to have output_to_stdout and output(...).to_stdout. For the case where you aren't specifying what is output (and just that something is), what do you think about this? expect { ... }.to output.to_stdout. That way it keeps the output(...).to_stdout form, but you can choose not to pass an arg to output, which indicates you aren't bothering to specify what is output (which makes sense, since that's what the arg represents).

Do you have a rough idea of when the release is coming? Just curious, maybe I can try to find a bit more time to work on this.

We were hoping to get beta2 out by the end of 2013. There are some pending items left to do, though. @JonRowe has a bunch of those in-flight in rspec-core, so it largely depends on his schedule.

Contributor

lucapette commented Jan 14, 2014

it's weird and actually that's why I asked :) I wasn't sure output.to_stdout would be exactly what we want. I'll try to come back as soon as possible but I can't promise it will be really fast :(. Thank you for everything!

Contributor

lucapette commented Jan 14, 2014

OK, I guess I was in the zone or something and I couldn't stop so I updated the PR to the cleaner API we discussed. Not sure this is exactly what you wanted from the implementation/documentation point of view but at least the API looks pretty good now.

@JonRowe JonRowe commented on an outdated diff Jan 14, 2014

features/built_in_matchers/output_to_stream.feature
@@ -0,0 +1,78 @@
+Feature: output to stream matchers
+
+ The `output` matcher provides a way to assert that the block-under-test
+ has emitted content to either `to_stdout` or `to_stderr`.
@JonRowe

JonRowe Jan 14, 2014

Owner

This has been indented three spaces rather than two...

@JonRowe

JonRowe Jan 14, 2014

Owner

You could also just say 'block under test" without hyphens, but it might be even cleared to say "code in the block"

@JonRowe JonRowe and 1 other commented on an outdated diff Jan 14, 2014

lib/rspec/matchers/built_in/output_to_stream.rb
@@ -0,0 +1,93 @@
+require 'stringio'
+
+module RSpec
+ module Matchers
+ module BuiltIn
+ class OutputToStream < BaseMatcher
+ def initialize(expected)
+ @expected = expected
@JonRowe

JonRowe Jan 14, 2014

Owner

You should set @stream here to a null object, to prevent 'nasty' errors when a developer makes a mistake with the matcher (e.g. just uses output)

@myronmarston

myronmarston Jan 15, 2014

Owner

Actually, IMO, given that we don't intend output to be used on its own w/o to_xyz being chained off of it, we should make matches? raise an error if they didn't call one of the to_ methods.

This is similar to be_within:

https://github.com/rspec/rspec-expectations/blob/79ac9aabe938a25a918c07077cfb9fe1ba222923/lib/rspec/matchers/built_in/be_within.rb#L13

expect(x).to be_within(0.1) will raise an error because you didn't chain of off of it.

@JonRowe

JonRowe Jan 15, 2014

Owner

We should at least set @stream = nil to avoid any potential warnings, also will raising in matches? then trigger an error for @stream.name in the description?

@myronmarston

myronmarston Jan 15, 2014

Owner

Good point, it's probably worth using a null object for that.

Owner

JonRowe commented Jan 14, 2014

Looks good, I just have a concern about @stream not being setup upon initialise, which opens the door for unexpected errors / obtuse error messages. I'd add a null object that implements message and raises an appropriate error in capture (ExpectationNotMet with a message explaining what they did wrong maybe?). I'd also be good to have a spec covering that eventuality.

@myronmarston myronmarston commented on an outdated diff Jan 15, 2014

features/built_in_matchers/output_to_stream.feature
@@ -0,0 +1,78 @@
+Feature: output to stream matchers
+
+ The `output` matcher provides a way to assert that the block-under-test
+ has emitted content to either `to_stdout` or `to_stderr`.
+
+ With no argument the matcher asserts that there has been output, when you
+ pass a string then it asserts the output is equal (`==`) to that string and
+ if you pass a regular expression then it asserts the output matches it. With
+ a regex or a matcher, it passes whenever the block-under-test outputs a
+ string that matches the given string.
+
+ * `output.to_stdout` matches if the block-under-test outputs to
+ $stdout.
+
+ * `output.to_stderr` matchets if the block-under-test outputs to
@myronmarston

myronmarston Jan 15, 2014

Owner

s/matchets/matches/

@myronmarston myronmarston commented on an outdated diff Jan 15, 2014

lib/rspec/matchers.rb
+ #
+ # expect { warn('foo') }.to output.to_stderr
+ # expect { warn('foo') }.to output('foo').to_stderr
+ # expect { warn('foo') }.to output(/foo/).to_stderr
+ #
+ # expect { do_something }.to_not output.to_stderr
+ #
+ # @note This matcher won't be able to intercept output to `STDOUT` or `STDERR`
+ # when the reference in a constant is used, like in `STDOUT.puts 'foo'`, or in
+ # case a reference to `$stdout` or `$stderr` is stored before the matcher is used.
+ def output(expected=nil)
+ BuiltIn::OutputToStream.new(expected)
+ end
+ alias_matcher :a_block_outputting, :output do |desc|
+ desc.sub('output', 'a block outputting')
+ end
@myronmarston

myronmarston Jan 15, 2014

Owner

Is the block still necessary here? It seems like the description more closely matches the method name and maybe the block isn't needed anymore.

@myronmarston myronmarston commented on an outdated diff Jan 15, 2014

lib/rspec/matchers/built_in.rb
@@ -26,6 +26,7 @@ module BuiltIn
autoload :Match, 'rspec/matchers/built_in/match'
autoload :NegativeOperatorMatcher, 'rspec/matchers/built_in/operators'
autoload :OperatorMatcher, 'rspec/matchers/built_in/operators'
+ autoload :OutputToStream, 'rspec/matchers/built_in/output_to_stream'
@myronmarston

myronmarston Jan 15, 2014

Owner

Given the RSpec::Matchers method is now output, I think I'd like the class name to be BuiltIn::Output and the file name to build_in/output to match. For all the other matchers I think they match exactly and it's nice to be consistent. The cuke and spec file should be updated to match as well.

@myronmarston myronmarston commented on an outdated diff Jan 15, 2014

lib/rspec/matchers/built_in/output_to_stream.rb
+
+ def matches?(block)
+ @actual = @stream.capture(block)
+
+ @expected ? values_match?(@expected, @actual) : captured?
+ end
+
+ def to_stdout
+ @stream = CaptureStdout.new
+ self
+ end
+
+ def to_stderr
+ @stream = CaptureStderr.new
+ self
+ end
@myronmarston

myronmarston Jan 15, 2014

Owner

I'm really happy with how this decomposed nicely into a composition-based solution :). This is really clean and well done!

Contributor

lucapette commented Jan 15, 2014

Here I'm again :) I think I applied all the feedback you gave me. I'm just not sure about the name of the null object, it's currently NullCapture (I'd go for something like CaptureNothing I think but I see we have a NullSolution somewhere so I kept the prefix).
On the same topic, aka naming is hard, there is the message error for when the user doesn't set any stream expectation. Not sure that's good enough.
Anyway I'm pretty sure that if there is room for improvement you'll point me into the right direction! Thank you for taking care of RSpec.

myronmarston merged commit ffd25a8 into rspec:master Jan 16, 2014

1 check passed

default The Travis CI build passed
Details

@myronmarston myronmarston added a commit that referenced this pull request Jan 16, 2014

@myronmarston myronmarston Tweak a few things post-merge #410.
- Add changelog entry.
- Copy the explanation from the YARD comments to
  the cuke as I think it was worded a bit better.
- Rephrase the note a bit to explain why.
- Since the the stream capturers are stateless,
  use singleton modules for them. Less garbage for the GC!
- Rename ivar to `@stream_capturer` since it's not a stream
  itself.
- Improve error message when the user forgets to chain
  `to_stdout` or `to_stderr` off of it.
- Still provide a description in this case.
4a8bb9f
Owner

myronmarston commented Jan 16, 2014

Thanks, @lucapette! I merged and I also applied a few final tweaks in 4a8bb9f if you want to take a look. This is going to be a nice new feature in RSpec 3 :).

Contributor

lucapette commented Jan 16, 2014

thank you @myronmarston and @JonRowe for helping me writing this feature, and thanks to @matthias-guenther for the original idea. It was really a pleasure to contribute to RSpec! I hope I'll find a way to help more.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment