New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OutputToStdout and OutputToStderr matchers #410

Merged
merged 25 commits into from Jan 16, 2014

Conversation

Projects
None yet
4 participants
@lucapette
Contributor

lucapette commented Jan 5, 2014

This pull-request is a very raw draft of how I would like to implement two new built-in matchers that have been discussed in #399, since @matthias-guenther said he wouldn't have the time to move on with it I thought it would have been great to try to contribute to my favourite Ruby open-source project.

What I did so far is implementing the matchers following the specs suggested by @myronmarston to a point all the specs I have are green on MRI 2.1.0 (not sure about other implementations but travis-ci will answer this question soon).

Since it's the first time I try to contribute to RSpec please bare with me because I have some questions:

  • If I understood it correctly we need alias methods (and tests for them) for output_to_stdout and output_to_stderr like output_from_stdout in order to support composition of the matchers.
  • We need cukes because all the matchers have them, for documentation reasons if I understood it.
  • I'm not sure you like the fact we have one OutputToStream class handling both $stdout and stderr. I did it in order to fight a clear duplication. What I don't like about this approach is the way we find out which stream is requested (@stream == $stdout for example).
  • I can't say I like the naming of all the things. But this is a problem I always have, I'm never satisfied with names. Never. It's always a trade-off for me and I'd like to know if you like the one we currently have.
  • In the current implementation we have the shared examples for both matchers in the existing file we have for a more general shared example. I can live with this but I can't say if it's OK with you.
  • A problem connected with the previous is that now we have two different spec files for the matchers but we have one file (with a different name) with the actual implementation.
  • I think we're covering all the basic cases but since it's the first time I'm trying to test a testing framework (that's so nicely meta btw) that I can't say the coverage we have it's enough. I added some tests to the one suggests in #399 and changed the wording a bit.

Sorry for asking so many questions :)

@JonRowe

View changes

Show outdated Hide outdated lib/rspec/matchers/built_in/output_to_stream.rb Outdated
@JonRowe

View changes

Show outdated Hide outdated lib/rspec/matchers/built_in/output_to_stream.rb Outdated
@JonRowe

View changes

Show outdated Hide outdated lib/rspec/matchers/built_in/output_to_stream.rb Outdated
@JonRowe

View changes

Show outdated Hide outdated spec/rspec/matchers/built_in/output_to_stderr_spec.rb Outdated
@JonRowe

This comment has been minimized.

Show comment
Hide comment
@JonRowe

JonRowe Jan 5, 2014

Member

Hey thanks! This is a good start, I left a few minor comments :)

Member

JonRowe commented Jan 5, 2014

Hey thanks! This is a good start, I left a few minor comments :)

@myronmarston

View changes

Show outdated Hide outdated lib/rspec/matchers/built_in/output_to_stream.rb Outdated
@myronmarston

View changes

Show outdated Hide outdated lib/rspec/matchers/built_in/output_to_stream.rb Outdated
@myronmarston

View changes

Show outdated Hide outdated spec/support/shared_examples.rb Outdated
@myronmarston

View changes

Show outdated Hide outdated spec/support/shared_examples.rb Outdated
@myronmarston

View changes

Show outdated Hide outdated lib/rspec/matchers/built_in/output_to_stream.rb Outdated
@myronmarston

View changes

Show outdated Hide outdated spec/support/shared_examples.rb Outdated
@myronmarston

This comment has been minimized.

Show comment
Hide comment
@myronmarston

myronmarston Jan 6, 2014

Member

If I understood it correctly we need alias methods (and tests for them) for output_to_stdout and output_to_stderr like output_from_stdout in order to support composition of the matchers.

To be consistent with the other matcher aliases, I think the alias we want is a_block_outputting_to_stdout and a_block_outputting_to_stderr. That will allow it to be used in matcher expressions like:

expect(manager.callback_blocks).to include( a_block_outputting_to_stdout("foo") )

The matcher also needs to be able to accept a matchers as an argument (rather than just a string) -- I left some more detail about that in an inline comment.

This also needs the it_behaves_like "an RSpec matcher" shared example group applied to these two matchers -- we use that on all built in matchers to ensure some consistent things about all matchers.

BTW, I think that there's some potential gotchas around usage of these matchers. There are ways to output to stdout and stderr that this matcher won't be able to intercept:

  • Using STDOUT.puts or STDERR.puts.
  • Storing a reference to $stdout and $stderr before the matcher is used:
class MyClass
  def initialize(log_to=$stdout)
    @log_to = log_to
  end

  def do_it
    @log_to.print "I did it"
  end
end

describe MyClass do
  it "logs to stdout" do
    instance = MyClass.new
    expect { instance.do_it }.to output_to_stdout("I did it")
  end
end

I'm not sure that we can or should do anything about these gotchas...but it would be good to document them with specs (e.g. demonstrating known, documented cases where this matcher cannot work properly) and by adding a @note to the YARD docs for the matcher methods and the cuke file. (Speaking of which, yes, please do add that cuke!).

Thanks for your hard work on this @lucapette -- this is going to be a nice addition to rspec-expectations :).

Member

myronmarston commented Jan 6, 2014

If I understood it correctly we need alias methods (and tests for them) for output_to_stdout and output_to_stderr like output_from_stdout in order to support composition of the matchers.

To be consistent with the other matcher aliases, I think the alias we want is a_block_outputting_to_stdout and a_block_outputting_to_stderr. That will allow it to be used in matcher expressions like:

expect(manager.callback_blocks).to include( a_block_outputting_to_stdout("foo") )

The matcher also needs to be able to accept a matchers as an argument (rather than just a string) -- I left some more detail about that in an inline comment.

This also needs the it_behaves_like "an RSpec matcher" shared example group applied to these two matchers -- we use that on all built in matchers to ensure some consistent things about all matchers.

BTW, I think that there's some potential gotchas around usage of these matchers. There are ways to output to stdout and stderr that this matcher won't be able to intercept:

  • Using STDOUT.puts or STDERR.puts.
  • Storing a reference to $stdout and $stderr before the matcher is used:
class MyClass
  def initialize(log_to=$stdout)
    @log_to = log_to
  end

  def do_it
    @log_to.print "I did it"
  end
end

describe MyClass do
  it "logs to stdout" do
    instance = MyClass.new
    expect { instance.do_it }.to output_to_stdout("I did it")
  end
end

I'm not sure that we can or should do anything about these gotchas...but it would be good to document them with specs (e.g. demonstrating known, documented cases where this matcher cannot work properly) and by adding a @note to the YARD docs for the matcher methods and the cuke file. (Speaking of which, yes, please do add that cuke!).

Thanks for your hard work on this @lucapette -- this is going to be a nice addition to rspec-expectations :).

@lucapette

This comment has been minimized.

Show comment
Hide comment
@lucapette

lucapette Jan 6, 2014

Contributor

Thank you the fantastic feedback! I'll happily do the changes you asked for. Everything makes sense to me. I'll come back as soon as possible with code/questions!

Contributor

lucapette commented Jan 6, 2014

Thank you the fantastic feedback! I'll happily do the changes you asked for. Everything makes sense to me. I'll come back as soon as possible with code/questions!

@wikimatze

This comment has been minimized.

Show comment
Hide comment
@wikimatze

wikimatze Jan 6, 2014

Thanks to @lucapette, @myronmarston, and @JonRowe, it's such a pleasure to read about your interesting discussions and thoughts about how to solve this problem.

wikimatze commented Jan 6, 2014

Thanks to @lucapette, @myronmarston, and @JonRowe, it's such a pleasure to read about your interesting discussions and thoughts about how to solve this problem.

@lucapette

This comment has been minimized.

Show comment
Hide comment
@lucapette

lucapette Jan 12, 2014

Contributor

@myronmarston @JonRowe I finally found some hours to add more code/docs based on your feedback. The only thing I'm not sure is my English in documentation and in the cukes, being a non-native speaker doesn't help. I tried to clean up the history of the branch a bit but I'm not sure you want all those commits or you prefer me to squash them in one commit, please let me know what you prefer.

So far I like the output but I honestly can't say if the branch can be considered ready. I'll gladly follow your feedback!

Contributor

lucapette commented Jan 12, 2014

@myronmarston @JonRowe I finally found some hours to add more code/docs based on your feedback. The only thing I'm not sure is my English in documentation and in the cukes, being a non-native speaker doesn't help. I tried to clean up the history of the branch a bit but I'm not sure you want all those commits or you prefer me to squash them in one commit, please let me know what you prefer.

So far I like the output but I honestly can't say if the branch can be considered ready. I'll gladly follow your feedback!

@lucapette

This comment has been minimized.

Show comment
Hide comment
@lucapette

lucapette Jan 14, 2014

Contributor

OK, I found some time during the week, that's rare. I think I applied all the changes you requested. I love this PR/code review cycle.

Contributor

lucapette commented Jan 14, 2014

OK, I found some time during the week, that's rare. I think I applied all the changes you requested. I love this PR/code review cycle.

@myronmarston

This comment has been minimized.

Show comment
Hide comment
@myronmarston

myronmarston Jan 14, 2014

Member

Thanks, @lucapette!

I would probably do it in a separate PR if you're OK with it.

Normally that would be fine, but we're trying to get 3.0.0.beta2 out real soon, and I don't want to merge this and release with an API we plan to change before 3.0 final. So I'll leave this PR as-is for now, and if we release before you get around to changing it, I can merge this as-is; otherwise, if you start working on changing it before we release you can keep adding it to this PR. Does that sound fine?

Member

myronmarston commented Jan 14, 2014

Thanks, @lucapette!

I would probably do it in a separate PR if you're OK with it.

Normally that would be fine, but we're trying to get 3.0.0.beta2 out real soon, and I don't want to merge this and release with an API we plan to change before 3.0 final. So I'll leave this PR as-is for now, and if we release before you get around to changing it, I can merge this as-is; otherwise, if you start working on changing it before we release you can keep adding it to this PR. Does that sound fine?

@lucapette

This comment has been minimized.

Show comment
Hide comment
@lucapette

lucapette Jan 14, 2014

Contributor

It sounds very reasonable to me. So to move forward I have a couple of questions.

Just to be sure we move in the right direction, what we want is something like:

        specify { expect { print('foo') }.to output_to_stdout }
        specify { expect { print('foo') }.to output('foo').to_stdout }
        specify { expect { print('foo') }.to output(/foo/).to_stdout }
        specify { expect { }.to_not output_to_stdout }
        specify { expect { print('foo') }.to_not output('bar').to_stdout }
        specify { expect { print('foo') }.to_not output(/bar/).to_stdout }

Do you have a rough idea of when the release is coming? Just curious, maybe I can try to find a bit more time to work on this.

Contributor

lucapette commented Jan 14, 2014

It sounds very reasonable to me. So to move forward I have a couple of questions.

Just to be sure we move in the right direction, what we want is something like:

        specify { expect { print('foo') }.to output_to_stdout }
        specify { expect { print('foo') }.to output('foo').to_stdout }
        specify { expect { print('foo') }.to output(/foo/).to_stdout }
        specify { expect { }.to_not output_to_stdout }
        specify { expect { print('foo') }.to_not output('bar').to_stdout }
        specify { expect { print('foo') }.to_not output(/bar/).to_stdout }

Do you have a rough idea of when the release is coming? Just curious, maybe I can try to find a bit more time to work on this.

@myronmarston

This comment has been minimized.

Show comment
Hide comment
@myronmarston

myronmarston Jan 14, 2014

Member

It's a little weird to have output_to_stdout and output(...).to_stdout. For the case where you aren't specifying what is output (and just that something is), what do you think about this? expect { ... }.to output.to_stdout. That way it keeps the output(...).to_stdout form, but you can choose not to pass an arg to output, which indicates you aren't bothering to specify what is output (which makes sense, since that's what the arg represents).

Do you have a rough idea of when the release is coming? Just curious, maybe I can try to find a bit more time to work on this.

We were hoping to get beta2 out by the end of 2013. There are some pending items left to do, though. @JonRowe has a bunch of those in-flight in rspec-core, so it largely depends on his schedule.

Member

myronmarston commented Jan 14, 2014

It's a little weird to have output_to_stdout and output(...).to_stdout. For the case where you aren't specifying what is output (and just that something is), what do you think about this? expect { ... }.to output.to_stdout. That way it keeps the output(...).to_stdout form, but you can choose not to pass an arg to output, which indicates you aren't bothering to specify what is output (which makes sense, since that's what the arg represents).

Do you have a rough idea of when the release is coming? Just curious, maybe I can try to find a bit more time to work on this.

We were hoping to get beta2 out by the end of 2013. There are some pending items left to do, though. @JonRowe has a bunch of those in-flight in rspec-core, so it largely depends on his schedule.

@lucapette

This comment has been minimized.

Show comment
Hide comment
@lucapette

lucapette Jan 14, 2014

Contributor

it's weird and actually that's why I asked :) I wasn't sure output.to_stdout would be exactly what we want. I'll try to come back as soon as possible but I can't promise it will be really fast :(. Thank you for everything!

Contributor

lucapette commented Jan 14, 2014

it's weird and actually that's why I asked :) I wasn't sure output.to_stdout would be exactly what we want. I'll try to come back as soon as possible but I can't promise it will be really fast :(. Thank you for everything!

@lucapette

This comment has been minimized.

Show comment
Hide comment
@lucapette

lucapette Jan 14, 2014

Contributor

OK, I guess I was in the zone or something and I couldn't stop so I updated the PR to the cleaner API we discussed. Not sure this is exactly what you wanted from the implementation/documentation point of view but at least the API looks pretty good now.

Contributor

lucapette commented Jan 14, 2014

OK, I guess I was in the zone or something and I couldn't stop so I updated the PR to the cleaner API we discussed. Not sure this is exactly what you wanted from the implementation/documentation point of view but at least the API looks pretty good now.

@JonRowe

View changes

Show outdated Hide outdated features/built_in_matchers/output_to_stream.feature Outdated
@JonRowe

View changes

Show outdated Hide outdated lib/rspec/matchers/built_in/output_to_stream.rb Outdated
@JonRowe

This comment has been minimized.

Show comment
Hide comment
@JonRowe

JonRowe Jan 14, 2014

Member

Looks good, I just have a concern about @stream not being setup upon initialise, which opens the door for unexpected errors / obtuse error messages. I'd add a null object that implements message and raises an appropriate error in capture (ExpectationNotMet with a message explaining what they did wrong maybe?). I'd also be good to have a spec covering that eventuality.

Member

JonRowe commented Jan 14, 2014

Looks good, I just have a concern about @stream not being setup upon initialise, which opens the door for unexpected errors / obtuse error messages. I'd add a null object that implements message and raises an appropriate error in capture (ExpectationNotMet with a message explaining what they did wrong maybe?). I'd also be good to have a spec covering that eventuality.

@myronmarston

View changes

Show outdated Hide outdated features/built_in_matchers/output_to_stream.feature Outdated
@myronmarston

View changes

Show outdated Hide outdated lib/rspec/matchers.rb Outdated
@myronmarston

View changes

Show outdated Hide outdated lib/rspec/matchers/built_in.rb Outdated
@myronmarston

View changes

Show outdated Hide outdated lib/rspec/matchers/built_in/output_to_stream.rb Outdated
@lucapette

This comment has been minimized.

Show comment
Hide comment
@lucapette

lucapette Jan 15, 2014

Contributor

Here I'm again :) I think I applied all the feedback you gave me. I'm just not sure about the name of the null object, it's currently NullCapture (I'd go for something like CaptureNothing I think but I see we have a NullSolution somewhere so I kept the prefix).
On the same topic, aka naming is hard, there is the message error for when the user doesn't set any stream expectation. Not sure that's good enough.
Anyway I'm pretty sure that if there is room for improvement you'll point me into the right direction! Thank you for taking care of RSpec.

Contributor

lucapette commented Jan 15, 2014

Here I'm again :) I think I applied all the feedback you gave me. I'm just not sure about the name of the null object, it's currently NullCapture (I'd go for something like CaptureNothing I think but I see we have a NullSolution somewhere so I kept the prefix).
On the same topic, aka naming is hard, there is the message error for when the user doesn't set any stream expectation. Not sure that's good enough.
Anyway I'm pretty sure that if there is room for improvement you'll point me into the right direction! Thank you for taking care of RSpec.

myronmarston added a commit that referenced this pull request Jan 16, 2014

Merge pull request #410 from lucapette/output_to_stream_matchers
OutputToStdout and OutputToStderr matchers

@myronmarston myronmarston merged commit ffd25a8 into rspec:master Jan 16, 2014

1 check passed

default The Travis CI build passed
Details

myronmarston added a commit that referenced this pull request Jan 16, 2014

Tweak a few things post-merge #410.
- Add changelog entry.
- Copy the explanation from the YARD comments to
  the cuke as I think it was worded a bit better.
- Rephrase the note a bit to explain why.
- Since the the stream capturers are stateless,
  use singleton modules for them. Less garbage for the GC!
- Rename ivar to `@stream_capturer` since it's not a stream
  itself.
- Improve error message when the user forgets to chain
  `to_stdout` or `to_stderr` off of it.
- Still provide a description in this case.
@myronmarston

This comment has been minimized.

Show comment
Hide comment
@myronmarston

myronmarston Jan 16, 2014

Member

Thanks, @lucapette! I merged and I also applied a few final tweaks in 4a8bb9f if you want to take a look. This is going to be a nice new feature in RSpec 3 :).

Member

myronmarston commented Jan 16, 2014

Thanks, @lucapette! I merged and I also applied a few final tweaks in 4a8bb9f if you want to take a look. This is going to be a nice new feature in RSpec 3 :).

@lucapette

This comment has been minimized.

Show comment
Hide comment
@lucapette

lucapette Jan 16, 2014

Contributor

thank you @myronmarston and @JonRowe for helping me writing this feature, and thanks to @matthias-guenther for the original idea. It was really a pleasure to contribute to RSpec! I hope I'll find a way to help more.

Contributor

lucapette commented Jan 16, 2014

thank you @myronmarston and @JonRowe for helping me writing this feature, and thanks to @matthias-guenther for the original idea. It was really a pleasure to contribute to RSpec! I hope I'll find a way to help more.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment