Incremental SBT Print-Out #39

Closed
djspiewak opened this Issue Oct 14, 2011 · 18 comments

Comments

Projects
None yet
2 participants
@djspiewak

Is there any way we can get the SBT Specs2 runner to print out examples as they pass/fail, rather than waiting until the entire specification has run? I have several specifications that take on the order of several minutes to run, and it's a little annoying having to constantly remind myself that "no, SBT isn't stuck, it's just waiting for Specs".

I don't know if this is an SBT issue or a Specs2 issue. I suspect SBT since I've seen this with other tasks before, but I figured I'd file here first since you are generally quicker to respond to issues. :-)

@etorreborre

This comment has been minimized.

Show comment
Hide comment
@etorreborre

etorreborre Oct 14, 2011

Owner

This is something I already thought of but dismissed the idea on the ground that:

  1. the spec examples would be printed out of order if the execution is concurrent
  2. if there are included specifications the totals might not be correct

I can try to add a flag to enable the reporting of examples as soon as they're executed and see how this flies. Or maybe something like the equivalent of a progress bar in the console would be a good idea (not sure how to do that with sbt logs)?

Owner

etorreborre commented Oct 14, 2011

This is something I already thought of but dismissed the idea on the ground that:

  1. the spec examples would be printed out of order if the execution is concurrent
  2. if there are included specifications the totals might not be correct

I can try to add a flag to enable the reporting of examples as soon as they're executed and see how this flies. Or maybe something like the equivalent of a progress bar in the console would be a good idea (not sure how to do that with sbt logs)?

@etorreborre

This comment has been minimized.

Show comment
Hide comment
@etorreborre

etorreborre Oct 14, 2011

Owner

Btw you're looking toward the left on Github :-)

Owner

etorreborre commented Oct 14, 2011

Btw you're looking toward the left on Github :-)

@djspiewak

This comment has been minimized.

Show comment
Hide comment
@djspiewak

djspiewak Oct 14, 2011

A progress bar would be pretty awesome, but without control over the TTY, I don't know that you can make it look decent. That might actually be a feature we can kick upstream and request from Mark. It seems to me that more than one SBT task would have use of that sort of logging functionality (the progress bar, not necessarily full TTY control).

In the meantime, I would be thrilled with a default-off switch that prints specs as they execute. Almost all of my specifications avoid the issues you raised anyway, so it would work just fine for me.

Oh, and the GitHub avatar is based on Gravatar. I seriously thought about using the flipped version, just because of GitHub, but the avatar is used in enough other places (e.g. StackOverflow) that I felt the benefits would be lost and the original should be used instead. :-)

A progress bar would be pretty awesome, but without control over the TTY, I don't know that you can make it look decent. That might actually be a feature we can kick upstream and request from Mark. It seems to me that more than one SBT task would have use of that sort of logging functionality (the progress bar, not necessarily full TTY control).

In the meantime, I would be thrilled with a default-off switch that prints specs as they execute. Almost all of my specifications avoid the issues you raised anyway, so it would work just fine for me.

Oh, and the GitHub avatar is based on Gravatar. I seriously thought about using the flipped version, just because of GitHub, but the avatar is used in enough other places (e.g. StackOverflow) that I felt the benefits would be lost and the original should be used instead. :-)

@etorreborre

This comment has been minimized.

Show comment
Hide comment
@etorreborre

etorreborre Oct 15, 2011

Owner

I started working on reporting the examples as soon as they are executed. Hopefully if I get a bit of time this week-end you can get it on Monday.

Owner

etorreborre commented Oct 15, 2011

I started working on reporting the examples as soon as they are executed. Hopefully if I get a bit of time this week-end you can get it on Monday.

@etorreborre

This comment has been minimized.

Show comment
Hide comment
@etorreborre

etorreborre Oct 16, 2011

Owner

Daniel, you can have a try at the latest snapshot. Just pass the "streaming" argument in the sbt test options:

test-only MySpec -- streaming

Owner

etorreborre commented Oct 16, 2011

Daniel, you can have a try at the latest snapshot. Just pass the "streaming" argument in the sbt test options:

test-only MySpec -- streaming

@djspiewak

This comment has been minimized.

Show comment
Hide comment
@djspiewak

djspiewak Oct 16, 2011

Something funky happened in your +publish task. The SHA1 hashes for 1.7-SNAPSHOT are invalid. Republish?

Something funky happened in your +publish task. The SHA1 hashes for 1.7-SNAPSHOT are invalid. Republish?

@etorreborre

This comment has been minimized.

Show comment
Hide comment
@etorreborre

etorreborre Oct 16, 2011

Owner

I needed to do some refactoring anyway. I just republished and tested by updating on another machine and it works fine.

I've only tested on a simple spec though:

  package examples

  import org.specs2._

  class TestSpec extends mutable.Specification  { 
     val random = new scala.util.Random

      "Random tests" >> {
        (1 to 10) foreach { i =>
          "test "+i >> {
            Thread.sleep(50*random.nextInt(i))
            success
         }
       }
     }
   }

I hope it's also ok on the big specs of yours.

Owner

etorreborre commented Oct 16, 2011

I needed to do some refactoring anyway. I just republished and tested by updating on another machine and it works fine.

I've only tested on a simple spec though:

  package examples

  import org.specs2._

  class TestSpec extends mutable.Specification  { 
     val random = new scala.util.Random

      "Random tests" >> {
        (1 to 10) foreach { i =>
          "test "+i >> {
            Thread.sleep(50*random.nextInt(i))
            success
         }
       }
     }
   }

I hope it's also ok on the big specs of yours.

@etorreborre

This comment has been minimized.

Show comment
Hide comment
@etorreborre

etorreborre Oct 16, 2011

Owner

And btw, the argument has been shortened to "stream" on the command line.

Owner

etorreborre commented Oct 16, 2011

And btw, the argument has been shortened to "stream" on the command line.

@djspiewak

This comment has been minimized.

Show comment
Hide comment
@djspiewak

djspiewak Oct 16, 2011

It works! However, there do appear to be two problems. First, streaming output is missing a newline between should/>> sections (not sure exactly what they're called). Second, it appears to be by-passing SBT's logging mechanism since none of it is prefixed with [info]. :-) I know the second issue is a bit petty, but it seems quite jarring to me to have "raw" output on my console in the midst of all of SBT's log statements. I would almost prefer you explicitly prefix everything with "[info] " in order to make it more uniform, though obviously that's quite ugly.

It works! However, there do appear to be two problems. First, streaming output is missing a newline between should/>> sections (not sure exactly what they're called). Second, it appears to be by-passing SBT's logging mechanism since none of it is prefixed with [info]. :-) I know the second issue is a bit petty, but it seems quite jarring to me to have "raw" output on my console in the midst of all of SBT's log statements. I would almost prefer you explicitly prefix everything with "[info] " in order to make it more uniform, though obviously that's quite ugly.

@djspiewak

This comment has been minimized.

Show comment
Hide comment
@djspiewak

djspiewak Oct 16, 2011

Oh, it also appears that exception output is incorrectly handled and ends up being mostly lost. I had two specs which threw exceptions (they are currently failing) but only the second spec displayed its full stack trace. The first spec displayed the first line of the exception output, and the rest was trimmed.

Oh, it also appears that exception output is incorrectly handled and ends up being mostly lost. I had two specs which threw exceptions (they are currently failing) but only the second spec displayed its full stack trace. The first spec displayed the first line of the exception output, and the rest was trimmed.

@etorreborre

This comment has been minimized.

Show comment
Hide comment
@etorreborre

etorreborre Oct 17, 2011

Owner

Ok, take this as a proof-of-concept... I'm going to work on a more elaborate and realistic specification and ship you a new SNAPSHOT tomorrow.

Owner

etorreborre commented Oct 17, 2011

Ok, take this as a proof-of-concept... I'm going to work on a more elaborate and realistic specification and ship you a new SNAPSHOT tomorrow.

@djspiewak

This comment has been minimized.

Show comment
Hide comment
@djspiewak

djspiewak Oct 17, 2011

:-) Awesome! I hope you don't feel too pressured; it's not like this is a critical, data-munching blocker bug that we're talking about.

:-) Awesome! I hope you don't feel too pressured; it's not like this is a critical, data-munching blocker bug that we're talking about.

@etorreborre

This comment has been minimized.

Show comment
Hide comment
@etorreborre

etorreborre Oct 17, 2011

Owner

I've made some progress on this issue (see last snapshot). However I need to find a way to avoid the text of 2 should sections being interleaved. One way is to place a "step" between the 2 sections but then you lose the concurrent execution of examples between those 2 (i.e. examples of section 1 will always be executed before the examples of section 2). I need to think if there a better-clever-non-too-invasive way of having a nice display.

Owner

etorreborre commented Oct 17, 2011

I've made some progress on this issue (see last snapshot). However I need to find a way to avoid the text of 2 should sections being interleaved. One way is to place a "step" between the 2 sections but then you lose the concurrent execution of examples between those 2 (i.e. examples of section 1 will always be executed before the examples of section 2). I need to think if there a better-clever-non-too-invasive way of having a nice display.

@etorreborre

This comment has been minimized.

Show comment
Hide comment
@etorreborre

etorreborre Oct 20, 2011

Owner

I think I have now a good strategy on how to do things but it will take me some time to get to it because I need to revisit a few places (which is actually good). So don't expect anything too soon :-).

Owner

etorreborre commented Oct 20, 2011

I think I have now a good strategy on how to do things but it will take me some time to get to it because I need to revisit a few places (which is actually good). So don't expect anything too soon :-).

@djspiewak

This comment has been minimized.

Show comment
Hide comment
@djspiewak

djspiewak Oct 20, 2011

I can wait! :-)

I can wait! :-)

@etorreborre

This comment has been minimized.

Show comment
Hide comment
@etorreborre

etorreborre Oct 25, 2011

Owner

Daniel,

I've published a new snapshot where you don't need any special argument on the command line but you need to make sure that you have logBuffered := false in you build.sbt file (and no junitxml or html in the test options but I don't think that's the case for you).

Thanks for telling me how it works for you and especially if the performances are still ok.

Owner

etorreborre commented Oct 25, 2011

Daniel,

I've published a new snapshot where you don't need any special argument on the command line but you need to make sure that you have logBuffered := false in you build.sbt file (and no junitxml or html in the test options but I don't think that's the case for you).

Thanks for telling me how it works for you and especially if the performances are still ok.

@djspiewak

This comment has been minimized.

Show comment
Hide comment
@djspiewak

djspiewak Nov 1, 2011

Sorry it took me so long to get back to you. I've been tied up in other things. Gave it a try just now. Verdict: AWESOME!! I'll leave you the pleasure of clicking the "Close Issue" button. :-)

Thanks so very much for fixing this!

Sorry it took me so long to get back to you. I've been tied up in other things. Gave it a try just now. Verdict: AWESOME!! I'll leave you the pleasure of clicking the "Close Issue" button. :-)

Thanks so very much for fixing this!

@etorreborre

This comment has been minimized.

Show comment
Hide comment
@etorreborre

etorreborre Nov 1, 2011

Owner

Cool, I'm closing it then :-)

Owner

etorreborre commented Nov 1, 2011

Cool, I'm closing it then :-)

@etorreborre etorreborre closed this Nov 1, 2011

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment