New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Errors show up as a success with 0 specs instead of a failure #58

Closed
bjeanes opened this Issue May 10, 2012 · 14 comments

Comments

Projects
None yet
3 participants
@bjeanes

bjeanes commented May 10, 2012

E.g. if using the asset pipeline and you have a bad require line, instead of showing that error or drawing attention to it, it yields a false positive:

example

This is with PhantomJS 1.5.0

@netzpirat

This comment has been minimized.

Show comment
Hide comment
@netzpirat

netzpirat May 10, 2012

Contributor

Thanks for reporting. Will have a look asap.

Contributor

netzpirat commented May 10, 2012

Thanks for reporting. Will have a look asap.

@netzpirat

This comment has been minimized.

Show comment
Hide comment
@netzpirat

netzpirat May 11, 2012

Contributor

I cannot reproduce the issue.

I'm using the latest Guard::Jasmine version and also PhantomJS 1.5.0. When I for example change my spec.js.coffee from

#= require application
#= require sinon
#= require jasmine-sinon
#= require_tree .

to

#= require application
#= require sinon_not_existing
#= require jasmine-sinon
#= require_tree .

Then Guard::Jasmine fails with the following output:

Run all Jasmine suites
Run Jasmine suite at http://dnndev.me:8888/jasmine
ERROR: An error occurred: Timeout waiting for the Jasmine test results!

Sprockets::FileNotFound in Jasminerice/spec#index

Showing /Users/michi/.rvm/gems/ruby-1.9.3-p194/gems/jasminerice-0.0.8/app/views/jasminerice/spec/index.html.haml where line #7 raised:

couldn't find file 'sinon_not_existing'
  (in /Users/michi/Repositories/extranett/spec/javascripts/spec.js.coffee:2)
Extracted source (around line #7):

4:     = stylesheet_link_tag "jasmine"
5:     = stylesheet_link_tag "spec"
6:     = javascript_include_tag "jasminerice", :debug => Rails.env.development?
7:     = javascript_include_tag "spec", :debug => Rails.env.development?
8:     = csrf_meta_tags
9:   %body
Rails.root: /Users/michi/Repositories/extranett

Application Trace | Framework Trace | Full Trace
lib/extranett/middleware/ensure_development_host.rb:29:in `call'
Request

Parameters:

None
Show session dump

Show env dump

Response

Headers:

None

Do you have a special spec setup?

Guard::Jasmine implements a Jasmine Reporter and sends back the result when receiving reportRunnerResults. If this doesn't happen within the defined timeout, an error is thrown and the output of the html is shown as error message, as seen above.

So somehow in your setup, the runner is executed even with a missing require.

Contributor

netzpirat commented May 11, 2012

I cannot reproduce the issue.

I'm using the latest Guard::Jasmine version and also PhantomJS 1.5.0. When I for example change my spec.js.coffee from

#= require application
#= require sinon
#= require jasmine-sinon
#= require_tree .

to

#= require application
#= require sinon_not_existing
#= require jasmine-sinon
#= require_tree .

Then Guard::Jasmine fails with the following output:

Run all Jasmine suites
Run Jasmine suite at http://dnndev.me:8888/jasmine
ERROR: An error occurred: Timeout waiting for the Jasmine test results!

Sprockets::FileNotFound in Jasminerice/spec#index

Showing /Users/michi/.rvm/gems/ruby-1.9.3-p194/gems/jasminerice-0.0.8/app/views/jasminerice/spec/index.html.haml where line #7 raised:

couldn't find file 'sinon_not_existing'
  (in /Users/michi/Repositories/extranett/spec/javascripts/spec.js.coffee:2)
Extracted source (around line #7):

4:     = stylesheet_link_tag "jasmine"
5:     = stylesheet_link_tag "spec"
6:     = javascript_include_tag "jasminerice", :debug => Rails.env.development?
7:     = javascript_include_tag "spec", :debug => Rails.env.development?
8:     = csrf_meta_tags
9:   %body
Rails.root: /Users/michi/Repositories/extranett

Application Trace | Framework Trace | Full Trace
lib/extranett/middleware/ensure_development_host.rb:29:in `call'
Request

Parameters:

None
Show session dump

Show env dump

Response

Headers:

None

Do you have a special spec setup?

Guard::Jasmine implements a Jasmine Reporter and sends back the result when receiving reportRunnerResults. If this doesn't happen within the defined timeout, an error is thrown and the output of the html is shown as error message, as seen above.

So somehow in your setup, the runner is executed even with a missing require.

@bjeanes

This comment has been minimized.

Show comment
Hide comment
@bjeanes

bjeanes May 11, 2012

Not as far as I know. Let me try to reproduce with that exact test case
when I get a chance else I'll try to find what is different about my set up
since I have the latest versions too.

bjeanes commented May 11, 2012

Not as far as I know. Let me try to reproduce with that exact test case
when I get a chance else I'll try to find what is different about my set up
since I have the latest versions too.

@bjeanes

This comment has been minimized.

Show comment
Hide comment
@bjeanes

bjeanes May 23, 2012

It took me a while to come back around to this, but it turns out I'm seeing
this when there is an error on the page. In my case, coffeescript syntax
error. Is there anything guard-jasmine can do to detect errors that
happen outside tests and represent them as a failure?

bjeanes commented May 23, 2012

It took me a while to come back around to this, but it turns out I'm seeing
this when there is an error on the page. In my case, coffeescript syntax
error. Is there anything guard-jasmine can do to detect errors that
happen outside tests and represent them as a failure?

@bjeanes

This comment has been minimized.

Show comment
Hide comment
@bjeanes

bjeanes May 23, 2012

Even without a syntax error, this can happen. For example, my spec.js is currently simply:

throw Error("ExecJS::ProgramError: Cannot call method 'toString' of undefined\n (in /Users/bjeanes/Code/<project>/spec/javascripts/backbone/ext/filtered_collection_spec.js.coffee)")

bjeanes commented May 23, 2012

Even without a syntax error, this can happen. For example, my spec.js is currently simply:

throw Error("ExecJS::ProgramError: Cannot call method 'toString' of undefined\n (in /Users/bjeanes/Code/<project>/spec/javascripts/backbone/ext/filtered_collection_spec.js.coffee)")
@netzpirat

This comment has been minimized.

Show comment
Hide comment
@netzpirat

netzpirat May 23, 2012

Contributor

I cannot reproduce it with a CoffeeScript error. I just added inexistent.toString())) to my spec/javascripts/spec.js.coffee file and Guard::Jasmine reports:

> jasmine
jasmine
Run all Jasmine suites
Run Jasmine suite at http://dnndev.me:8888/jasmine
ERROR: An error occurred: Timeout waiting for the Jasmine test results!

ExecJS::RuntimeError in Jasminerice/spec#index

Showing /Users/michi/.rvm/gems/ruby-1.9.3-p194/gems/jasminerice-0.0.8/app/views/jasminerice/spec/index.html.haml where line #7 raised:

SyntaxError: unmatched ) on line 14
  (in /Users/michi/Repositories/extranett/spec/javascripts/spec.js.coffee)
Extracted source (around line #7):

4:     = stylesheet_link_tag "jasmine"
5:     = stylesheet_link_tag "spec"
6:     = javascript_include_tag "jasminerice", :debug => Rails.env.development?
7:     = javascript_include_tag "spec", :debug => Rails.env.development?
8:     = csrf_meta_tags
9:   %body
Rails.root: /Users/michi/Repositories/extranett

Application Trace | Framework Trace | Full Trace
lib/extranett/middleware/ensure_development_host.rb:29:in `call'
Request

Parameters:

None
Show session dump

Show env dump

Response

Headers:

None

When i make it valid like inexistent.toString(), but with a runtime error, I get a lot of error messages:

Extranett.Views.WebsiteList
  #render
    ✘ contains the websites
      ➤ TypeError: 'undefined' is not an object
      ➤ TypeError: 'undefined' is not an object

So all fine here. Since we both use the latest PhantomJS, Guard::Jasmine I can only imagine that we use a different JavaScript runtime that behaves different.

I'm using node as Runtime,

[1] pry(main)> ExecJS.runtime
=> #<ExecJS::ExternalRuntime:0x007fd9c53a50d8
 @binary="nodejs",
 @command=["nodejs", "node"],
 @deprecated=false,
 @encoding="UTF-8",
 @name="Node.js (V8)",
 @runner_path=
  "/Users/michi/.rvm/gems/ruby-1.9.3-p194/gems/execjs-1.4.0/lib/execjs/support/node_runner.js",
 @test_args=nil,
 @test_match=nil>
[2] pry(main)>

what's your runtime?

Contributor

netzpirat commented May 23, 2012

I cannot reproduce it with a CoffeeScript error. I just added inexistent.toString())) to my spec/javascripts/spec.js.coffee file and Guard::Jasmine reports:

> jasmine
jasmine
Run all Jasmine suites
Run Jasmine suite at http://dnndev.me:8888/jasmine
ERROR: An error occurred: Timeout waiting for the Jasmine test results!

ExecJS::RuntimeError in Jasminerice/spec#index

Showing /Users/michi/.rvm/gems/ruby-1.9.3-p194/gems/jasminerice-0.0.8/app/views/jasminerice/spec/index.html.haml where line #7 raised:

SyntaxError: unmatched ) on line 14
  (in /Users/michi/Repositories/extranett/spec/javascripts/spec.js.coffee)
Extracted source (around line #7):

4:     = stylesheet_link_tag "jasmine"
5:     = stylesheet_link_tag "spec"
6:     = javascript_include_tag "jasminerice", :debug => Rails.env.development?
7:     = javascript_include_tag "spec", :debug => Rails.env.development?
8:     = csrf_meta_tags
9:   %body
Rails.root: /Users/michi/Repositories/extranett

Application Trace | Framework Trace | Full Trace
lib/extranett/middleware/ensure_development_host.rb:29:in `call'
Request

Parameters:

None
Show session dump

Show env dump

Response

Headers:

None

When i make it valid like inexistent.toString(), but with a runtime error, I get a lot of error messages:

Extranett.Views.WebsiteList
  #render
    ✘ contains the websites
      ➤ TypeError: 'undefined' is not an object
      ➤ TypeError: 'undefined' is not an object

So all fine here. Since we both use the latest PhantomJS, Guard::Jasmine I can only imagine that we use a different JavaScript runtime that behaves different.

I'm using node as Runtime,

[1] pry(main)> ExecJS.runtime
=> #<ExecJS::ExternalRuntime:0x007fd9c53a50d8
 @binary="nodejs",
 @command=["nodejs", "node"],
 @deprecated=false,
 @encoding="UTF-8",
 @name="Node.js (V8)",
 @runner_path=
  "/Users/michi/.rvm/gems/ruby-1.9.3-p194/gems/execjs-1.4.0/lib/execjs/support/node_runner.js",
 @test_args=nil,
 @test_match=nil>
[2] pry(main)>

what's your runtime?

@bjeanes

This comment has been minimized.

Show comment
Hide comment
@bjeanes

bjeanes May 25, 2012

The undefined.toString() thing was evidently happening inside the coffee
script compiler. It happened when calling describe() without a closure,
iirc.

There may be environmental differences that are causing me to have
different behavior than you, but I can reproducibly create a scenario that
causes a pass with 0 specs. It always comes down to the asset pipeline not
actually generating any specs to run and instead throwing a JS exception.

I'll see if I can boil it down to a small separate repo that I can push
somewhere. It may have to be next week though as I'm about to travel.

bjeanes commented May 25, 2012

The undefined.toString() thing was evidently happening inside the coffee
script compiler. It happened when calling describe() without a closure,
iirc.

There may be environmental differences that are causing me to have
different behavior than you, but I can reproducibly create a scenario that
causes a pass with 0 specs. It always comes down to the asset pipeline not
actually generating any specs to run and instead throwing a JS exception.

I'll see if I can boil it down to a small separate repo that I can push
somewhere. It may have to be next week though as I'm about to travel.

@netzpirat

This comment has been minimized.

Show comment
Hide comment
@netzpirat

netzpirat May 25, 2012

Contributor

When I remove the describe closure, then the compilation fails with a HTTP error 500:

Completed 500 Internal Server Error in 11ms

ActionView::Template::Error (Parse error on line 3: Unexpected 'INDENT'

and Guard::Jasmine shows the status correctly:

Run all Jasmine suites
Run Jasmine suite at http://localhost:8888/jasmine
ERROR: An error occurred: Unable to access Jasmine specs at http://localhost:8888/jasmine

So I'm looking forward to your example app...

Contributor

netzpirat commented May 25, 2012

When I remove the describe closure, then the compilation fails with a HTTP error 500:

Completed 500 Internal Server Error in 11ms

ActionView::Template::Error (Parse error on line 3: Unexpected 'INDENT'

and Guard::Jasmine shows the status correctly:

Run all Jasmine suites
Run Jasmine suite at http://localhost:8888/jasmine
ERROR: An error occurred: Unable to access Jasmine specs at http://localhost:8888/jasmine

So I'm looking forward to your example app...

@pushbang

This comment has been minimized.

Show comment
Hide comment
@pushbang

pushbang May 30, 2012

just wanted to +1 this since we are also having this issue. in our case, if there is a syntax error in the coffeescript that spec is ignored silently.

have not fully investigated however, i suspect its a problem with jasminerice config. will comment again after i've taken a stab at tracing it

pushbang commented May 30, 2012

just wanted to +1 this since we are also having this issue. in our case, if there is a syntax error in the coffeescript that spec is ignored silently.

have not fully investigated however, i suspect its a problem with jasminerice config. will comment again after i've taken a stab at tracing it

@pushbang

This comment has been minimized.

Show comment
Hide comment
@pushbang

pushbang Jun 2, 2012

https://github.com/pushbang/guard-jasmine-problem-example shows the problem. just run guard as normal, and it will report 0 specs

The problem is that, out of the box, using :server_env => :test causes guard-jasmine to falsely report "0 specs, 0 failures" when there's a syntax error in your coffeescript.

By default in the test environment, rails does not throw an exception on asset pipeline compilation failure, but it does in the development environment. This can be shown independent of guard-jasmine by just hitting the example app root url in the 2 environments. I havent yet found the rails config setting to change this behavior, maybe you know?

desired behavior:

  1. obviously, never show me green when the specs i was trying to run didnt run!
  2. give me enough error output to find my syntax error and fix it. @netzpirat in your paste you seem to be getting that but i dont see it, even in development env. The syntax error IS available in the server log in development.

pushbang commented Jun 2, 2012

https://github.com/pushbang/guard-jasmine-problem-example shows the problem. just run guard as normal, and it will report 0 specs

The problem is that, out of the box, using :server_env => :test causes guard-jasmine to falsely report "0 specs, 0 failures" when there's a syntax error in your coffeescript.

By default in the test environment, rails does not throw an exception on asset pipeline compilation failure, but it does in the development environment. This can be shown independent of guard-jasmine by just hitting the example app root url in the 2 environments. I havent yet found the rails config setting to change this behavior, maybe you know?

desired behavior:

  1. obviously, never show me green when the specs i was trying to run didnt run!
  2. give me enough error output to find my syntax error and fix it. @netzpirat in your paste you seem to be getting that but i dont see it, even in development env. The syntax error IS available in the server log in development.
@pushbang

This comment has been minimized.

Show comment
Hide comment
@pushbang

pushbang Jun 3, 2012

aha! my problem was with jasminerice. the latest rubygem for jasminerice hardcodes config.assets.debug to false, despite whats in your test.rb. afaict config.assets.debug = true does not concatenate assets, but also will throw an exception that guard-jasmine can pick up and display.

bradphelan/jasminerice#45 fixes the problem and is in HEAD now. With that and explicitly setting config.assets.debug = true in test.rb, i can set :server_env => :test and get syntax errors reported by the jasmine runner. Might want to add to the README.

pushbang commented Jun 3, 2012

aha! my problem was with jasminerice. the latest rubygem for jasminerice hardcodes config.assets.debug to false, despite whats in your test.rb. afaict config.assets.debug = true does not concatenate assets, but also will throw an exception that guard-jasmine can pick up and display.

bradphelan/jasminerice#45 fixes the problem and is in HEAD now. With that and explicitly setting config.assets.debug = true in test.rb, i can set :server_env => :test and get syntax errors reported by the jasmine runner. Might want to add to the README.

@netzpirat

This comment has been minimized.

Show comment
Hide comment
@netzpirat

netzpirat Jun 4, 2012

Contributor

Thanks for sharing your findings. I've open a pull request for the Jasmine 1.2 upgrade and asked for a release, or if he needs help maintaining Jasminerice, so I can release. I'll update the README as soon as a new gem version is released.

Contributor

netzpirat commented Jun 4, 2012

Thanks for sharing your findings. I've open a pull request for the Jasmine 1.2 upgrade and asked for a release, or if he needs help maintaining Jasminerice, so I can release. I'll update the README as soon as a new gem version is released.

@pushbang

This comment has been minimized.

Show comment
Hide comment
@pushbang

pushbang Jun 5, 2012

cheers, thanks.

pushbang commented Jun 5, 2012

cheers, thanks.

@netzpirat

This comment has been minimized.

Show comment
Hide comment
@netzpirat

netzpirat Jun 18, 2012

Contributor

Jasminerice 0.0.9 is finally released, which includes the fix for this issue.

Contributor

netzpirat commented Jun 18, 2012

Jasminerice 0.0.9 is finally released, which includes the fix for this issue.

@netzpirat netzpirat closed this Jun 18, 2012

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment