New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

unhandled api call not revealed in Visual Studio #187

Closed
cResults opened this Issue Jul 10, 2015 · 9 comments

Comments

Projects
None yet
2 participants
@cResults

cResults commented Jul 10, 2015

the failing test indicated below is just an unhandle api call. The way this will be resolved is to add a jasmine spy. The problems is that wallaby doesn't give any indication of the problem.

I restarted wallaby twice before taking this image. Keep in mind also that we have a few tests that are not run in PhantomJS which accounts for some of the disparity between karma's total of 287 and wallaby's total of 276.
image

after restarting a third time
image

If I leave the Wallaby.js - Failing Tests, go to Wallaby.js - Console and go sifting through the logs I find this entry:

Fri, 10 Jul 2015 20:55:58 GMT wallaby:middleware ENOENT: no such file or directory, open 'C:\Users\dsmith\AppData\Local\Temp\.wallaby\projects\btZ5zHKlkkRq+PJs3VGcPvuhEg=\cache\instrumented\forms\api\Form'

which I suspect is telling me the same thing as karma. The difference is that karma screams at me "there is a problem" and if the number of passing tests in wallaby is close to what is expected, it would go completely unnoticed.

As I look at these images another thought came to mind. This is a new solution with only 287 tests and their already taking karma 3-4 seconds to run. We have other solutions with significantly larger test base that karma runs in less than a second. It would be great if wallaby out put the tests into Visual Studio Test Explorer or some other sortable list with the time it took to execute each test, like the C# tests do. This way we could find out what tests are taking so long and rewrite them. I greatly appreciate the things that wallaby does to make test execution lightning fast, but if it could also help us figure out where our tests are slow it would be beneficial. More importantly, it may lead us to performance issues in our source code. If this idea is not already in your backlog, please consider adding it.

Please let me know if you have any questions.

Respectfully,

cResults

@cResults cResults changed the title from unhandled api call not revealed to unhandled api call not revealed in Visual Studio Jul 10, 2015

@ArtemGovorov

This comment has been minimized.

Show comment
Hide comment
@ArtemGovorov

ArtemGovorov Jul 11, 2015

Member

the failing test indicated below is just an unhandle api call.

How can I reproduce it? Could you please share some test code/details how do you call the missing API? I can create something similar to cause the similar issue and make wallaby report the warning to the Failing tests console, but what I don't yet understand is how that missing API affects the number of executed tests (in both wallaby and karma)?

It would be great if wallaby out put the tests into Visual Studio Test Explorer

Yep, there's this feature request that should address it.

Member

ArtemGovorov commented Jul 11, 2015

the failing test indicated below is just an unhandle api call.

How can I reproduce it? Could you please share some test code/details how do you call the missing API? I can create something similar to cause the similar issue and make wallaby report the warning to the Failing tests console, but what I don't yet understand is how that missing API affects the number of executed tests (in both wallaby and karma)?

It would be great if wallaby out put the tests into Visual Studio Test Explorer

Yep, there's this feature request that should address it.

@cResults

This comment has been minimized.

Show comment
Hide comment
@cResults

cResults Jul 13, 2015

From what I'm seeing, I would think that you'd be able to reproduce the issue, by running a test that exersizes a source code method which makes an ajax call. We're currently using jQuery to make our ajax calls and that may be significant.

Being notified that our test has caused an unexpected (ie not mocked) ajax call, is quite valuable. Especially when adding tests to legacy code.

Karma runs all the tests each time. My guess is that wallaby hits the error and bails out. And since test execution order can be different, the point at which it fails and bails is different. I'll see if I can make that happen again and I look through the console log.

On that note, if you added a setting that enable us to prevent all of our tests info from writing to the console, it would be easier for us to give you the whole console. As it is, info that is meaningful for debugging wallaby is mixed in with stuff that our company doesn't want us to post on a public site, which makes it more time consuming for us to get you what you need. If there is some place else where this info is logged. Please let me know.

cResults commented Jul 13, 2015

From what I'm seeing, I would think that you'd be able to reproduce the issue, by running a test that exersizes a source code method which makes an ajax call. We're currently using jQuery to make our ajax calls and that may be significant.

Being notified that our test has caused an unexpected (ie not mocked) ajax call, is quite valuable. Especially when adding tests to legacy code.

Karma runs all the tests each time. My guess is that wallaby hits the error and bails out. And since test execution order can be different, the point at which it fails and bails is different. I'll see if I can make that happen again and I look through the console log.

On that note, if you added a setting that enable us to prevent all of our tests info from writing to the console, it would be easier for us to give you the whole console. As it is, info that is meaningful for debugging wallaby is mixed in with stuff that our company doesn't want us to post on a public site, which makes it more time consuming for us to get you what you need. If there is some place else where this info is logged. Please let me know.

@ArtemGovorov

This comment has been minimized.

Show comment
Hide comment
@ArtemGovorov

ArtemGovorov Jul 14, 2015

Member

Ok, thanks will try to reproduce with jQuery.ajax call to some dummy URL.

Karma runs all the tests each time

You screenshot for Karma run says Executed 241 of 287. According to wallaby screenshot, it executes 230 tests. Given the expected disparity of 11 tests, it looks to me that both Karma and Wallaby in this specific case are not executing 46 tests. I'd really like to understand why it happens.

stuff that our company doesn't want us to post on a public site

What part of the logged info is sensitive? Test names, file paths, something else?

Member

ArtemGovorov commented Jul 14, 2015

Ok, thanks will try to reproduce with jQuery.ajax call to some dummy URL.

Karma runs all the tests each time

You screenshot for Karma run says Executed 241 of 287. According to wallaby screenshot, it executes 230 tests. Given the expected disparity of 11 tests, it looks to me that both Karma and Wallaby in this specific case are not executing 46 tests. I'd really like to understand why it happens.

stuff that our company doesn't want us to post on a public site

What part of the logged info is sensitive? Test names, file paths, something else?

@ArtemGovorov

This comment has been minimized.

Show comment
Hide comment
@ArtemGovorov

ArtemGovorov Jul 14, 2015

Member

In core v1.0.79, I have added the report404AsError setting to the config so you can set it to true to make wallaby report missing resources/APIs access as errors.

{
  files: [...],
  tests: [...],
  env: {
    report404AsError: true
  }
}

With this setting on, if I do something like:

$.ajax('/missing/api');

in my test, wallaby.js will report the error to the Wallaby Console (even without the debug config option set to true):

untitled

We'll also make Visual Studio to attract more attention to the error messages in the Wallaby Console, for now they look like other messages, but should really stand out.

I'll close the issue as fixed, but we should really get to the bottom of why both karma an wallaby execute less tests in this case. The cause of this behaviour may be the reason for other issues you're having with wallaby.

Member

ArtemGovorov commented Jul 14, 2015

In core v1.0.79, I have added the report404AsError setting to the config so you can set it to true to make wallaby report missing resources/APIs access as errors.

{
  files: [...],
  tests: [...],
  env: {
    report404AsError: true
  }
}

With this setting on, if I do something like:

$.ajax('/missing/api');

in my test, wallaby.js will report the error to the Wallaby Console (even without the debug config option set to true):

untitled

We'll also make Visual Studio to attract more attention to the error messages in the Wallaby Console, for now they look like other messages, but should really stand out.

I'll close the issue as fixed, but we should really get to the bottom of why both karma an wallaby execute less tests in this case. The cause of this behaviour may be the reason for other issues you're having with wallaby.

@cResults

This comment has been minimized.

Show comment
Hide comment
@cResults

cResults Jul 14, 2015

I'll close the issue as fixed, but we should really get to the bottom of why both karma an wallaby execute less tests in this case. The cause of this behaviour may be the reason for other issues you're having with wallaby.

Yes, I agree. I shouldn't have any trouble reproducing this. I take a closer look when I get back next week.

cResults commented Jul 14, 2015

I'll close the issue as fixed, but we should really get to the bottom of why both karma an wallaby execute less tests in this case. The cause of this behaviour may be the reason for other issues you're having with wallaby.

Yes, I agree. I shouldn't have any trouble reproducing this. I take a closer look when I get back next week.

@cResults

This comment has been minimized.

Show comment
Hide comment
@cResults

cResults Jul 14, 2015

We'll also make Visual Studio to attract more attention to the error messages in the Wallaby Console, for now they look like other messages, but should really stand out.

Could this be displayed in the test output window? Even if you change the font to make it stand out in the Console output window, Users would need to toggle from test output to console output to see that there is a problem.

cResults commented Jul 14, 2015

We'll also make Visual Studio to attract more attention to the error messages in the Wallaby Console, for now they look like other messages, but should really stand out.

Could this be displayed in the test output window? Even if you change the font to make it stand out in the Console output window, Users would need to toggle from test output to console output to see that there is a problem.

@cResults

This comment has been minimized.

Show comment
Hide comment
@cResults

cResults Jul 14, 2015

Test names, file paths

Yes. Aren't these just noise to you?

cResults commented Jul 14, 2015

Test names, file paths

Yes. Aren't these just noise to you?

@ArtemGovorov

This comment has been minimized.

Show comment
Hide comment
@ArtemGovorov

ArtemGovorov Jul 14, 2015

Member

Even if you change the font to make it stand out in the Console output window, Users would need to toggle from test output to console output to see that there is a problem.

Sure, can do that as well.

Test names, file paths

Yes. Aren't these just noise to you?

Wallaby is only logging this data in the Wallaby Console so that user can better understand what is going on. For example see what tests were executed and what files are affected. Sometimes this information is helpful when troubleshooting issues. I may just introduce another setting to not log paths and test names. Alternatively, feel free to just send the console output directly to me.

Member

ArtemGovorov commented Jul 14, 2015

Even if you change the font to make it stand out in the Console output window, Users would need to toggle from test output to console output to see that there is a problem.

Sure, can do that as well.

Test names, file paths

Yes. Aren't these just noise to you?

Wallaby is only logging this data in the Wallaby Console so that user can better understand what is going on. For example see what tests were executed and what files are affected. Sometimes this information is helpful when troubleshooting issues. I may just introduce another setting to not log paths and test names. Alternatively, feel free to just send the console output directly to me.

@ArtemGovorov

This comment has been minimized.

Show comment
Hide comment
@ArtemGovorov

ArtemGovorov Jul 22, 2015

Member

In v1.0.12 we have implemented focusing Wallaby Console automatically if there're any errors in the log, let me know if it solves the issue with the missing API error visibility.

Member

ArtemGovorov commented Jul 22, 2015

In v1.0.12 we have implemented focusing Wallaby Console automatically if there're any errors in the log, let me know if it solves the issue with the missing API error visibility.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment