From Travis CI: "Cannot read property 'numToReport' of undefined" #874
Comments
|
When you run tests locally, are you using the 'selenium' tunnel (the default), or just running them in Node? |
|
I simply run |
|
I'm not yet sure if it's related with Intern, as I removed the "pretty" reporter but still got some unexpected errors from Travis CI. I opened an issue on their side too -- travis-ci/travis-ci#9142. In that case, the error read "Driver info: driver.version: unknown". |
|
It finally was not related to Intern. Chrome was not available (not installed, or didn't have enough privileges to run), possibly because some updates they did on the Travis CI side recently. |
|
Ah. I apparently didn't scroll over far enough in the original error message, because it does say |
|
Indeed, I also didn't realize initially 😄 |
|
I have stumbled upon this issue again. This is my configuration: {
"environments": ["firefox"],
"capabilities": {
"fixSessionCapabilities": "no-detect"
},
"suites": ["build/**/*.spec.js"],
"coverage": ["build/**/!(*.spec).js"],
"reporters": ["pretty"]
}And this is the error: ! Cannot read property 'numToReport' of undefined
TypeError: Cannot read property 'numToReport' of undefined
at Pretty.suiteEnd <src/lib/reporters/Pretty.ts:162:41>
at <src/lib/reporters/Reporter.ts:94:31>
at <src/lib/executors/Executor.ts:302:30>
at <node_modules/@dojo/core/async/Task.ts:351:12>
at handler <node_modules/@dojo/core/async/ExtensiblePromise.ts:229:14>
at process._tickCallback <internal/process/next_tick.js:109:7>
! [POST http://localhost:4444/wd/hub/session / {"desiredCapabilities":{"idle-timeout":60,"name":"intern","browserName":"firefox","moz:firefoxOptions":{"args":["headless"]}}}] Unable to create new service: GeckoDriverService
Build info: version: '3.5.2', revision: '10229a9020', time: '2017-08-21T17:54:21.164Z'
System info: host: 'dlaptop', ip: '127.0.1.1', os.name: 'Linux', os.arch: 'amd64', os.version: '4.4.0-116-generic', java.version: '1.8.0_151'
Driver info: driver.version: unknown
SessionNotCreatedException: [POST http://localhost:4444/wd/hub/session / {"desiredCapabilities":{"idle-timeout":60,"name":"intern","browserName":"firefox","moz:firefoxOptions":{"args":["headless"]}}}] Unable to create new service: GeckoDriverService
Build info: version: '3.5.2', revision: '10229a9020', time: '2017-08-21T17:54:21.164Z'
System info: host: 'dlaptop', ip: '127.0.1.1', os.name: 'Linux', os.arch: 'amd64', os.version: '4.4.0-116-generic', java.version: '1.8.0_151'
Driver info: driver.version: unknown
at Server.post <node_modules/src/Server.ts:367:14>
at Server.createSession <node_modules/src/Server.ts:412:14>
at Suite.before <src/lib/executors/Node.ts:471:8>
at <src/lib/Suite.ts:388:19>
at new Task <node_modules/@dojo/core/async/Task.ts:239:3>
at runLifecycleMethod <src/lib/Suite.ts:355:10>
at before <src/lib/Suite.ts:458:10>
at Suite.run <src/lib/Suite.ts:477:6>
at <src/lib/executors/Node.ts:799:20>
at FunctionQueue.next <src/lib/executors/Node.ts:923:16>I'm using elementary OS (0.4.1 Loki), and have Firefox 58 installed. |
|
I am on OS X 10.11.6 and Chrome v64.0.3282.186. I am also getting the same error: My config for the test is: This error only occurs when I comment out |
|
Same here: |
|
What version of Intern are you using? Would it be possible for anyone to post a sample project that reproduces the issue? I have so far been unable to reproduce it. |
|
I have just reproduced the error. Intern 4.3.2. I attempted to run functional tests on my local PC with IE 11, but the "Protected Mode" was not set to the same value (I think a Windows update reset this setting). This caused the which seems to lead to errors of the form Some |
|
Any chance that you had the time to look into this? Any clues or workarounds? |
|
Sorry, have't gotten to it yet. Just to double check -- are you only seeing the issue when using the Pretty reporter on Travis, or is it happening locally now, too? |
|
At the moment I am seeing this problem locally only. I ignored it for a long while because it was working in Travis CI. I came to this issue, however, because Travis CI fails now too, although with a different error message: As you point out, the original issue ("Cannot read property 'numToReport' of undefined") seems to be related to the "pretty" reporter. However, I see different errors when trying "simple": Honestly, the whole Intern experience it's been a total pain in the ass from the beginning. I wanted to stick with it because the idea of it looked good, but I think I cannot stand it anymore. Without maturity and without stability, there is no point on having many amazing features. |
|
After looking into this a bit more, the pretty reporter error is a bit of a red herring. It's throwing an error because it's being asked to handle a
So the core issue is that the browser isn't able to start for some reason, and the Pretty reporter just isn't handling that gracefully (or at least it's not being particularly helpful). As to why the browser is failing to start,
In any case, the reporter should be handling the failure more gracefully. |
|
@jason0x43 I am sorry for my late-night mini-rant. Once you think to have tried everything you consider reasonable, you just go mad. In my opinion, both error messages and the documentation has plenty of room for improvement. (As a suggestion, error messages should when possible point to the right section of the documentation.) The thing is, when trying to make things "just work", I'm flooded with questions which are hard or impossible to answer with the documentation, such as:
Perhaps my expectations were misguided. What I was expecting Intern to be able to do (i.e. the reason I chose it, and I believe I'm not alone) is to run tests in a browser that I can choose from a given list of supported browsers, both locally and in a CI environment. Knowing some of the details of how it works can interest me as a developer, but I don't want to be forced to care about them while acting as a mere user of the framework. That is, I expect to be able to say "firefox", and have the tests run in Firefox. And if some extra step is required in order to make this work (e.g. installing Firefox or setting a specific driver version), I expect to see what those extra steps are as Intern's output. One last piece of feedback that I want to communicate is that Intern's configuration flexibility is no good at all. Of course, this might be only my personal opinion, but I have the impression that it is mentioned as a good feature in the docs, but the effect it has on me is no other than creating confusion. There should be a place for each property, not one thousand. I constantly ask myself, "is this the right place? will Intern be smart enough to pick it up?" Or if I'm copying some config from the web, "is this according to the latest Intern version? could this person have made a mistake?" And it really is not possible to answer these questions other than by trial and error, which is not a very rewarding way to work. (As a suggestion for this, it would be awesome to take forward TypeScript's support and allow defining the config in a "intern.ts" file, where you can have typed configuration.) For the sake of completion (in case someone is browsing the web for this error message), here is what I see when trying to use "firefox" as the browser: |
|
After trying the "tunnelOptions" you suggested, I see the following error: |
|
OK, after installing Google Chrome on my machine, I see the same error as the one in Travis CI, which looks like some progress to me: Edit I confirm the exact same problem occurs with @dojo/loader 0.1.0, 0.1.1 and 2.0.0. |
|
Interestingly, when I run the tests as "functionalSuites" as opposed to "suites", many of them run, meaning the loader seems to be able to find "intern" in this case. Other tests can't pass because they depend on a global |
|
Holy heck, I can't believe I made it "work"! I removed I am happy enough with the current state of things, but suggestions to make VS Code aware of Intern's types are very much welcomed. And Firefox continues not to work. This was only with "chrome". |
You're not wrong.
Intern is a tool that runs tests in Node or in browsers. It can manage browsers (opening and closing them and loading pages) using WebDriver, or you can open a browser window yourself and point it at Intern. In either case, though, Intern can only work with browsers that are available, it doesn't include one. (That would likely only work for Chrome and Firefox, and Intern attempts to be browser-agnostic as much as possible.)
Hmmm...that's a good question. You'd need to specify the location to Chromium on your system using the "binary" capability (that kind of thing is where WebDriver can be a pain to configure), something like: Since you have to specify the binary location, you may need to a different config on your local system vs a CI system (unless Chromium is in the same location).
That is a goal, although there are limitations to what Intern can easily do. For example, if you specify "firefox", Intern doesn't know what version of Firefox you're using. That version affects which version of geckodriver (Firefox's WebDriver server) you need. Using an invalid webdriver version might work, or it might partially work, or it might fail without providing a helpful error. Intern can certainly do better about informing the user of what's going on, but at some point a user will likely have to be involved.
Intern's configuration system is declarative so that it can work work on Node and in browsers, and be easily serialized and transmitted from Node to remote browsers. A side benefit of this is that configuration properties can be provided on the command line or as query args, or specified in a config file, all using the same property names and with the same data format. JSON isn't as flexible as an imperative script would be, but on the other hand it's easier to serialize and it avoids footguns like trying to run a config written in ES2015 on a browser that doesn't support it like IE.
I'm not really sure what is meant here. Intern has a well defined config format, and config options will only be used when they're in the right place. That said, I agree that having some way of validating a config would be very helpful. Someone started writing a JSON schema for Intern's config format at one point, although I don't believe it was ever completed. Adding such a schema to Intern's source would definitely help out.
If it's JSON, it should be current (or current enough). Intern prior to v4 (which came out a couple of years ago) used AMD modules for config.
I totally agree. Better documentation and a JSON schema would be very helpful here.
Intern supports two types of test: functional and unit. Functional tests always run in Node and remotely drive a browser (tell it to go to addresses and click elements and such). Unit tests run directly in Node or a browser. Which property your test be listed in (
If the test needs Having this distinction between test types does make things a bit complex, but it's not done without reason. Test modules that use Node modules or are written in ES6 generally can't be loaded directly in a browser, while tests that access browser resources may not load or execute in Node.
Hmmm...I'm not really sure about this. VSCode should be happy enough with |
|
Thanks a lot for taking the time to reply to my comments. A couple of clarifications:
When I referred to Intern configuration's excessive flexibility, I was referring to the fact that there is a "normalization" process. For example, you can write Not sure if you would consider this part of the normalization, but an even bigger part of the problem is the unpredictable inheritance model that there is in place. For instance, there is a This whole process has no measurable benefit, but adds a great deal of confusion for users, and unnecessary complexity to the project.
I know about the conceptual difference between the two, but we should not be expected to know about the implementation details of the two. It simply surprised me to see that one of them was able to deal with
The project I'm using Intern in is https://github.com/inad9300/Soil. If you check it out and open any |
Ah, I see what you mean now. That behavior was intended to be a convenience, mostly to handle input from the command line (so, for example, you could type
That's actually the weirdest one of those, and is a bit of a historical artifact. "capabilities" is what Selenium / WebDriver calls the config you're asking for from a Selenium server. "environments" is Intern's property for specifying which environments you want your test to run in. In Intern's config, That's definitely something else that could use more documentation, or a clearer structure.
I'm not really clear how that would help (at least not for this particular issue). I mean, the config structure is what it is, so whether you're specifying it in a JSON object, or constructing that config object using JS, or setting individual config properties with JS, you'll still need some knowledge about what the config property values should be. There are definitely cases where allowing a JS config file would be useful, but it would also introduce problems that might not make for a worthwhile tradeoff.
I'll have to take a look at your project to know what's going on there, but it likely is loader-related. For an
I'll take a look. |
|
@inad9300 I took a look at your project. It reminded me a bit of Polymer's lit-element and lit-html which I was just reading about yesterday. :) In any case, here are some things I noticed. ConfigThe "environments": [
{
"browserName": "chrome",
"goog:chromeOptions": {
"args": ["headless", "disable-gpu", "window-size=1024,768"]
}
},
{
"browserName": "firefox",
"moz:firefoxOptions": {
"args": ["-headless", "--window-size=1024,768"]
}
}
],If you want to set "environments": [
{
"browserName": "chrome",
"fixSessionCapabilities": "no-detect"
}
]or "capabilities": {
"fixSessionCapabilities": "no-detect"
}Also, since you're using Firefox, you'll need to tell Intern (actually the SeleniumTunnel in Leadfoot, which is Intern's WebDriver library) to download the Firefox driver. "tunnelOptions": {
"drivers": ["firefox", "chrome"]
}(I know, having to specify the drivers and browsers separately seems terribly duplicative. It's historical, and that requirement will be removed at some point.) TypingsThe problem with typings is that VScode is using the default project config ( To get around that you have a couple of options. One is to let the base Another option is to include a type reference at the top of each of your test files, like: /// <reference types="intern" />This is preferable to Speaking of module loaders, I did see that you were using a module loader. The issue you were running into is that AMD loaders such as the Dojo loader or requireJS require some configuration; for example, you'd need to tell the loader where various packages are (like Intern). Intern's own self tests currently use the Dojo loader, and have this sort of config. In this case it's much easier to just reference the types, though, rather than to configure the loader to make an unnecessary module load. |
|
Amazing feedback, thank you so much... I'm much more clear about the configuration after our discussion. And thanks for the reminder about VS Code picking only the tsconfig.json at the root! I wasn't thinking of this at all. It is definitely similar to lit-element and lit-html. Funny enough, I had this same observation and mentioned it to them in lit/lit#700 (although it was kindly ignored, which I totally understand). |
|
Closing this. I created #1002 to track improvements to the Pretty reporter. |
When running the tests locally I actually don't have any problem, but when Travis CI runs them, the following error raises:
Here is how my Intern configuration:
{ "environments": [{ "browserName": "chrome", "chromeOptions": {"args": ["headless"]} }], "capabilities": {"fixSessionCapabilities": false}, "loader": {"script": "dojo2"}, "suites": ["build/**/*.test.js"], "coverage": ["build/!(testing)/**/!(*.test).js"], "reporters": [{ "name": "pretty" }, { "name": "jsoncoverage", "options": {"filename": "coverage.json"} }] }The text was updated successfully, but these errors were encountered: