Unhandled reject Error: Failed to load url when using SlimerJS

[winghouchan]

When using Horseman with SlimerJS the following error is returned:

>   horseman phantom created +637ms
  horseman phantom version 1.9.8 +147ms
  horseman page created +154ms
  horseman phantomjs onLoadFinished triggered +117ms [ 'success', 'about:blank', false ] NaN
  horseman .close(). +6ms
Unhandled rejection Error: Failed to load url
    at checkStatus (/Users/user/Projects/project/node_modules/node-horseman/lib/index.js:276:16)
    at tryCatcher (/Users/user/Projects/project/node_modules/bluebird/js/release/util.js:16:23)
    at Function.Promise.attempt.Promise.try (/Users/user/Projects/project/node_modules/bluebird/js/release/method.js:39:29)
    at Object.loadFinishedSetup [as onLoadFinished] (/Users/user/Projects/project/node_modules/node-horseman/lib/index.js:274:43)
    at /Users/user/Projects/project/node_modules/node-phantom-simple/node-phantom-simple.js:636:30
    at Array.forEach (native)
    at IncomingMessage.<anonymous> (/Users/user/Projects/project/node_modules/node-phantom-simple/node-phantom-simple.js:617:17)
    at emitNone (events.js:72:20)
    at IncomingMessage.emit (events.js:166:7)
    at endReadableNT (_stream_readable.js:905:12)
    at nextTickCallbackWith2Args (node.js:478:9)
    at process._tickDomainCallback (node.js:433:17)

The following test case will reproduce it:

const Horseman = require('node-horseman');

const config = {
  browser: {
    clientScripts: [],
    timeout: 5000,
    interval: 50,
    loadImages: true,
    switchToNewTab: false,
    // no support from SlimerJS cookiesFile: null,
    // no support from SlimerJS ignoreSSLErrors: false,
    sslProtocol: 'any',
    // no support from SlimerJS webSecurity: true,
    injectJquery: true,
    injectBluebird: false,
    bluebirdDebug: false,
    // no support from SlimerJS proxy: null,
    // no support from SlimerJS proxyType: null,
    // no support from SlimerJS proxyAuth: null,
    phantomPath: '/usr/local/Cellar/slimerjs/0.10.0/bin/slimerjs',
    debugPort: null,
    debugAutorun: true,
  },
  userAgent: 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/600.4.10 (KHTML, like Gecko) Version/8.0.4 Safari/600.4.10',
}

const horseman = new Horseman(config.browser);

horseman
  .userAgent(config.userAgent)
  .open('http://google.co.uk/')
  .screenshot('google.jpg')
  .close();

[awlayton]

Please run again as DEBUG='horseman*' BLUEBIRD_DEBUG=1 node script.js and post the output.

[winghouchan]

My internet went down 😣 and in that time I started getting somewhere.

Taking the first part of the stack trace:

Unhandled rejection Error: Failed to load url
    at checkStatus (/Users/user/Projects/project/node_modules/node-horseman/lib/index.js:276:16)

We'll head to /lib/index.js:276:

            return self.ready = HorsemanPromise.try(function checkStatus() {
                if (status !== 'success') {
                    var err = new Error('Failed to load url');
                    return HorsemanPromise.reject(err);
                }
            })

From there we can see the error is only thrown if status !== 'success'. Unfortunately, a difference between SlimerJS and PhantomJS is causing the assignment of status to be an array instead of a string. You can see that in the original output:

  horseman phantomjs onLoadFinished triggered +117ms [ 'success', 'about:blank', false ] NaN

A test with logging the value of status inside of checkStatus also confirmed this.

[ 'success', 'about:blank', false ]

As a result I've temporarily changed the condition as to when the error is thrown to:

if (status !== 'success' && status[0] !== 'success') {
// ...

[awlayton]

I believe that the callback is supposed to be called with 3 arguments, not one argument that is an array. Perhaps there is a bug in node-phnatom-simple (or horseman) causing the arguments to be wrong.

[winghouchan]

Ahh I see. I'll keep my temporary workaround until baudehlo/node-phantom-simple#133 is resolved. Thanks again for your assistance @awlayton!

[kirillrogovoy]

Hi, @awlayton!

One more issue regarding this Failed to load url error. Sorry for such big text. :)

A small pre-history. I have a QA-automation tool based on the Horseman. Before Horseman 3 I used only .on event to handle request errors like this:

    let lastError
    horsemanInstance.on('resourceError', (err) => {
      lastError = err
    })

    horsemanInstance.on('resourceReceived', (err) => {
      const err = lastError
      // ...check the error and maybe fire a lib-level error...
     lastError = null
    })

I didn't really like it, but it was straight-forward and seemed robust.

The key part was not treating Operation cancelled as an error because it always got fired when I had a page redirect (reason: the new page starts loading before the previous one finished).

However, Horseman 3 presents a solid system of error handling via promises rejections which is a good move to me. However, actions like .click() get rejected now with a Failed to load url error, so I can't get any information from it.

Therefore, I need to silence all this errors and continue using my old mechanism to filter out request errors, which looks terrible (empty catch body in a number of places). While the good way would be to have the error details in the error object that is being rejected in the promise.

My point is this exception is quite useless without any details. While promise-rejection-style-of-error-handling is a very good way, it's still not usable enough.

So, I see three options now:

1. Make the exception more detailed, so there is no need to use phantom's events.
2. Make an option to globally ignore this exception (I mean, not to throw it).
3. Don't do anything with horseman's code and find a dirty workaround. This one is the least preferable way.

I'm aware that this error gets thrown at the callback on phantom's event, so we can operate with that only data phantom provided us. However, I believe horseman may obtain some additional information to make that error more detailed to make it actually useful.

Thanks!

[awlayton]

Number 1 sounds good @flashhhh, but I am not sure how horseman could go about improving the exceptions.

If you have an idea it would be great if you could submit a PR, or maybe just open a new issue for this improvement and give me detail as to how to go about implementing it.

[minhchu]

I'm having the same issue but with PhantomJS. Even I run the default example:

const Horseman = require('node-horseman');
const users = ['silver_no_tsuki'];

users.forEach((user) => {
    const horseman = new Horseman();
    horseman
        .open(`http://twitter.com/${user}`)
        .text('.ProfileNav-item--followers .ProfileNav-value')
        .then((text) => {
            console.log(`${user}: ${text}`);
        })
        .close();
});

It throws Unhandled reject Error: Failed to load url

[winghouchan]

hey @minhchu I think it'll be best to create a separate issue for the problem you are facing. This thread will be tracking the issue specifically caused by using SlimerJS with Horseman. I'll be changing the title of this thread to reflect this as well 😄

[winghouchan]

I just gave a quick test with the code you have in your comment and was unable to reproduce the issue you're facing.

[awlayton]

I cannot reproduce either @minhchu. When I run it I get this output:

silver_no_tsuki: 10

Are you sure it didn't actually fail to get the URL? Maybe you just couldn't reach Twitter.

[minhchu]

@winghouchan thanks for your advice.
@awlayton sorry. I forgot to mention. This bug doesnt always appear.
Sometimes I run above codes and it works, sometime it doesnt.

[awlayton]

What makes you think this is actually a bug @minhchu? Failing to fetch a URL really can happen. Are you sure it's not your network or something?

[minhchu]

sorry if I misunderstand the meaning of bug. In this case, you can considered it as does not work as expected.
I know that fetch a url can fail. But it shouldnt fail for other url as well.
I tried to fetch twitter and google. They worked then they didnt then they worked again. I'm sure it's not my network.
But as you mentioned, I'll check again.

[awlayton]

A bug is if the software does not work as expected, but your network not working as expected would not be a bug (but it is a problem).

If there is a bug, this sort of thing is most likely caused by PhantomJS, not horseman.

[minhchu]

This is the error I got when run node horseman.js:

Unhandled rejection Error: Failed to GET url: https://www.google.com/search?site
=&tbm=isch&source=hp&biw=1366&bih=640&q=moon
    at checkStatus (D:\PhantomJS\node_modules\node-horseman\lib\actions.js:78:16
)
    at PassThroughHandlerContext.finallyHandler (D:\PhantomJS\node_modules\blueb
ird\js\release\finally.js:56:23)
    at PassThroughHandlerContext.tryCatcher (D:\PhantomJS\node_modules\bluebird\
js\release\util.js:16:23)
    at Promise._settlePromiseFromHandler (D:\PhantomJS\node_modules\bluebird\js\
release\promise.js:502:31)
    at Promise._settlePromise (D:\PhantomJS\node_modules\bluebird\js\release\pro
mise.js:559:18)
    at Promise._settlePromise0 (D:\PhantomJS\node_modules\bluebird\js\release\pr
omise.js:604:10)
    at Promise._settlePromises (D:\PhantomJS\node_modules\bluebird\js\release\pr
omise.js:683:18)
    at Promise._fulfill (D:\PhantomJS\node_modules\bluebird\js\release\promise.j
s:628:18)
    at D:\PhantomJS\node_modules\bluebird\js\release\nodeback.js:42:21
    at D:\PhantomJS\node_modules\node-phantom-simple\node-phantom-simple.js:60:1
8
    at IncomingMessage.<anonymous> (D:\PhantomJS\node_modules\node-phantom-simpl
e\node-phantom-simple.js:645:9)
    at emitNone (events.js:72:20)
    at IncomingMessage.emit (events.js:166:7)
    at endReadableNT (_stream_readable.js:905:12)
    at nextTickCallbackWith2Args (node.js:455:9)
    at process._tickCallback (node.js:369:17)

Here is horseman.js:

var Horseman = require('node-horseman');
var horseman = new Horseman();

horseman
  .userAgent('Mozilla/5.0 (Windows NT 6.1; WOW64; rv:27.0) Gecko/20100101 Firefox/27.0')
  .open('https://www.google.com/search?site=&tbm=isch&source=hp&biw=1366&bih=640&q=moon')
  .screenshot('example.jpg')
  .close();

But when I run phantomjs phantom.js it runs normally. Here is phantom.js:

var page = require('webpage').create();
page.settings.userAgent = 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:27.0) Gecko/20100101 Firefox/27.0';
page.open('https://www.google.com/search?site=&tbm=isch&source=hp&biw=1366&bih=640&q=moon', function(status) {
  console.log("Status: " + status);
  if(status === "success") {
    page.render('example.png');
  }
  phantom.exit();
});

So I guess the problem from horseman ?

[winghouchan]

Hey @minhchu this definitely deseveres a new thread. I was able to reproduce the case you posted above.

Also please consider posting all the output, not just the lines describing the error because the lead-up to the error is also very important.

[awlayton]

Closing since this is a node-phantom-simple issue and not a horseman one.

See baudehlo/node-phantom-simple#147 for my proposed fix, if you care.