phantomjs 1.9.7 uses SSLv3 per default/fallback

[Wayneoween]

As stated in issue #12181 (comment) SSLv3 is used as default.

Since POODLE this is discouraged by every server.

Workaround is to call phantomjs with --ssl-protocol=tlsv1 but it should not be the default.

[jrf0110]

God, it took me an hour to figure out what the hell was suddenly going on.

[jcready]

👍

[mrfatboy]

This is probably not the appropriate place to ask this buy my problem led me to here.

My app that scrapes several https website worked perfectly yesterday, . This morning I am getting "SSL handshake failed" in my Phantomjs (injected casperjs) script. With phantomjs reporting this:

[warning] [phantom] Loading resource failed with status=fail: https://www.myurl.com

I have tried the --ignore-ssl-errors=true as well as the --ssl-protocol=tlsv1 flags but getting same error. I'm new to all of this but what the heck changed in the last 24 hours? My code has been working perfectly last 6 months.

[jrf0110]

Maybe try --ssl-protocol=any?

[mrfatboy]

i tried
--ssl-protocol=any
--ssl-protocol=[sslv3|sslv2|tlsv1|any]
--ssl-protocol=[sslv3|sslv2|tlsv1|any'] - I found this in another post but the ' crashes my script. Maybe typo?

with the same result. SSL Handshake error.

[Hainish]

👍

console.log('Loading a web page');
var page = require('webpage').create();
var url = "https://eff.org";
page.open(url, function (status) {
  console.log(status);
  //Page is loaded!
  phantom.exit();
});

$ phantomjs --debug=yes sslv3_test.js
2014-10-16T15:54:25 [DEBUG] CookieJar - Created but will not store cookies (use option '--cookies-file=' to enable persisten cookie storage) 
2014-10-16T15:54:25 [DEBUG] Phantom - execute: Configuration 
2014-10-16T15:54:25 [DEBUG]      0 objectName : "" 
2014-10-16T15:54:25 [DEBUG]      1 cookiesFile : "" 
2014-10-16T15:54:25 [DEBUG]      2 diskCacheEnabled : "false" 
2014-10-16T15:54:25 [DEBUG]      3 maxDiskCacheSize : "-1" 
2014-10-16T15:54:25 [DEBUG]      4 ignoreSslErrors : "false" 
2014-10-16T15:54:25 [DEBUG]      5 localToRemoteUrlAccessEnabled : "false" 
2014-10-16T15:54:25 [DEBUG]      6 outputEncoding : "UTF-8" 
2014-10-16T15:54:25 [DEBUG]      7 proxyType : "http" 
2014-10-16T15:54:25 [DEBUG]      8 proxy : ":1080" 
2014-10-16T15:54:25 [DEBUG]      9 proxyAuth : ":" 
2014-10-16T15:54:25 [DEBUG]      10 scriptEncoding : "UTF-8" 
2014-10-16T15:54:25 [DEBUG]      11 webSecurityEnabled : "true" 
2014-10-16T15:54:25 [DEBUG]      12 offlineStoragePath : "" 
2014-10-16T15:54:25 [DEBUG]      13 offlineStorageDefaultQuota : "-1" 
2014-10-16T15:54:25 [DEBUG]      14 printDebugMessages : "true" 
2014-10-16T15:54:25 [DEBUG]      15 javascriptCanOpenWindows : "true" 
2014-10-16T15:54:25 [DEBUG]      16 javascriptCanCloseWindows : "true" 
2014-10-16T15:54:25 [DEBUG]      17 sslProtocol : "sslv3" 
2014-10-16T15:54:25 [DEBUG]      18 sslCertificatesPath : "" 
2014-10-16T15:54:25 [DEBUG]      19 webdriver : ":" 
2014-10-16T15:54:25 [DEBUG]      20 webdriverLogFile : "" 
2014-10-16T15:54:25 [DEBUG]      21 webdriverLogLevel : "INFO" 
2014-10-16T15:54:25 [DEBUG]      22 webdriverSeleniumGridHub : "" 
2014-10-16T15:54:25 [DEBUG] Phantom - execute: Script & Arguments 
2014-10-16T15:54:25 [DEBUG]      script: "sslv3_test.js" 
2014-10-16T15:54:25 [DEBUG] Phantom - execute: Starting normal mode 
2014-10-16T15:54:25 [DEBUG] WebPage - setupFrame "" 
2014-10-16T15:54:25 [DEBUG] FileSystem - _open: ":/modules/fs.js" QMap(("mode", QVariant(QString, "r") ) )  
2014-10-16T15:54:25 [DEBUG] FileSystem - _open: ":/modules/system.js" QMap(("mode", QVariant(QString, "r") ) )  
2014-10-16T15:54:25 [DEBUG] FileSystem - _open: ":/modules/_coffee-script.js" QMap(("mode", QVariant(QString, "r") ) )  
2014-10-16T15:54:25 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/package.json" QMap(("mode", QVariant(QString, "r") ) )  
2014-10-16T15:54:25 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/coffee-script.js" QMap(("mode", QVariant(QString, "r") ) )  
2014-10-16T15:54:25 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/./lexer.js" QMap(("mode", QVariant(QString, "r") ) )  
2014-10-16T15:54:25 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/././rewriter.js" QMap(("mode", QVariant(QString, "r") ) )  
2014-10-16T15:54:25 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/././helpers.js" QMap(("mode", QVariant(QString, "r") ) )  
2014-10-16T15:54:25 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/./parser.js" QMap(("mode", QVariant(QString, "r") ) )  
2014-10-16T15:54:25 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/./helpers.js" QMap(("mode", QVariant(QString, "r") ) )  
2014-10-16T15:54:25 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/./nodes.js" QMap(("mode", QVariant(QString, "r") ) )  
2014-10-16T15:54:25 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/././scope.js" QMap(("mode", QVariant(QString, "r") ) )  
2014-10-16T15:54:25 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/./././helpers.js" QMap(("mode", QVariant(QString, "r") ) )  
2014-10-16T15:54:25 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/././lexer.js" QMap(("mode", QVariant(QString, "r") ) )  
2014-10-16T15:54:25 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/./././rewriter.js" QMap(("mode", QVariant(QString, "r") ) )  
2014-10-16T15:54:25 [DEBUG] FileSystem - _open: ":/modules/webpage.js" QMap(("mode", QVariant(QString, "r") ) )  
Loading a web page
2014-10-16T15:54:25 [DEBUG] WebPage - updateLoadingProgress: 10 
2014-10-16T15:54:25 [DEBUG] Network - Resource request error: 6 ( "SSL handshake failed" ) URL: "https://eff.org/" 
2014-10-16T15:54:25 [DEBUG] WebPage - updateLoadingProgress: 100 

$ phantomjs --ssl-protocol=TLSv1 --debug=true sslv3_test.js 
2014-10-16T15:56:44 [DEBUG] CookieJar - Created but will not store cookies (use option '--cookies-file=' to enable persisten cookie storage) 
2014-10-16T15:56:44 [DEBUG] Phantom - execute: Configuration 
2014-10-16T15:56:44 [DEBUG]      0 objectName : "" 
2014-10-16T15:56:44 [DEBUG]      1 cookiesFile : "" 
2014-10-16T15:56:44 [DEBUG]      2 diskCacheEnabled : "false" 
2014-10-16T15:56:44 [DEBUG]      3 maxDiskCacheSize : "-1" 
2014-10-16T15:56:44 [DEBUG]      4 ignoreSslErrors : "false" 
2014-10-16T15:56:44 [DEBUG]      5 localToRemoteUrlAccessEnabled : "false" 
2014-10-16T15:56:44 [DEBUG]      6 outputEncoding : "UTF-8" 
2014-10-16T15:56:44 [DEBUG]      7 proxyType : "http" 
2014-10-16T15:56:44 [DEBUG]      8 proxy : ":1080" 
2014-10-16T15:56:44 [DEBUG]      9 proxyAuth : ":" 
2014-10-16T15:56:44 [DEBUG]      10 scriptEncoding : "UTF-8" 
2014-10-16T15:56:44 [DEBUG]      11 webSecurityEnabled : "true" 
2014-10-16T15:56:44 [DEBUG]      12 offlineStoragePath : "" 
2014-10-16T15:56:44 [DEBUG]      13 offlineStorageDefaultQuota : "-1" 
2014-10-16T15:56:44 [DEBUG]      14 printDebugMessages : "true" 
2014-10-16T15:56:44 [DEBUG]      15 javascriptCanOpenWindows : "true" 
2014-10-16T15:56:44 [DEBUG]      16 javascriptCanCloseWindows : "true" 
2014-10-16T15:56:44 [DEBUG]      17 sslProtocol : "tlsv1" 
2014-10-16T15:56:44 [DEBUG]      18 sslCertificatesPath : "" 
2014-10-16T15:56:44 [DEBUG]      19 webdriver : ":" 
2014-10-16T15:56:44 [DEBUG]      20 webdriverLogFile : "" 
2014-10-16T15:56:44 [DEBUG]      21 webdriverLogLevel : "INFO" 
2014-10-16T15:56:44 [DEBUG]      22 webdriverSeleniumGridHub : "" 
2014-10-16T15:56:44 [DEBUG] Phantom - execute: Script & Arguments 
2014-10-16T15:56:44 [DEBUG]      script: "sslv3_test.js" 
2014-10-16T15:56:44 [DEBUG] Phantom - execute: Starting normal mode 
2014-10-16T15:56:44 [DEBUG] WebPage - setupFrame "" 
2014-10-16T15:56:44 [DEBUG] FileSystem - _open: ":/modules/fs.js" QMap(("mode", QVariant(QString, "r") ) )  
2014-10-16T15:56:44 [DEBUG] FileSystem - _open: ":/modules/system.js" QMap(("mode", QVariant(QString, "r") ) )  
2014-10-16T15:56:44 [DEBUG] FileSystem - _open: ":/modules/_coffee-script.js" QMap(("mode", QVariant(QString, "r") ) )  
2014-10-16T15:56:44 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/package.json" QMap(("mode", QVariant(QString, "r") ) )  
2014-10-16T15:56:44 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/coffee-script.js" QMap(("mode", QVariant(QString, "r") ) )  
2014-10-16T15:56:44 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/./lexer.js" QMap(("mode", QVariant(QString, "r") ) )  
2014-10-16T15:56:44 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/././rewriter.js" QMap(("mode", QVariant(QString, "r") ) )  
2014-10-16T15:56:44 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/././helpers.js" QMap(("mode", QVariant(QString, "r") ) )  
2014-10-16T15:56:44 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/./parser.js" QMap(("mode", QVariant(QString, "r") ) )  
2014-10-16T15:56:44 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/./helpers.js" QMap(("mode", QVariant(QString, "r") ) )  
2014-10-16T15:56:44 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/./nodes.js" QMap(("mode", QVariant(QString, "r") ) )  
2014-10-16T15:56:44 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/././scope.js" QMap(("mode", QVariant(QString, "r") ) )  
2014-10-16T15:56:44 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/./././helpers.js" QMap(("mode", QVariant(QString, "r") ) )  
2014-10-16T15:56:44 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/././lexer.js" QMap(("mode", QVariant(QString, "r") ) )  
2014-10-16T15:56:44 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/./././rewriter.js" QMap(("mode", QVariant(QString, "r") ) )  
2014-10-16T15:56:44 [DEBUG] FileSystem - _open: ":/modules/webpage.js" QMap(("mode", QVariant(QString, "r") ) )  
Loading a web page
2014-10-16T15:56:44 [DEBUG] WebPage - updateLoadingProgress: 10 
2014-10-16T15:56:45 [DEBUG] WebPage - updateLoadingProgress: 15 
2014-10-16T15:56:45 [DEBUG] WebPage - updateLoadingProgress: 16 
2014-10-16T15:56:45 [DEBUG] WebPage - updateLoadingProgress: 23 
2014-10-16T15:56:45 [DEBUG] WebPage - updateLoadingProgress: 27 
2014-10-16T15:56:45 [DEBUG] WebPage - setupFrame "" 
2014-10-16T15:56:45 [DEBUG] WebPage - updateLoadingProgress: 34 
2014-10-16T15:56:45 [DEBUG] WebPage - updateLoadingProgress: 41 
2014-10-16T15:56:45 [DEBUG] WebPage - updateLoadingProgress: 43 
2014-10-16T15:56:45 [DEBUG] WebPage - updateLoadingProgress: 48 
2014-10-16T15:56:45 [DEBUG] CookieJar - Saved "has_js=1; domain=www.eff.org; path=/" 
2014-10-16T15:56:45 [DEBUG] WebPage - updateLoadingProgress: 48 
2014-10-16T15:56:45 [DEBUG] WebPage - updateLoadingProgress: 52 
2014-10-16T15:56:45 [DEBUG] WebPage - updateLoadingProgress: 55 
2014-10-16T15:56:45 [DEBUG] WebPage - updateLoadingProgress: 57 
2014-10-16T15:56:45 [DEBUG] WebPage - updateLoadingProgress: 60 
2014-10-16T15:56:45 [DEBUG] WebPage - updateLoadingProgress: 62 
2014-10-16T15:56:45 [DEBUG] WebPage - updateLoadingProgress: 64 
2014-10-16T15:56:45 [DEBUG] WebPage - updateLoadingProgress: 67 
2014-10-16T15:56:45 [DEBUG] WebPage - updateLoadingProgress: 70 
2014-10-16T15:56:45 [DEBUG] WebPage - updateLoadingProgress: 72 
2014-10-16T15:56:45 [DEBUG] WebPage - updateLoadingProgress: 77 
2014-10-16T15:56:45 [DEBUG] WebPage - updateLoadingProgress: 80 
2014-10-16T15:56:45 [DEBUG] WebPage - updateLoadingProgress: 82 
2014-10-16T15:56:45 [DEBUG] WebPage - updateLoadingProgress: 82 
2014-10-16T15:56:45 [DEBUG] WebPage - updateLoadingProgress: 100 
success

[mrfatboy]

Thanks for gut checking me. I finally got it to work. I had to add two arguments for phantomjs PLUS I had the problem of the arguments where at the of cmd line. The arguments at the end work for casperjs but not for phantomjs. My phantomjs script injects casperjs and I switch around sometimes.

phantomjs --debug=yes --ignore-ssl-errors=true --ssl-protocol=any ssltest.js

[mrfatboy]

Well, I spoke a little too soon. I can scrape most of my websites but I can't scrape British Airways using the the option --ssl-protocol=any. It hangs halfway thru the page load. I have to use --ssl-protocol=tlsv1 for it to work.

The url is https://www.britishairways.com/travel/home/public/en_us

As I have many websites that I scrape, am I now going to have to know exactly what ssl version each of them require and use the correct --ssl-protocol version for each one? Before Poodle I did not have to worry about it. It just worked.

Will --ssl-protocol=all work in the future? Is this just a BA website issue?

Thanks

[micahwedemeyer]

+1

[pasangsherpa]

+1