Fix specs on Qt 4.8

[fxposter]

There are some differences in capybara_webkit and selenium drivers for
now.

Also, selenium-webdriver was updated in Gemfile.lock.

[jferris]

I've started working on some of this in the jf-4-8 branch.

The segfaults are caused by a race condition; in Qt 4.8, all HTTP requests are handled in the background thread, which means any data that makes it into the HTTP layer must now be immutable or otherwise threadsafe. The segfault comes from trying to access a deleted string or deleting an already deleted string depending on which thread finishes first, but I haven't been able to track down which string is getting in there yet.

I think there may be some fixes in my branch related to the URL and UnsupportedContent issues, so that may be worth looking into.

[mhoran]

@jferris, your fixes do seem to resolve the URL issues for me. For the record, it seems that the behavior of QWebFrame::requestedUrl() was changed from the previous behavior which included pushState/replaceState changes to instead returning only the requested URL, which makes sense.

This leaves us with the segfault issue, which I've not been able to track down, and the status code/header issue partially addressed by @fxposter.

The status code and headers issues seem to be coming from WebPage::networkAccessManagerFinishedReply(). This method handles setting the status code and headers for the WebPage. The code determines whether or not to set the status code and headers based on the URL of the incoming HTTP response and the URL of the currently focused frame. Somewhere between Qt 4.7 and 4.8, the behavior of the WebPage::finished() signal was changed such that it is fired before the URL of the focused frame is set, causing this issue. I'd opened an upstream bug for this, but haven't heard back (see #315).

[fxposter]

@jferris This pull request is based on your jf-4-8 branch.

[jferris]

@fxposter sorry, I missed that. I agree with @mhoran on the setUrl changes. What spec were you fixing by adding the renderNonHtmlContent call there?

[mhoran]

Using valgrind I was able to trace down at least one race condition. It looks like in some cases reset is getting processed after another command has already started. This results in the WebPage which is currently executing an invokeCapybaraFunction() trying to respond to a deleted WebPage. I was able to work around this by commenting out m_pages->first()->deleteLater() in WebPageManager::reset().

Unfortunately once I got past this isuse, I ran into another very cryptic segfault. valgrind wasn't much help this time around.

For reference, here's the backtrace before commenting out deleteLater(): https://gist.github.com/3444902. Commenting out deleteLater() reliably gets me past invokeCapybaraFunction(), but instead blows up elsewhere: https://gist.github.com/3444924.

[mhoran]

It looks like the source of our problems is documented here: http://goo.gl/5Fs3S. Specifically,

This signal is emitted when the page finishes loading content. This signal is independant of script execution or page rendering. ok will indicate whether the load was successful or any error occurred.

Putting some debugging in place, I found that the WebPage::pageFinished() signal is fired by the WebPage before WebPage::injectJavascriptHelpers() is called. If the browser receives reset! before the JavaScript helpers have loaded, we get a segmentation fault.

Many thanks to @Huuf for providing a second pair of eyes in debugging.

[jferris]

@mhoran Excellent find. To make sure I understand correctly, in Qt 4.8, the following sequence can happen and results in a segfault: pageFinished is fired, a reset command is sent, the pages are deleted, and then a deleted frame gets the signal javascriptWindowObjectCleared?

Is it possible that this is happening because there's an iframe on the page that gets loaded later, and it's the iframe's window that's getting the javascriptWindowObjectCleared signal after the main frame is finished loading? Does this ever happen when there's only one frame on the page?

[mhoran]

@jferris, that looks to be the case.

I don't think the bug is limited to iframes, unfortunately. Every WebPage has a main frame that the JavaScript helpers need to be injected into. However, from further investigation it looks like the JavaScript Window object doesn't always get cleared. I tried adding a makeshift mutex around the frame->evaluateJavaScript call in WebPage::InjectJavascriptHelpers, but this just caused the test suite to hang.

[jferris]

@mhoran I think we're running into different issues.

Here's the backtrace I get pretty reliably:

Received "Visit" 
Started "Visit" 
Load started 
"Visit" started page load 
Started request to "http://127.0.0.1:53053/" 
Finished "Visit" with response "Success()" 

Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: KERN_INVALID_ADDRESS at address: 0x000000010c5b70fe

#0  0x00007fff8897e6c0 in malloc_error_break ()
#1  0x00007fff8897e805 in free ()
#2  0x000000010315bd3e in QString::free ()
#3  0x00000001031c2c29 in QUrlPrivate::authority ()
#4  0x00000001031ca002 in QUrlPrivate::toEncoded ()
#5  0x00000001031cb0a4 in QUrlPrivate::validate ()
#6  0x00000001031cb305 in QUrl::port ()
#7  0x0000000102faaedb in QHttpThreadDelegate::startRequest ()
#8  0x0000000103012c1b in QHttpThreadDelegate::qt_static_metacall ()
#9  0x000000010322f861 in QObject::event ()
#10 0x00000001022fbfad in QApplicationPrivate::notify_helper ()
#11 0x0000000102302fe4 in QApplication::notify ()
#12 0x000000010321b75c in QCoreApplication::notifyInternal ()
#13 0x000000010321d380 in QCoreApplicationPrivate::sendPostedEvents ()
#14 0x0000000103250832 in QEventDispatcherUNIX::processEvents ()
#15 0x000000010321a664 in QEventLoop::processEvents ()
#16 0x000000010321aa14 in QEventLoop::exec ()
#17 0x0000000103105198 in QThread::exec ()
#18 0x00000001031085aa in QThreadPrivate::start ()
#19 0x00007fff8892c8bf in _pthread_start ()
#20 0x00007fff8892fb75 in thread_start ()

It always happened directly after finishing a Visit command, and it's always in QString::free with QHttpThreadDelegate further up the trace.

Removing the m_pages->first()->deleteLater() call causes the driver to hang for me at random specs; I have no idea why.

I haven't been able to reproduce the same backtrace you've been getting. I'm going to try updating Qt (I'm on 4.8.0) to see if anything changes.

[jferris]

I get a similar trace in Qt 4.8.1:

#0  0x00007fff8897e6c0 in malloc_error_break ()
#1  0x00007fff8897e805 in free ()
#2  0x0000000102d7cf68 in QString::free ()
#3  0x0000000102d7d1d9 in QString::realloc ()
#4  0x0000000102ded9e9 in operator+=<QLatin1Char, QString> ()
#5  0x0000000102de48cd in QUrlPrivate::authority ()
#6  0x0000000102dea0f2 in QUrlPrivate::toEncoded ()
#7  0x0000000102deb194 in QUrlPrivate::validate ()
#8  0x0000000102deb3f5 in QUrl::port ()
#9  0x00000001030d37db in QHttpThreadDelegate::startRequest ()
#10 0x000000010313cb2b in QHttpThreadDelegate::qt_static_metacall ()
#11 0x0000000102e528e1 in QObject::event ()
#12 0x000000010209f93d in QApplicationPrivate::notify_helper ()
#13 0x00000001020a5dc4 in QApplication::notify ()
#14 0x0000000102e3e17c in QCoreApplication::notifyInternal ()
#15 0x0000000102e3f5a0 in QCoreApplicationPrivate::sendPostedEvents ()
#16 0x0000000102e73502 in QEventDispatcherUNIX::processEvents ()
#17 0x0000000102e3d094 in QEventLoop::processEvents ()
#18 0x0000000102e3d444 in QEventLoop::exec ()
#19 0x0000000102d25f08 in QThread::exec ()
#20 0x0000000102d295ba in QThreadPrivate::start ()
#21 0x00007fff8892c8bf in _pthread_start ()
#22 0x00007fff8892fb75 in thread_start ()

[mhoran]

I've seen this one as well when running the entire test suite. However, when I run just ./spec/integration/driver_spec.rb under valgrind I get the injectJavascriptHelpers segfault regularly. I think valgrind slows down the app enough such that the deleteLater runs and we get the segfault. Under normal circumstances, it's even more difficult to reproduce these issues.

[mhoran]

@jferris, the injectJavascriptHelpers segfaults seem only reproducible in valgrind, though they may be related to some of the other issues. I managed to come across three different segfaults this weekend running the malloc test outside of valgrind. The first traced back to WebPageManager::requestCreated. It seems that sometimes the QNetworkReply is NULL, so the logger blows up trying to read the url(). See the backtrace here: https://gist.github.com/3615538.

I attempted to mitigate this issue by disconnecting the signals from each page in WebPageManager::reset(), but that didn't seem to help.

I didn't have much luck analyzing the backtraces from the other segfaults: https://gist.github.com/3615583 and https://gist.github.com/3615579.

I ran these tests against Qt 4.8.2, and saw similar issues with 4.8.0.

[jferris]

@mhoran cool, that's consistent with what I've been seeing. I think we're getting a call to requestCreated after the request has already been deleted. I'm not sure why yet. I'll let you know if I find anything else out.

[mhoran]

We've addressed the segfaults in #403 and the test suite is passing.