Don't wait for finished replies #428

mhoran · 2012-12-04T04:37:06Z

Synchronous requests will have emitted the finished signal by the time
we connect to it.

Synchronous requests will have emitted the finished signal by the time we connect to it.

lucassus · 2012-12-04T09:16:21Z

Unfortunately it still does not work for me.

webkit_debug gives me the following output:

Received 200 from "https://ajax.googleapis.com/ajax/libs/jquery/1.6/jquery.min.js" 
2 requests remaining 
Page finished with true 
Started request to "https://js.appcenter.intuit.com/Content/IA/intuit.ipp.anywhere.css" 
Started request to "https://appcenter.intuit.com/trackapi/TrackingActions?remote=true&pn=Activecell%20%7C%20The%20small%20business%20management%20platform&nm=iaLoginButtonHorizontal&typ=Render&txt=SignInWithIntuit&et=load&ref=&url=http%3A%2F%2F127.0.0.1%3A39593%2Fusers%2Fsign_in%3F_subdomain%3Dlaunchpad&host=127.0.0.1%3A39593&callback=jQuery164030159705178812146_1354612306948&_=1354612306953" 
Received 200 from "https://js.appcenter.intuit.com/Content/IA/intuit.ipp.anywhere.css" 
3 requests remaining 
Started request to "https://js.appcenter.intuit.com/Content/IA/button_signinwithintuit_horiz_small.png" 
Received 200 from "https://js.appcenter.intuit.com/Content/IA/button_signinwithintuit_horiz_small.png" 
3 requests remaining 
Received 200 from "https://appcenter.intuit.com/trackapi/TrackingActions?remote=true&pn=Activecell | The small business management platform&nm=iaLoginButtonHorizontal&typ=Render&txt=SignInWithIntuit&et=load&ref=&url=http://127.0.0.1:39593/users/sign_in?_subdomain=launchpad&host=127.0.0.1:39593&callback=jQuery164030159705178812146_1354612306948&_=1354612306953" 
2 requests remaining

Sometimes in the last line I have Page finished with true.
With the previous capybara-webkit version everything works fine. It's strange, because I use the most recent version of capybara-webkit in one of my open source project (https://github.com/lucassus/mongo_browser/tree/master/spec/features) and everything works fine.

Here is how my Gemfile.lock looks: https://gist.github.com/985d54f7eb2558f453a9

mhoran · 2012-12-04T12:36:55Z

Thanks for your feedback. I think there's still a possible race condition here. Do you happen to be doing anything that could be opening a new window?

lucassus · 2012-12-04T13:48:25Z

I don't think so, the race condition by definition is sth unpredictable.
I have this result every time time I execute the spec.

ryreitsma · 2012-12-04T17:32:06Z

FYI: we experienced the issue that Capybara hangs unexpectedly after a multipart fileupload and a page refresh, using some nifty javascript. This actually fixes our issue, so great work, thanks.

jferris · 2012-12-04T17:47:36Z

src/WebPageManager.cpp

@@ -51,7 +49,18 @@ void WebPageManager::emitLoadStarted() {

 void WebPageManager::requestCreated(QByteArray &url, QNetworkReply *reply) {
  logger() << "Started request to" << url;
-  m_started += reply;
+  if (reply->isFinished())


Is there a race condition here? Is it possible that we get a false from this method and then it finishes before we call connect below? It's not clear to me how much of the reply behavior is asynchronous.

Another way to handle this would be to remove finished replies when we get the page load signal. Currently, the logic is "are there any replies left;" we can change that to "are there any unfinished replies left."

I think that's a possibility. We've used this pattern elsewhere, so I went with it.

I believe the reply in progress logic was added for multi-window sync. However, it probably makes more sense to check whether there are any windows loading instead of any replies, since the pageLoading signal is more reliable. This would probably resolve the other deadlocks as well.

I originally changed this to check replies for a few reasons:

Multi-window sync

iframes

It now waits for dependent assets, including JavaScript

Unfortunately, the pageLoading signal is just differently reliable; for example, you'll get multiple signals for a page with iframes, and the number of start signals doesn't always equal the number of finished signals. I think we're going to need to deduce the actual page state based on a combination of reply and page signals. That's sort of what we're doing now, but there are clearly holes in the existing logic.

Let's merge what you have; we can change the way we handle replies in a later commit if we need to. The UnsupportedContent changes are nice, and this at least reduces the possibility of a deadlock.

Related: I'd like to build up a suite of tests that reproduce these kinds of issues, similar to the malloc_spec we used for finding the segfault in 4.8. That way, we have tests we can run before release that reduce the possibility of reintroducing race conditions. I'll give that a shot this week.

Anyway, nice work on finding the finished signal edge case.

mhoran · 2012-12-05T03:52:31Z

@lucassus, could paste the full debug output of the failing scenario? I was debugging this tonight and noticed that if some resources are requested over HTTP and others over HTTPS, WebKit seems to hang, likely for security reasons. It looks like the jquery request is hanging, but I'm not sure what else.

I've not been able to come up with a failing scenario for your case. I'm not sure if @jferris has had luck; he mentioned something in one of the related issues.

mhoran · 2012-12-05T04:25:53Z

src/UnsupportedContentHandler.cpp

@@ -10,8 +10,7 @@
 void UnsupportedContentHandler::renderNonHtmlContent() {
  QByteArray text = m_reply->readAll();
  m_page->mainFrame()->setContent(text, QString("text/plain"), m_reply->url());
-  m_page->setUnsupportedContentLoaded();
-  m_page->networkAccessManagerFinishedReply(m_reply);
+  m_page->unsupportedContentFinishedReply(m_reply);
  m_page->loadFinished(true);


If we leave the loadFinished signal connected to the page, this will happen automatically. Any reason this was disconnected?

I can't remember.

We went through a few iterations with the unsupported content stuff before we got one that worked right. Issues I remember:

Multiple loadFinished signals

No loadFinished signals

loadFinished signal with a false parameter (meaning that it failed)

I don't think there were any race conditions here, so I think you just need to get the tests passing. The commit log may know more.

The tests pass without with the signal hooked up and the line removed. There was a bunch of rework leading up to what we have now, so it's possible that we can just leave the signal hooked up and remove the line.

Okay - if we don't know why the code is there, and the tests pass, let's remove it. I don't want to have dead code slowing us down.

mhoran · 2012-12-05T18:47:12Z

@jferris, we don't ever seem to get inside http://git.io/ZKtWlQ. This looks to be a relic from this refactoring: http://git.io/EmaD7A. This means we only emit pageFinished from setPageStatus. I'm wondering if there's a situation where some outgoing requests may be sitting around in m_started, unrelated to the current request. Because they're never removed, and may never finish, we incorrectly assume that the WebPage is still loading.

I'm wondering if a simpler solution would be to catch all the loadStarted signals and only emit pageFinished when they all finish. If we keep them in a QSet (like we do with m_started), wouldn't that take care of duplicate loadStarted events?

I also think some of the confusing behavior regarding event firing may have been resolved by http://git.io/e_NaMg. This ensures that all pending events have processed by the time we check whether or not the page is loading.

lucassus · 2012-12-05T18:47:55Z

@mhoran here is the full debug output from my hanging scenario https://gist.github.com/4218354

jferris · 2012-12-05T18:57:51Z

@mhoran that sounds fine to me if it works. I've definitely run into situations where the number of loadStarted signals wasn't the same as the number of pageFinished signals. Also, is there an object that represents a "page load?" How will we track which loads are ongoing if they don't have a unique reference?

Instead of emitting loadStarted when a reply is created and pageFinished when all pending replies have finished, emit these signals when the corresponding WebPage events fire.

WebPage::setContent will cause the loadFinished signal to fire, so there's no need to call loadFinished if the signal is connected.

mhoran · 2012-12-06T02:57:07Z

@lucassus, could you give the latest changes on this branch a shot? I've changed the logic to not care about whether or not all replies have finished but instead depend on whether or not the WebPage has finished loading

@jferris, let me know your thoughts regarding b28452e. I'm not sure if this approach wasn't working for you before, or if I'm missing something. However, the test suite is passing reliably for me (I even ran it on Windows!)

lucassus · 2012-12-06T08:51:52Z

@mhoran it was tested on this version mhoran@5bef336

mhoran · 2012-12-06T12:32:07Z

@lucassus, I believe b28452e may be relevant to you. If you've got the git requirement in your gemfile, you should be able to bundle update capybara-webkit to get the latest.

jferris · 2012-12-06T14:47:45Z

@mhoran I think that's worth a try. Looks good to merge.

Don't wait for finished replies

5bef336

Synchronous requests will have emitted the finished signal by the time we connect to it.

This was referenced Dec 4, 2012

Capybara-webkit stops / hangs unexpectedly and inconsistently #399

Closed

Stalling when running javascript tests #425

Closed

jferris reviewed Dec 4, 2012
View reviewed changes

mhoran reviewed Dec 5, 2012
View reviewed changes

mhoran added 2 commits December 5, 2012 21:05

Emit loadStarted and pageFinished on WebPage state change

b28452e

Instead of emitting loadStarted when a reply is created and pageFinished when all pending replies have finished, emit these signals when the corresponding WebPage events fire.

Don't disconnect loadFinished in UnsupportedContentHandler

2eb38b3

WebPage::setContent will cause the loadFinished signal to fire, so there's no need to call loadFinished if the signal is connected.

mhoran merged commit 2eb38b3 into thoughtbot:master Dec 6, 2012

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Don't wait for finished replies #428

Don't wait for finished replies #428

mhoran commented Dec 4, 2012

lucassus commented Dec 4, 2012

mhoran commented Dec 4, 2012

lucassus commented Dec 4, 2012

ryreitsma commented Dec 4, 2012

jferris Dec 4, 2012

mhoran Dec 4, 2012

jferris Dec 4, 2012

mhoran commented Dec 5, 2012

mhoran Dec 5, 2012

jferris Dec 5, 2012

mhoran Dec 5, 2012

jferris Dec 5, 2012

mhoran commented Dec 5, 2012

lucassus commented Dec 5, 2012

jferris commented Dec 5, 2012

mhoran commented Dec 6, 2012

lucassus commented Dec 6, 2012

mhoran commented Dec 6, 2012

jferris commented Dec 6, 2012

Don't wait for finished replies #428

Don't wait for finished replies #428

Conversation

mhoran commented Dec 4, 2012

lucassus commented Dec 4, 2012

mhoran commented Dec 4, 2012

lucassus commented Dec 4, 2012

ryreitsma commented Dec 4, 2012

jferris Dec 4, 2012

Choose a reason for hiding this comment

mhoran Dec 4, 2012

Choose a reason for hiding this comment

jferris Dec 4, 2012

Choose a reason for hiding this comment

mhoran commented Dec 5, 2012

mhoran Dec 5, 2012

Choose a reason for hiding this comment

jferris Dec 5, 2012

Choose a reason for hiding this comment

mhoran Dec 5, 2012

Choose a reason for hiding this comment

jferris Dec 5, 2012

Choose a reason for hiding this comment

mhoran commented Dec 5, 2012

lucassus commented Dec 5, 2012

jferris commented Dec 5, 2012

mhoran commented Dec 6, 2012

lucassus commented Dec 6, 2012

mhoran commented Dec 6, 2012

jferris commented Dec 6, 2012