Implicit page load not occurring on Back, Forward, Refresh, and Element Click

[hjwp]

Hi there,

Minimal repro here: https://github.com/hjwp/selenium3-no-more-implicit-wait-for-page-load

• clicking on form submit consistently causes StaleElementReferenceException
• clicking on normal links does not
• tested on geckodriver 0.11.1, selenium 3.0.1, firefox 45ESR and firefox 49
• bug does not happen for selenium 2

I know I'm like a broken record, asking for these "magical" implicit waits to work. Actually I really don't mind if they don't work, but I would like confirmation on whether the old magical behaviour from selenium 2 is something y'all intend to keep maintaining in selenium 3 / geckodriver, or whether it's gone forever.

[hjwp]

More info:

• I'm aware this isn't a case of "implicit wait" (which I understand as blocking for a given timeout in cases where elements are not found, to see if they appear on a page. the one you control with the implicitly_wait method)
• Instead, this is a wait where, effectively, the click command blocks until the new page load. I'm seeing that this works for normal hyperlink clicks, but doesn't work for form submit clicks.
• check out the commit history in the repo if you want to see a version with no forms, it clicks through hyperlinks perfectly consistently.
• also, this only happens for servers that are a little bit slow. again, see commit history, but my initial attempt at a minimal repro just used static files (instead of flask), and the server seemed to be "too fast", so that wouldn't show up the bug consistently.

[hjwp]

More info:

• I'm aware this isn't a case of "implicit wait" (which I understand as blocking for a given timeout in cases where elements are not found, to see if they appear on a page. the one you control with the implicitly_wait method)
• Instead, this is a wait where, effectively, the click command blocks until the new page load. I'm seeing that this works for normal hyperlink clicks, but doesn't work for form submit clicks.
• check out the commit history in the repo if you want to see a version with no forms, it clicks through hyperlinks perfectly consistently.
• also, this only happens for servers that are a little bit slow. again, see commit history, but my initial attempt at a minimal repro just used static files (instead of flask), and the server seemed to be "too fast", so that wouldn't show up the bug consistently.

[hjwp]

(reading the spec, I guess this is more about "page load strategy", and not about "session implicit wait timeout"?)

[hjwp]

(reading the spec, I guess this is more about "page load strategy", and not about "session implicit wait timeout"?)

[andreastt]

For what it is worth, the spec does not say that we should wait implicitly for the element to become interactable when clicking on elements. In Selenium I’m being told different driver implementations do different things.

For what it is worth, there was a discussion about this on the WebDriver WG mailing list earlier this month: https://lists.w3.org/Archives/Public/public-browser-tools-testing/2016OctDec/thread.html

[andreastt]

For what it is worth, the spec does not say that we should wait implicitly for the element to become interactable when clicking on elements. In Selenium I’m being told different driver implementations do different things.

For what it is worth, there was a discussion about this on the WebDriver WG mailing list earlier this month: https://lists.w3.org/Archives/Public/public-browser-tools-testing/2016OctDec/thread.html

[davehunt]

@andreastt I don't think this is about implicit waits for elements to become interactable. I think it's referring to the "wait for navigation to complete" from the spec. This should be implemented (according to the spec):

• Go
• Refresh
• Element Click (when click causes navigation)

From reading various bug reports, I suspect that Refresh and Element Click (when click causes navigation) are not supported by GeckoDriver/Marionette. I haven't run any tests myself to confirm this, however.

[davehunt]

@andreastt I don't think this is about implicit waits for elements to become interactable. I think it's referring to the "wait for navigation to complete" from the spec. This should be implemented (according to the spec):

• Go
• Refresh
• Element Click (when click causes navigation)

From reading various bug reports, I suspect that Refresh and Element Click (when click causes navigation) are not supported by GeckoDriver/Marionette. I haven't run any tests myself to confirm this, however.

[davehunt]

I found https://bugzilla.mozilla.org/show_bug.cgi?id=1291320 regarding the refreshing issue.

[davehunt]

I found https://bugzilla.mozilla.org/show_bug.cgi?id=1291320 regarding the refreshing issue.

[murdog05]

I have run my tests using Marrionette and it most certainly does not wait for the navigation to finish. Our tests, on their own, wait for some key elements to be loaded and ready to interact with before proceeding. Unfortunately, it does rely on the page load to be complete, which Marrionette does not support right now. This is a blocking issue for us. I feel this bug should be a high priority.

[murdog05]

I have run my tests using Marrionette and it most certainly does not wait for the navigation to finish. Our tests, on their own, wait for some key elements to be loaded and ready to interact with before proceeding. Unfortunately, it does rely on the page load to be complete, which Marrionette does not support right now. This is a blocking issue for us. I feel this bug should be a high priority.

[hjwp]

@davehunt that sounds right to me, it's about that step 8 in element click, and the "run post-navigation checks" and "wait for navigation to complete" steps, I think
(sorry for using the word "implicit wait" in the title, that rather threw everyone off on the wrong track...)

[hjwp]

@davehunt that sounds right to me, it's about that step 8 in element click, and the "run post-navigation checks" and "wait for navigation to complete" steps, I think
(sorry for using the word "implicit wait" in the title, that rather threw everyone off on the wrong track...)

[gcorpian]

I find it hard to understand why https://bugzilla.mozilla.org/show_bug.cgi?id=1291320 still hasn't been assigned to anyone to get fixed after a month. This is a serious issue that needs to be looked at if people need to migrate to the gecko driver/marionette.

[gcorpian]

I find it hard to understand why https://bugzilla.mozilla.org/show_bug.cgi?id=1291320 still hasn't been assigned to anyone to get fixed after a month. This is a serious issue that needs to be looked at if people need to migrate to the gecko driver/marionette.

[jgraham]

Please note that this kind of issue evangelism is generally not helpful and should be avoided. We understand that there are some issues that make it difficult for people to use geckodriver without adjusting their tests. But resources are of course limited and we can only solve a finite number of problems in a given amount of time. As it happens we have recently been concentrating on support for bypassing certificate checks when using WebDriver, and on Actions. Those are both blockers for people who use those features (and harder to work around than this issue, which could be dealt with in user code by e.g. using executeAsyncScript to wait for a relevant load event, or to check the document.readyState).

Note that future advocacy-type comments may lead to issues being locked or similar. Dealing with unnecessary GitHub email directly reduces the amount of time we have available to actually produce fixes.

[jgraham]

Please note that this kind of issue evangelism is generally not helpful and should be avoided. We understand that there are some issues that make it difficult for people to use geckodriver without adjusting their tests. But resources are of course limited and we can only solve a finite number of problems in a given amount of time. As it happens we have recently been concentrating on support for bypassing certificate checks when using WebDriver, and on Actions. Those are both blockers for people who use those features (and harder to work around than this issue, which could be dealt with in user code by e.g. using executeAsyncScript to wait for a relevant load event, or to check the document.readyState).

Note that future advocacy-type comments may lead to issues being locked or similar. Dealing with unnecessary GitHub email directly reduces the amount of time we have available to actually produce fixes.

[andreastt]

We are working on https://bugzilla.mozilla.org/show_bug.cgi?id=1330348 to (partially) resolve this issue by decoupling the wait-for-page-to-load algorithm from the navigation command, so that it can be reused for clicking, navigating backwards and forwards, and refreshing the document.

[andreastt]

We are working on https://bugzilla.mozilla.org/show_bug.cgi?id=1330348 to (partially) resolve this issue by decoupling the wait-for-page-to-load algorithm from the navigation command, so that it can be reused for clicking, navigating backwards and forwards, and refreshing the document.

[whimboo]

Just a quick update here... I'm about to land a patch for refresh which will hopefully happen today via bug 1291320. It lays the base for what's needed here. So if this is done and sticks in the tree, I will get started to work on the click and wait feature.

[whimboo]

Just a quick update here... I'm about to land a patch for refresh which will hopefully happen today via bug 1291320. It lays the base for what's needed here. So if this is done and sticks in the tree, I will get started to work on the click and wait feature.

[dekelb]

Having the exact same problem.
Tested on geckodriver v0.16.0, selenium 3.4.0, firefox 53 32bit

The problem I'm getting into is that once I click a submit button, the server replies with a 302 redirect, but also with Set-Cookie header, and since after the click I have other actions (one of the actions is getting a new page using driver.get) the browser doesn't save the cookie it got from the server.

[dekelb]

Having the exact same problem.
Tested on geckodriver v0.16.0, selenium 3.4.0, firefox 53 32bit

The problem I'm getting into is that once I click a submit button, the server replies with a 302 redirect, but also with Set-Cookie header, and since after the click I have other actions (one of the actions is getting a new page using driver.get) the browser doesn't save the cookie it got from the server.

[whimboo]

What you are referring to is an implicit wait for the click() command. I'm working on this on bug 1335778, and it will hopefully land soon for Firefox Nightly 55.0.

[whimboo]

What you are referring to is an implicit wait for the click() command. I'm working on this on bug 1335778, and it will hopefully land soon for Firefox Nightly 55.0.

[dekelb]

@whimboo thanks for your reply! I had the exact same problem with submitting the form using ENTER key (and not using click).

[dekelb]

@whimboo thanks for your reply! I had the exact same problem with submitting the form using ENTER key (and not using click).

[andreastt]

Although I don’t think the specification calls it out explicitly (and we should fix that), we will want to check for a potential document navigation on any of the “do as I mean” high-level interaction commands, which includes both Element Click and Element Send Keys.

[andreastt]

Although I don’t think the specification calls it out explicitly (and we should fix that), we will want to check for a potential document navigation on any of the “do as I mean” high-level interaction commands, which includes both Element Click and Element Send Keys.

[whimboo]

Having click working, it will be easy to add it to other high level commands too. It's just a matter of updating the specification. Once done I can make those changes.

[whimboo]

Having click working, it will be easy to add it to other high level commands too. It's just a matter of updating the specification. Once done I can make those changes.

[whimboo]

Starting with the Firefox Nightly build from today the click + page load feature will be available. If interested parties could have a look at this it would be great! I still have to fix some smaller issues, and those will be landed later today and will improve the reliability of this feature.

It means everything on this issue is fixed now and we can close it.

Btw. if we get a good amount of feedback that it works and doesn't regress something, we might consider to uplift this addition to the current Firefox 54.0 beta. So please let us know how it works! Thanks.

[whimboo]

Starting with the Firefox Nightly build from today the click + page load feature will be available. If interested parties could have a look at this it would be great! I still have to fix some smaller issues, and those will be landed later today and will improve the reliability of this feature.

It means everything on this issue is fixed now and we can close it.

Btw. if we get a good amount of feedback that it works and doesn't regress something, we might consider to uplift this addition to the current Firefox 54.0 beta. So please let us know how it works! Thanks.

[dekelb]

@whimboo thanks for that update! Does this fix relate to geckodriver or the firefox build itself? (in order to test - should I only install nightly version of firefox or also nightly (if there is) of geckodriver?) I would like to check the issue I mentioned. Thanks!

[dekelb]

@whimboo thanks for that update! Does this fix relate to geckodriver or the firefox build itself? (in order to test - should I only install nightly version of firefox or also nightly (if there is) of geckodriver?) I would like to check the issue I mentioned. Thanks!

[whimboo]

It's all handled by Firefox. So just install the latest nightly of it. Am 30.04.2017 1:46 nachm. schrieb "Dekel" <notifications@github.com>: @whimboo <https://github.com/whimboo> thanks for that update! Does this fix relate to geckodriver or the firefox build itself? (in order to test - should I only install nightly version of firefox or also nightly (if there is) of geckodriver? I would like to check the issue I mentioned. Thanks! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#308 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAH6Q-ogZ_hAQ-FqEXAY6c9U3XA0w1MDks5r1HSngaJpZM4KiJzL> .

[whimboo]

It's all handled by Firefox. So just install the latest nightly of it. Am 30.04.2017 1:46 nachm. schrieb "Dekel" <notifications@github.com>: @whimboo <https://github.com/whimboo> thanks for that update! Does this fix relate to geckodriver or the firefox build itself? (in order to test - should I only install nightly version of firefox or also nightly (if there is) of geckodriver? I would like to check the issue I mentioned. Thanks! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#308 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAH6Q-ogZ_hAQ-FqEXAY6c9U3XA0w1MDks5r1HSngaJpZM4KiJzL> .

[dekelb]

@whimboo I'm afraid it isn't fixed yet :(
Checked with the nightly (55.0a1 (2017-04-30) (64-bit)) and there are still problems there.

The form-submit wait for the returned data, but once the data is returned firefox isn't saving the cookies it got from the server, so the next request is done with the old cookies (and not the one it got from the request).

I hope the following explains my problem:

driver.get(`/login/`) # all good, received the cookie is_logged_in =False
submit_btn = driver.find_element(...) # correct element was found
submit_btn.click() # page was submitted and response has the `Set-Cookie` header with `is_logged_in =True` along with `302` header redirect to `/`
driver.get('/innerpage') # here is the problem - firefox sends the cookie is_logged_in =False and not the cookie is_logged_in = True the we just got in the above step

My way to solve this issue is to add time.sleep(2) after the submit_btn.click(), but it doesn't really makes sense to add this here. The driver/browser should finish the handling of the response before moving to the next driver.get line

[dekelb]

@whimboo I'm afraid it isn't fixed yet :(
Checked with the nightly (55.0a1 (2017-04-30) (64-bit)) and there are still problems there.

The form-submit wait for the returned data, but once the data is returned firefox isn't saving the cookies it got from the server, so the next request is done with the old cookies (and not the one it got from the request).

I hope the following explains my problem:

driver.get(`/login/`) # all good, received the cookie is_logged_in =False
submit_btn = driver.find_element(...) # correct element was found
submit_btn.click() # page was submitted and response has the `Set-Cookie` header with `is_logged_in =True` along with `302` header redirect to `/`
driver.get('/innerpage') # here is the problem - firefox sends the cookie is_logged_in =False and not the cookie is_logged_in = True the we just got in the above step

My way to solve this issue is to add time.sleep(2) after the submit_btn.click(), but it doesn't really makes sense to add this here. The driver/browser should finish the handling of the response before moving to the next driver.get line

[whimboo]

I got feedback from a colleague that at least one Mozilla owned web property is working just fine with the click+load changes.

@dekelb, this might be a special condition. So can you please file a new issue including the geckodriver trace log? Thanks

[whimboo]

I got feedback from a colleague that at least one Mozilla owned web property is working just fine with the click+load changes.

@dekelb, this might be a special condition. So can you please file a new issue including the geckodriver trace log? Thanks

[whimboo]

Btw. the remaining issue might be bug 1357634 which I'm going to fix hopefully tomorrow.

[whimboo]

Btw. the remaining issue might be bug 1357634 which I'm going to fix hopefully tomorrow.

[whimboo]

Please have a look at the Nightly build from today. All the remaining known issues have been fixed. Please let me know if it works now for you, @dekelb. Thanks.

[whimboo]

Please have a look at the Nightly build from today. All the remaining known issues have been fixed. Please let me know if it works now for you, @dekelb. Thanks.

[gcorpian]

Is there a Python 3.x version of this driver? From: Henrik Skupin <notifications@github.com<mailto:notifications@github.com>> Reply-To: mozilla/geckodriver <reply@reply.github.com<mailto:reply@reply.github.com>> Date: Tuesday, May 9, 2017 at 7:11 AM To: mozilla/geckodriver <geckodriver@noreply.github.com<mailto:geckodriver@noreply.github.com>> Cc: Graydon Corpian <graydon_corpian@harvard.edu<mailto:graydon_corpian@harvard.edu>>, Comment <comment@noreply.github.com<mailto:comment@noreply.github.com>> Subject: Re: [mozilla/geckodriver] Implicit page load not occurring on Back, Forward, Refresh, and Element Click (#308) Please have a look at the Nightly build from today. All the remaining known issues have been fixed. Please let me know if it works now for you, @dekelb<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_dekelb&d=CwMCaQ&c=WO-RGvefibhHBZq3fL85hQ&r=66ISrhiCzpaTt9YUrd6pRzO6uwQEPDOlSnt9NQhpLnE&m=zk7mvmf7D-_hZsQzONhGj4AgS2S0HgIripM4B-XFMa0&s=OSopJrkPTh7TlJayYTh1El-vwDZWqX4pXJrmBP-MYv4&e=>. Thanks. — You are receiving this because you commented. Reply to this email directly, view it on GitHub<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_mozilla_geckodriver_issues_308-23issuecomment-2D300132586&d=CwMCaQ&c=WO-RGvefibhHBZq3fL85hQ&r=66ISrhiCzpaTt9YUrd6pRzO6uwQEPDOlSnt9NQhpLnE&m=zk7mvmf7D-_hZsQzONhGj4AgS2S0HgIripM4B-XFMa0&s=eWEuqPm4QjIWiWiPuaipCKwdLxB_T4178K31fIb6nqQ&e=>, or mute the thread<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AIT1pIYDj-2D5b9xLm7FknW0zr035OAx6vks5r4EnTgaJpZM4KiJzL&d=CwMCaQ&c=WO-RGvefibhHBZq3fL85hQ&r=66ISrhiCzpaTt9YUrd6pRzO6uwQEPDOlSnt9NQhpLnE&m=zk7mvmf7D-_hZsQzONhGj4AgS2S0HgIripM4B-XFMa0&s=LoaZ4I-sjcJhjkmRX42Xmfhhc6ATgzHt6z6GERlgZmg&e=>.

[gcorpian]

Is there a Python 3.x version of this driver? From: Henrik Skupin <notifications@github.com<mailto:notifications@github.com>> Reply-To: mozilla/geckodriver <reply@reply.github.com<mailto:reply@reply.github.com>> Date: Tuesday, May 9, 2017 at 7:11 AM To: mozilla/geckodriver <geckodriver@noreply.github.com<mailto:geckodriver@noreply.github.com>> Cc: Graydon Corpian <graydon_corpian@harvard.edu<mailto:graydon_corpian@harvard.edu>>, Comment <comment@noreply.github.com<mailto:comment@noreply.github.com>> Subject: Re: [mozilla/geckodriver] Implicit page load not occurring on Back, Forward, Refresh, and Element Click (#308) Please have a look at the Nightly build from today. All the remaining known issues have been fixed. Please let me know if it works now for you, @dekelb<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_dekelb&d=CwMCaQ&c=WO-RGvefibhHBZq3fL85hQ&r=66ISrhiCzpaTt9YUrd6pRzO6uwQEPDOlSnt9NQhpLnE&m=zk7mvmf7D-_hZsQzONhGj4AgS2S0HgIripM4B-XFMa0&s=OSopJrkPTh7TlJayYTh1El-vwDZWqX4pXJrmBP-MYv4&e=>. Thanks. — You are receiving this because you commented. Reply to this email directly, view it on GitHub<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_mozilla_geckodriver_issues_308-23issuecomment-2D300132586&d=CwMCaQ&c=WO-RGvefibhHBZq3fL85hQ&r=66ISrhiCzpaTt9YUrd6pRzO6uwQEPDOlSnt9NQhpLnE&m=zk7mvmf7D-_hZsQzONhGj4AgS2S0HgIripM4B-XFMa0&s=eWEuqPm4QjIWiWiPuaipCKwdLxB_T4178K31fIb6nqQ&e=>, or mute the thread<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AIT1pIYDj-2D5b9xLm7FknW0zr035OAx6vks5r4EnTgaJpZM4KiJzL&d=CwMCaQ&c=WO-RGvefibhHBZq3fL85hQ&r=66ISrhiCzpaTt9YUrd6pRzO6uwQEPDOlSnt9NQhpLnE&m=zk7mvmf7D-_hZsQzONhGj4AgS2S0HgIripM4B-XFMa0&s=LoaZ4I-sjcJhjkmRX42Xmfhhc6ATgzHt6z6GERlgZmg&e=>.

[dekelb]

@whimboo nightly fixed the problem I described :) thanks for that quick fix!
Also note that in the nightly I don't longer have problems with send_keys (which also was problematic before - I needed to add sleep after each send_keys request). Not sure if it's related, but letting you know that it also fixed.

[dekelb]

@whimboo nightly fixed the problem I described :) thanks for that quick fix!
Also note that in the nightly I don't longer have problems with send_keys (which also was problematic before - I needed to add sleep after each send_keys request). Not sure if it's related, but letting you know that it also fixed.

[whimboo]

Is there a Python 3.x version of this driver?

@gcorpian, as it seems you are looking for a Python 3 version of the Selenium Python bindings. Please note that this is not part of this repository. You should raise this question to the Selenium folks.

@dekelb thanks for testing and letting me know.

[whimboo]

Is there a Python 3.x version of this driver?

@gcorpian, as it seems you are looking for a Python 3 version of the Selenium Python bindings. Please note that this is not part of this repository. You should raise this question to the Selenium folks.

@dekelb thanks for testing and letting me know.

[dekelb]

@whimboo any chance this fix will get into the current stable or only in ver55?

[dekelb]

@whimboo any chance this fix will get into the current stable or only in ver55?

[whimboo]

It hasn't been decided yet if we want to uplift the patch series for Firefox 54. We will discuss this today. If we want that, I might still wait a bit and would like to get more feedback first.

[whimboo]

It hasn't been decided yet if we want to uplift the patch series for Firefox 54. We will discuss this today. If we want that, I might still wait a bit and would like to get more feedback first.

[andreastt]

I think it is likely we will want to uplift it to 54, but we need to let the patch mature for a few days before we can be confident in its quality.

[andreastt]

I think it is likely we will want to uplift it to 54, but we need to let the patch mature for a few days before we can be confident in its quality.

[dekelb]

@andreastt thanks for the answer, if there is something I can help with (run tests/something else) let me know :)

[dekelb]

@andreastt thanks for the answer, if there is something I can help with (run tests/something else) let me know :)

[whimboo]

The patch just got uplifted. If all works well over the weekend, the feature will be available with the next official beta release of Firefox 54.0.

[whimboo]

The patch just got uplifted. If all works well over the weekend, the feature will be available with the next official beta release of Firefox 54.0.

[mgruner]

With Firefox 47 (without marionette), calling submit() on a WebElement caused Selenium to wait until the page loaded. This is not the case with current Firefox (55.0.2 on geckodriver 0.18). Is that intended behaviour? Our test suite relies on that implicit wait currently. Thanks for all advice.

[mgruner]

With Firefox 47 (without marionette), calling submit() on a WebElement caused Selenium to wait until the page loaded. This is not the case with current Firefox (55.0.2 on geckodriver 0.18). Is that intended behaviour? Our test suite relies on that implicit wait currently. Thanks for all advice.

[whimboo]

@mgruner this has nothing to do with this issue. Please file a new one with details as requested. Thanks.

[whimboo]

@mgruner this has nothing to do with this issue. Please file a new one with details as requested. Thanks.

[mgruner]

Of course, opened as #878.

[mgruner]

Of course, opened as #878.