test fail although all commands pass in selenium-side-rinner due to hardcoded implicitWait

[mdobrzanski]

🐛 Bug Report

On slow/throttled internet connection (400 KB/s) the test fail with Operation timeout (src) although there's only one simple open command in the test.

Test passes just fine using Selenium IDE but not using selenium-side-runner.
The parameter --timeout for selenium-side-runner does not help.
Tampering with implicitWait in WebDriverExecutor in the source code does the trick but I don't see elegant way to modify the implicitWait. Is there?

To Reproduce

• throttle the internet connection to 400 KB/s (I used NetLimiter)
• run below simple side file that only opens https://beta.slashdb.com using selenium-side-runner

Expected behavior

Expected is that the test pass since all steps passed successfully or a way to pass implicitWait in command line argument.

Project file reproducing this issue (highly encouraged)

{
  "id": "0be60562-68da-41fb-a871-fc6943ccf780",
  "version": "2.0",
  "name": "implicit-timeout-issue",
  "url": "https://beta.slashdb.com",
  "tests": [{
    "id": "c1e829fb-fa2f-476c-906d-0394418d7307",
    "name": "simple open",
    "commands": [{
      "id": "d18dd331-7431-41aa-afa4-405f10be8d5a",
      "comment": "",
      "command": "open",
      "target": "/",
      "targets": [],
      "value": ""
    }]
  }],
  "suites": [{
    "id": "daa4df18-c054-45a9-8c04-1e719dce6dc3",
    "name": "Default Suite",
    "persistSession": false,
    "parallel": false,
    "timeout": 300,
    "tests": ["c1e829fb-fa2f-476c-906d-0394418d7307"]
  }],
  "urls": [],
  "plugins": []
}

Environment

OS: Windows 10
Selenium IDE Version: 4.0.0-alpha.30
Selenium SIDE Runner Version: 4.0.0-alpha.34
Node version: v18.12.1
Browser: Chrome
Browser Version: 108.0.5359.125 (Official Build) (64-bit)

[toddtarsi]

Thank you for providing a repro! There are a few places you can tweak this timeout. The implicit timeout is a suite level control. Additionally, it can be provided with a custom value via the -t flag ex selenium-side-runner -f "Default Suite" -t 15000 ./test.side

[toddtarsi]

So, to recap, you can:

1. Edit the side file json directly (it is currently only set to 300 milliseconds in your file)
2. In the IDE you can navigate to the suites tab, click the little pencil in the top right, and provide a custom "timeout" for your suite that makes sense for your needs.
3. Specify the timeout manually via the -t flag in the side-runner.

Please let me know if none of these solve your issue and we can chat about what you really need. Thanks!

[mdobrzanski]

@toddtarsi , when I try to run tests in the suite I get different error

mike@robocop:~/Downloads$ selenium-side-runner -f "Default Suite" -t 15000 ./test.side
 FAIL  ../.nvm/versions/node/v19.0.0/lib/node_modules/selenium-side-runner/dist/main.test.js
  Running project implicit-timeout-issue
    Running suite Default Suite
      ✕ Running test $name

  ● Running project implicit-timeout-issue › Running suite Default Suite › Running test $name

    Error: `.each` called with an empty Array of table data.

      at test (node_modules/jest-each/build/index.js:38:47)
      at src/main.test.ts:145:7
          at Array.forEach (<anonymous>)
      at src/main.test.ts:136:18

Test Suites: 1 failed, 1 total
Tests:       1 failed, 1 total
Snapshots:   0 total
Time:        0.628 s

I think it's caused by the same configuration.filter being applied to find test suits main.test.ts#L133 and test names main.test.ts#L140 as well. It seems these should be two separate filters.

I've modified the project side file so that the test suit name and the test name start the same way "Default Suite" and "Default Suite - simple open". Then the tests were found and executed. But, I suppose that is a separate issue.

BTW in real case I don't use suites at all. I run all tests or filter for something specific tests. To me, it would be nice to have control over implicitWait for each tests or all tests run in that session.

Back to original issue.

I've tried both of your suggestions i.e. setting timeout in suite or passing -t 15000 argument to the command but that unfortunately didn't help.

I suspect that commandExecutor.implicitWait command-node.ts#L136 is not set by timeout option.

Here's what I did:

• updated timeout in the test suite in the side file to 12000
  and test name to "Default Suite - simple open"

  {
  "id": "0be60562-68da-41fb-a871-fc6943ccf780",
  "version": "2.0",
  "name": "implicit-timeout-issue",
  "url": "https://beta.slashdb.com",
  "tests": [{
    "id": "c1e829fb-fa2f-476c-906d-0394418d7307",
    "name": "Default Suite - simple open",
    "commands": [{
      "id": "d18dd331-7431-41aa-afa4-405f10be8d5a",
      "comment": "",
      "command": "open",
      "target": "/",
      "targets": [],
      "value": ""
    }]
  }],
  "suites": [{
    "id": "daa4df18-c054-45a9-8c04-1e719dce6dc3",
    "name": "Default Suite",
    "persistSession": false,
    "parallel": false,
    "timeout": 12000,
    "tests": ["c1e829fb-fa2f-476c-906d-0394418d7307"]
  }],
  "urls": [],
  "plugins": []
  }
  

• use command argument -t 15000
• noted down that default commandExecutor.implicitWait is 5000 webdriver.ts#L162
• added console.log to print out value of commandExecutor.implicitWait just before running retryCommand in command-node.ts#L134
• ran command selenium-side-runner -f "Default Suite" -t 15000 ./test.side

The test failed and console.log returned 5000 which tells me that default implicitWait was used and other options don't override the default.

Am I missing something or the timeout options is used for something else?
Do you think there's another way to override the default implicitWait?

[toddtarsi]

@mdobrzanski - Good catch! I will ensure that timeout is passed through properly. I'm still trying to figure out how to deal with that each empty thing. Sorry this took a long time for follow up, I have been taking some distance from the project lately.

[toddtarsi]

Okay, should be resolved in .37, thanks for reporting!

[mdobrzanski]

@toddtarsi I see the fix for filter in the commit 2e721cb , thanks.

What do you think about original problem with implicitWait? I tried -t like you suggested but that didn't work. Do you think there's a way to set the implicitWait?

[toddtarsi]

Hm! I'm surprised. I amended the executor to pass that in here:

https://github.com/SeleniumHQ/selenium-ide/blob/trunk/packages/side-runner/src/run.ts#L60

If that didn't do it, rats! What command are you encountering this on? Maybe we're doing something funky with that command.

[mdobrzanski]

@toddtarsi I've rebuild everything from scratch and now it works like a charm. 🎉 Sorry for the false alarm.

Thank you again.

[toddtarsi]

@mdobrzanski - Hooray!

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.