Make sure the firefox menu can show up

[tinawang123]

Make sure the firefox menu can show up. Before update the code,
Checked https://openqa.opensuse.org/tests/2245717#next_previous.
From firefox first failed case, there are 2/19 test cases failed.
Run the case with this PR, for sle cases, this problem cannot
reproduce again. For TW cases, run 100 cases, all of them can
pass. According those data, I think this PR can fix the menu
cannot show up problem.

• Related ticket: https://progress.opensuse.org/issues/103533
• Verification run:
  opensuse:https://openqa.opensuse.org/t2232952
  https://openqa.opensuse.org/t2232953
  https://openqa.opensuse.org/t2232954
  https://openqa.opensuse.org/t2232955
  https://openqa.opensuse.org/t2232956
  https://openqa.opensuse.org/t2232957
  https://openqa.opensuse.org/t2232958
  https://openqa.opensuse.org/t2232959
  https://openqa.opensuse.org/t2232960
  https://openqa.opensuse.org/t2232961
  sles:
  http://openqa.suse.de/tests/8286759
  http://openqa.suse.de/tests/8286755

[okurz]

In your git commit message you repeat the same sentence twice.
Instead the commit message could have a bit more details, e.g. what issue is fixed, why this needs to change, to which versions of which product it applies, link to a bug or a fate entry, the choices you made, etc. Also see https://commit.style/ or http://chris.beams.io/posts/git-commit/ as a helpful guide how to write good commit messages.

I suggest to collect the current failure ratio and come up with a value in percent on a big enough data set. Then prove that with your fix the fail ratio is significantly lower

[tinawang123]

In your git commit message you repeat the same sentence twice. Instead the commit message could have a bit more details, e.g. what issue is fixed, why this needs to change, to which versions of which product it applies, link to a bug or a fate entry, the choices you made, etc. Also see https://commit.style/ or http://chris.beams.io/posts/git-commit/ as a helpful guide how to write good commit messages.

I suggest to collect the current failure ratio and come up with a value in percent on a big enough data set. Then prove that with your fix the fail ratio is significantly lower

Updated.

[okurz]

ok, this looks promising already. I just updated https://progress.opensuse.org/projects/openqatests/wiki/Wiki#Statistical-investigation so you could use the proposed solution there to trigger jobs in special builds so instead of referencing individual jobs you could reference a test overview page showing the complete results so that we could easily see the fail-ratio. Also I suggest to use http://open.qa/docs/#_defining_a_custom_test_schedule_or_custom_test_modules so for example you could exclude some module or use a custom schedule. I guess the most minimum case could be TESTS=tests/boot/boot_to_desktop,tests/x11/firefox

[tinawang123]

ok, this looks promising already. I just updated https://progress.opensuse.org/projects/openqatests/wiki/Wiki#Statistical-investigation so you could use the proposed solution there to trigger jobs in special builds so instead of referencing individual jobs you could reference a test overview page showing the complete results so that we could easily see the fail-ratio. Also I suggest to use http://open.qa/docs/#_defining_a_custom_test_schedule_or_custom_test_modules so for example you could exclude some module or use a custom schedule. I guess the most minimum case could be TESTS=tests/boot/boot_to_desktop,tests/x11/firefox

Run 100 times, https://openqa.opensuse.org/tests/overview?build=poo103533_investigation&version=Tumbleweed&distri=opensuse
All of them about firefox can pass

[BillAnastasiadis]

Thank you @tinawang123 for this contribution. I think if you change the commit message title from "To make sure the firefox menu can show up " to "Make sure the firefox menu can show up ", the commit would be ready to merge.

[tinawang123]

Thank you @tinawang123 for this contribution. I think if you change the commit message title from "To make sure the firefox menu can show up " to "Make sure the firefox menu can show up ", the commit would be ready to merge.

Thanks , changed

[BillAnastasiadis]

Thank you! Could you please also change it in the git message along with the github one? Meaning, git commit --amend and then changing the commit message title.

[tinawang123]

Thank you! Could you please also change it in the git message along with the github one? Meaning, git commit --amend and then changing the commit message title.

Updated

[BillAnastasiadis]

Thanks @tinawang123!
@okurz do you think we can give this a go?

[tinawang123]

@okurz any update?

[okurz]

ok, this looks promising already. I just updated https://progress.opensuse.org/projects/openqatests/wiki/Wiki#Statistical-investigation so you could use the proposed solution there to trigger jobs in special builds so instead of referencing individual jobs you could reference a test overview page showing the complete results so that we could easily see the fail-ratio. Also I suggest to use http://open.qa/docs/#_defining_a_custom_test_schedule_or_custom_test_modules so for example you could exclude some module or use a custom schedule. I guess the most minimum case could be TESTS=tests/boot/boot_to_desktop,tests/x11/firefox

Run 100 times, https://openqa.opensuse.org/tests/overview?build=poo103533_investigation&version=Tumbleweed&distri=opensuse All of them about firefox can pass

That's great. I think I still need to be more clear. A proper scientific approach to prove that the hypothesis is correct does need the fail ratio from before though. It is great to see that firefox can pass in 100/100 cases, meaning a fail ratio of 0.00%+-0.01% but if the fail ratio in before was below 0.01% percent this still would not be an improvement. I suggest to include such information in the git commit message now.

[tinawang123]

ok, this looks promising already. I just updated https://progress.opensuse.org/projects/openqatests/wiki/Wiki#Statistical-investigation so you could use the proposed solution there to trigger jobs in special builds so instead of referencing individual jobs you could reference a test overview page showing the complete results so that we could easily see the fail-ratio. Also I suggest to use http://open.qa/docs/#_defining_a_custom_test_schedule_or_custom_test_modules so for example you could exclude some module or use a custom schedule. I guess the most minimum case could be TESTS=tests/boot/boot_to_desktop,tests/x11/firefox

Run 100 times, https://openqa.opensuse.org/tests/overview?build=poo103533_investigation&version=Tumbleweed&distri=opensuse All of them about firefox can pass

That's great. I think I still need to be more clear. A proper scientific approach to prove that the hypothesis is correct does need the fail ratio from before though. It is great to see that firefox can pass in 100/100 cases, meaning a fail ratio of 0.00%+-0.01% but if the fail ratio in before was below 0.01% percent this still would not be an improvement. I suggest to include such information in the git commit message now.

Checked https://openqa.opensuse.org/tests/2245717#next_previous. From firefox first failed case, there are 2/19 test cases failed.

[okurz]

Checked https://openqa.opensuse.org/tests/2245717#next_previous. From firefox first failed case, there are 2/19 test cases failed.

ok, can you include that information about the current state and the results from your statistics after your changes in the git commit message?

[tinawang123]

Checked https://openqa.opensuse.org/tests/2245717#next_previous. From firefox first failed case, there are 2/19 test cases failed.

ok, can you include that information about the current state and the results from your statistics after your changes in the git commit message?

Updated git commit information