WIP: Add firefox html5 selenium test #14905

BillAnastasiadis · 2022-05-13T13:39:00Z

still WIP
This is a POC exploring ways to use python-selenium in openqa for specific tests. The test runs a python script which uses selenium to access https://html5test.opensuse.org/ and check a number of values and scores, which are then parsed in perl to determine success/failure of the test.

The test also uploads the geckodriver log and the full test result log.

(limits and checks are arbitrarily selected for the sake of demonstration)

Related ticket: https://progress.opensuse.org/issues/109896
Verification runs:
example of successful run: http://10.161.229.209/tests/640
example of failed run (enforced higher score expectations): http://10.161.229.209/tests/645#step/firefox_html5/31

BillAnastasiadis · 2022-05-17T10:33:44Z

@foursixnine the script is - for the time being - in data, exactly where the ticket said it shouldn't be :p

The question here is: ImageMagick also makes use of a (bash) script, which also resides in data. If you take this into account, it makes sense that possible future selenium scripts would reside in a dedicated folder in data too.

On the other hand, the scripts could also reside in the dedicated scripts repo of os-autoinst. No personal preference between the two. I'm open to suggestions.

b10n1k · 2022-05-19T06:02:33Z

tests/x11/firefox/firefox_html5.pm

+    assert_script_run "wget --quiet " . data_url('selenium/selenium_html5.py') . " -O selenium_html5.py";
+
+    # run selenium script, parse results and upload logs
+    assert_script_run "python3 selenium_html5.py >&1 | tee selenium_output.txt ", timeout => 120;


the problem with this is that the assert_script_run will always pass, as it returns the exit code from the second part of the pipe. you could use script_output i guess. i havent found a better way to solve this myself. wdyt?

You can make the script write the output to a file, so if something is off with the script, a failure is a failure. On top of that either pytest or TAP for python should be good enough (https://pypi.org/project/pytest-tap/)

True, that's why I write the first command output to the output.txt file and later read from that to make sure everything went fine. I do need to make the checks a bit more thorough though - thanks!

One can always set pipefail in bash

The test can be made to work in various ways (pipefail and pipestatus don't seem to work correctly though, don't know if it's openqa related or what is the exact problem) but it is not needed, as there are stuff to be done after the selenium script runs anyway, so aborting on failure was not the intended action.

foursixnine · 2022-05-19T08:22:15Z

tests/x11/firefox/firefox_html5.pm

+    foreach my $line (split(/\n/, $res)) {
+        if (index($line, 'FAIL') != -1) {
+            my ($match) = $line =~ /\[\[ ( (?:(?!\]\]).)* ) \]\]/x;
+            push(@errs, $match);


Maybe we can look into pytest? another idea is to rather evaluate the exit code of the script, anything that is a failure, is more than 0.

The idea here was that I did not want the script failing causing a test fail. I thought that as long as all scheduled tests and checks are conducted, it's not important if the script itself failed or not. This approach does have it's problems too, though - I don't even check for missing tests (due to the script failing early, for example). Let me make it better.

About pytest I wasn't so sure about the python3.7 requirement, but I can play with it and see.

Ultimately, I went with neither options, but rather checking the output file in the end. I could just let the python script fail and fail the test, but then the output file would not exist and be uploaded for the user to check details at their leisure. Pytest felt wrong for this purpose, see the related ticket for more details on that.

foursixnine · 2022-05-19T08:25:58Z

tests/x11/firefox/firefox_html5.pm

+    enter_cmd "exit";
+    assert_script_run("pip3 install selenium");
+    assert_script_run "mkdir temp_selenium && cd temp_selenium";
+    assert_script_run "wget https://github.com/mozilla/geckodriver/releases/download/v0.30.0/geckodriver-v0.30.0-linux64.tar.gz && tar -xf geckodriver-v0.30.0-linux64.tar.gz";


I wonder if this can be packaged or if it's packaged already (Not for us atm, more of a rethoric question)

@dcermak this is the package I was mentioning

I'd say that since the dirs and scripts used by different tests won't be the same (probably - unless we standardize the way selenium is used), the only "generic" part of this is installing selenium, which is a single command

foursixnine · 2022-05-19T08:26:52Z

tests/x11/firefox/firefox_html5.pm

    }
-    send_key_until_needlematch('firefox-testvideo', 'spc', 30, 5);
-    $self->exit_firefox;
+    upload_logs('geckodriver.log', log_name => 'html5-geckodriver-log.txt');


What does the geckodriver-log have that could be of interest for us? any errors or things whre it should be parsed or so?

I thought that it would be useful to have in case of driver errors that lead to abortion of the test. But I don't see a reason to parse - if the test fails, the reviewer can take a look at the log and determine what exactly the problem is, if it's driver related

stale · 2022-09-09T01:26:34Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
If it's still needed, you can add labels WIP or notready to it

stale · 2022-09-24T23:13:48Z

This Pull Request has been automatically closed as it didn't have any activity in the last 97 days. Thank you.

Add firefox html5 selenium test

ecddda8

BillAnastasiadis force-pushed the firefox_selenium branch from 0354e5f to ecddda8 Compare May 17, 2022 10:27

BillAnastasiadis requested a review from foursixnine May 17, 2022 10:29

b10n1k reviewed May 19, 2022

View reviewed changes

foursixnine reviewed May 19, 2022

View reviewed changes

BillAnastasiadis force-pushed the firefox_selenium branch from a4b6931 to 794e766 Compare June 7, 2022 11:00

Simplify selenium tests

c62220a

BillAnastasiadis force-pushed the firefox_selenium branch from 794e766 to c62220a Compare June 7, 2022 12:23

stale bot added the stale Stale pull request, are to be deleted within 90 days label Sep 9, 2022

stale bot closed this Sep 24, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: Add firefox html5 selenium test #14905

WIP: Add firefox html5 selenium test #14905

BillAnastasiadis commented May 13, 2022 •

edited

Loading

BillAnastasiadis commented May 17, 2022 •

edited

Loading

b10n1k May 19, 2022

foursixnine May 19, 2022 •

edited

Loading

BillAnastasiadis May 24, 2022

mloviska May 24, 2022

BillAnastasiadis Jun 7, 2022

foursixnine May 19, 2022

BillAnastasiadis May 24, 2022

BillAnastasiadis Jun 7, 2022

foursixnine May 19, 2022

foursixnine May 19, 2022

BillAnastasiadis May 24, 2022

foursixnine May 19, 2022

BillAnastasiadis May 24, 2022

stale bot commented Sep 9, 2022

stale bot commented Sep 24, 2022

WIP: Add firefox html5 selenium test #14905

WIP: Add firefox html5 selenium test #14905

Conversation

BillAnastasiadis commented May 13, 2022 • edited Loading

BillAnastasiadis commented May 17, 2022 • edited Loading

Choose a reason for hiding this comment

foursixnine May 19, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stale bot commented Sep 9, 2022

stale bot commented Sep 24, 2022

BillAnastasiadis commented May 13, 2022 •

edited

Loading

BillAnastasiadis commented May 17, 2022 •

edited

Loading

foursixnine May 19, 2022 •

edited

Loading