Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

WIP: Add firefox html5 selenium test #14905

Closed
wants to merge 2 commits into from
Closed

WIP: Add firefox html5 selenium test #14905

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ecddda8
Add firefox html5 selenium test
BillAnastasiadis May 13, 2022
c62220a
Simplify selenium tests
BillAnastasiadis Jun 7, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
100 changes: 100 additions & 0 deletions data/selenium/selenium_html5.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
import sys

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.firefox.options import Options

res = {}
audio_codecs = [
"row-audio.codecs.pcm",
"row-audio.codecs.mp3",
"row-audio.codecs.mp4.aac",
"row-audio.codecs.mp4.ac3",
"row-audio.codecs.mp4.ec3",
"row-audio.codecs.ogg.vorbis",
"row-audio.codecs.ogg.opus"
]
img_formats = [
"row-canvas.jpeg",
"row-canvas.png",
"row-canvas.jpegxr",
"row-canvas.webp"
]
video_codecs = [
"row-video.codecs.mp4.mpeg4",
"row-video.codecs.mp4.h264",
"row-video.codecs.mp4.h265",
"row-video.codecs.ogg.theora",
"row-video.codecs.webm.vp8",
"row-video.codecs.webm.vp9",
]

options = Options()
options.headless = True
driver = webdriver.Firefox(".", options=options)
driver.get("https://html5test.opensuse.org/")
WebDriverWait(driver, 1000000).until(EC.presence_of_element_located((By.XPATH, '//*[@id="score"]/div/h2/strong')))
element = driver.find_element_by_xpath('//*[@id="score"]/div/h2/strong')
print("\n==================\n")

# TOTAL SCORE CHECK
try:
score = int(element.text)
if score < 500:
print("OVERALL SCORE: FAIL [[" + str(score) + " (<500), too low!]]")
else:
print("OVERALL SCORE: OK " + str(score) + " (>500), all good!")
except:
print("OVERALL SCORE: ERROR [[Could not parse score from html]]")
print("\n==================\n")

# VIDEO SCORE + CODEC CHECK
res["video"] = []
element = driver.find_element_by_xpath('//*[@id="head-video"]/th/div/div/span')
try:
video_score = element.text
if int(video_score.split("/")[0]) < 28:
print("VIDEO SCORE: FAIL [[Supported video codecs are <28, too low!]]")
else:
print("VIDEO SCORE: OK (Supported video codecs are " + str(video_score) + " (>28), all good!)")
for codec in video_codecs:
element = driver.find_element_by_xpath('//*[@id="' + codec + '"]/td/div')
if "No" in element.text:
res["video"].append(codec)
if res["video"]:
print("Unsupported video codecs:\n----------\n")
for codec in res["video"]:
print(codec)
else:
print("All video codecs are supported!")
except:
print("VIDEO SCORE: ERROR [[Could not parse video score from html]]")
print("\n==================")

# AUDIO SCORE + CODEC CHECK
res["audio"] = []
element = driver.find_element_by_xpath('//*[@id="head-audio"]/th/div/div/span')
try:
audio_score = element.text
if int(audio_score.split("/")[0]) < 27:
print("AUDIO SCORE: FAIL [[Supported audio codecs are " + audio_score.split("/")[0] + " (<27), too low!]]")
else:
print("AUDIO SCORE: OK (Supported audio codecs are " + str(audio_score) + " (>=27), all good!)")
for codec in audio_codecs:
element = driver.find_element_by_xpath('//*[@id="' + codec + '"]/td/div')
if "No" in element.text:
res["audio"].append(codec)
if res["audio"]:
print("Unsupported audio codecs:\n----------\n")
for codec in res["video"]:
print(codec)
else:
print("All audio codecs are supported!")
except Exception as e:
print("AUDIO SCORE: ERROR [[Could not parse score from html]]")
print(e)
print("\n==================")
60 changes: 35 additions & 25 deletions tests/x11/firefox/firefox_html5.pm
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,41 +5,51 @@
# SPDX-License-Identifier: FSFAP

# Package: MozillaFirefox
# Summary: Case#1479221: Firefox: HTML5 Video
# - Launch xterm, kill firefox, cleanup previous firefox configuration, launch
# firefox
# - Open "youtube.com/html5" and check result
# - Open "youtube.com/watch?v=Z4j5rJQMdOU" and check result
# - Exit firefox
# Maintainer: wnereiz <wnereiz@gmail.com>
# Summary: Case#1479221: Firefox: HTML5
# - Launch xterm, install python and selenium
# - Open opensuse html5 test page
# - Access various page elements and check results
# - Close
# Maintainer: vanastasiadis <vasilios.anastasiadis@suse.com>

use strict;
use warnings;
use base "x11test";
use testapi;
use utils;

sub run {
my ($self) = @_;
$self->start_firefox_with_profile;
select_console "x11";
x11_start_program('xterm');

$self->firefox_open_url('youtube.com/watch?v=Z4j5rJQMdOU');
while (check_screen([qw(firefox-youtube-signin firefox-accept-youtube-cookies)], 15)) {
if (match_has_tag('firefox-accept-youtube-cookies')) {
# get to the accept button with tab and space
wait_still_screen(2);
send_key_until_needlematch('firefox-accept-youtube-cookies-agree', 'tab', 7, 1);
assert_and_click('firefox-accept-youtube-cookies-agree');
wait_still_screen(2);
next;
}
elsif (match_has_tag('firefox-youtube-signin')) {
assert_and_click('firefox-youtube-signin');
wait_still_screen(2);
next;
# prepare python + selenium
become_root;
quit_packagekit;
zypper_call("in python3");
enter_cmd "exit";
assert_script_run("pip3 install selenium");
assert_script_run "mkdir temp_selenium && cd temp_selenium";
assert_script_run "wget https://github.com/mozilla/geckodriver/releases/download/v0.30.0/geckodriver-v0.30.0-linux64.tar.gz && tar -xf geckodriver-v0.30.0-linux64.tar.gz";
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if this can be packaged or if it's packaged already (Not for us atm, more of a rethoric question)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dcermak this is the package I was mentioning

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd say that since the dirs and scripts used by different tests won't be the same (probably - unless we standardize the way selenium is used), the only "generic" part of this is installing selenium, which is a single command

assert_script_run "export PATH=\"\$(pwd):\$PATH\"";
assert_script_run "wget --quiet " . data_url('selenium/selenium_html5.py') . " -O selenium_html5.py";

# run selenium script, parse results and upload logs
assert_script_run "python3 selenium_html5.py >&1 | tee selenium_output.txt ", timeout => 120;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the problem with this is that the assert_script_run will always pass, as it returns the exit code from the second part of the pipe. you could use script_output i guess. i havent found a better way to solve this myself. wdyt?

Copy link
Member

@foursixnine foursixnine May 19, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can make the script write the output to a file, so if something is off with the script, a failure is a failure. On top of that either pytest or TAP for python should be good enough (https://pypi.org/project/pytest-tap/)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True, that's why I write the first command output to the output.txt file and later read from that to make sure everything went fine. I do need to make the checks a bit more thorough though - thanks!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One can always set pipefail in bash

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test can be made to work in various ways (pipefail and pipestatus don't seem to work correctly though, don't know if it's openqa related or what is the exact problem) but it is not needed, as there are stuff to be done after the selenium script runs anyway, so aborting on failure was not the intended action.

my $res = script_output("cat selenium_output.txt");
my @errs = ();
foreach my $line (split(/\n/, $res)) {
if (index($line, 'FAIL') != -1) {
my ($match) = $line =~ /\[\[ ( (?:(?!\]\]).)* ) \]\]/x;
push(@errs, $match);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can look into pytest? another idea is to rather evaluate the exit code of the script, anything that is a failure, is more than 0.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The idea here was that I did not want the script failing causing a test fail. I thought that as long as all scheduled tests and checks are conducted, it's not important if the script itself failed or not. This approach does have it's problems too, though - I don't even check for missing tests (due to the script failing early, for example). Let me make it better.

About pytest I wasn't so sure about the python3.7 requirement, but I can play with it and see.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ultimately, I went with neither options, but rather checking the output file in the end. I could just let the python script fail and fail the test, but then the output file would not exist and be uploaded for the user to check details at their leisure. Pytest felt wrong for this purpose, see the related ticket for more details on that.

}
last;
}
send_key_until_needlematch('firefox-testvideo', 'spc', 30, 5);
$self->exit_firefox;
upload_logs('geckodriver.log', log_name => 'html5-geckodriver-log.txt');
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does the geckodriver-log have that could be of interest for us? any errors or things whre it should be parsed or so?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought that it would be useful to have in case of driver errors that lead to abortion of the test. But I don't see a reason to parse - if the test fails, the reviewer can take a look at the log and determine what exactly the problem is, if it's driver related

upload_logs('selenium_output.txt', log_name => 'html5-selenium-results.txt');
assert_script_run 'cd .. && rm -rf temp_selenium';
if (@errs) {
my $errors = join(" | ", @errs);
die "$errors";
}
}
1;