Fail on unexpected Javascript console errors

[kalikiana]

The function merely returns a truthy value. Let's fail always without relying on the caller. This means we will see errors in all cases, not just when something else happens to fail.

I'm also dropping a comment claiming the function aborts.

See: https://progress.opensuse.org/issues/98952

[okurz]

I am not convinced to move the Test::More calls into the function as I learned that test failure feedback works best if the scope of such calls is the top-level test code and everything on a lower level should return the errors as boolean result

[Martchus]

The advantage of this approach is that we cannot forget to actually use the return value but I also get @okurz 's point. It would be great if Perl had a [[nodiscard]] attribute like C++ but it seems like it hasn't.

[kalikiana]

No idea what this is:

[18:23:14] t/full-stack.t .. 1/? Could not obtain new session: unknown error: Chrome failed to start: crashed.
  (chrome not reachable)
  (The process started from chrome location /usr/lib64/chromium/chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)
  (Driver info: chromedriver=94.0.4606.71 (1d32b169326531e600d836bd395efc1b53d0f6ef-refs/branch-heads/4606@{#1256}),platform=Linux 4.15.0-1110-aws x86_64) at /home/squamata/project/t/lib/OpenQA/SeleniumTest.pm line 98.
[18:23:14] t/full-stack.t .. 2/? # Tests were run but no plan was declared and done_testing() was not seen.
[18:23:14] t/full-stack.t ..      Dubious, test returned 254 (wstat 65024, 0xfe00)
All 2 subtests passed

[okurz]

chrome also crashed for me in #4289

[okurz]

chrome problems worked around in #4291 , I had already approved. Approval still holds :)

Looks like the test failures are actually related.

[Martchus]

Maybe this warning should be ignored?

Bailout called.  Further testing stopped:  Job 1 produced the wrong results
FAILED--Further testing stopped: Job 1 produced the wrong results
Calling retry hook ./tools/delete-coverdb-folder
Deleting cover_db_fullstack/
Retry 7 of 15 …
[09:08:22] t/full-stack.t .. 32/?     # Unexpected Javascript console errors: [
    #   {
    #     level     => "SEVERE",
    #     message   => "http://localhost:9526/asset/2f2b2ee008/test_result.js 289 WebSocket connection to 'ws://localhost:9528/liveviewhandler/tests/1/developer/ws-proxy/status' failed: Close received after close",
    #     source    => "network",
    #     timestamp => 1634893738701,
    #   },
    # ]

    #   Failed test 'No unexected js warnings'

[kalikiana]

This looks like the symptom of the json file never being written:

not ok 60 - developer session locked for other developers
Subtest: developer session locked for other developers
[debug] [pid:2726] Unable to read result-shutdown.json: Can't open file "/tmp/zm6SQvzhGB/full-stack.d/openqa/pool/1/testresults/result-shutdown.json": No such file or directory at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/File.pm line 132.

[kalikiana]

Maybe this warning should be ignored?

Bailout called.  Further testing stopped:  Job 1 produced the wrong results
FAILED--Further testing stopped: Job 1 produced the wrong results
Calling retry hook ./tools/delete-coverdb-folder
Deleting cover_db_fullstack/
Retry 7 of 15 …
[09:08:22] t/full-stack.t .. 32/?     # Unexpected Javascript console errors: [
    #   {
    #     level     => "SEVERE",
    #     message   => "http://localhost:9526/asset/2f2b2ee008/test_result.js 289 WebSocket connection to 'ws://localhost:9528/liveviewhandler/tests/1/developer/ws-proxy/status' failed: Close received after close",
    #     source    => "network",
    #     timestamp => 1634893738701,
    #   },
    # ]

    #   Failed test 'No unexected js warnings'

This is the one I'd added before but dropped when making all fatal. Which means I'm adding it back now.

[codecov]

Codecov Report

Merging #4288 (48664aa) into master (ba3ae18) will decrease coverage by 0.00%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #4288      +/-   ##
==========================================
- Coverage   97.94%   97.94%   -0.01%     
==========================================
  Files         371      371              
  Lines       33705    33704       -1     
==========================================
- Hits        33012    33011       -1     
  Misses        693      693              

Impacted Files	Coverage Δ
t/full-stack.t	96.96% <100.00%> (ø)
t/lib/OpenQA/SeleniumTest.pm	97.46% <100.00%> (-0.02%)	⬇️
t/lib/OpenQA/Test/FullstackUtils.pm	100.00% <100.00%> (ø)
t/ui/15-comments.t	100.00% <100.00%> (ø)
t/ui/16-tests_dependencies.t	100.00% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ba3ae18...48664aa. Read the comment docs.

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏