https://github.com/jakartaee/faces/pull/1732: Selenium tests for the ajax TCK (master branch) #1770

werpu · 2022-12-14T14:52:39Z

I have re-ported the Ajax tests again to Selenium but this time properly in master.
Main differences: Latest framework base version (note this is under ASL2 license)
tests now can be started via -Dtest.selenium=true this will skip the HTMLUnitTests and run the selenium tests
if you leave that out the tests behave as before.
I will close the old pull request, but link it in.

…tests!

werpu · 2022-12-14T14:54:30Z

Old pull request (now closed): #1732

(accidentally added debugging version)

BalusC · 2022-12-14T15:48:02Z

tck/faces22/ajax/src/test/java/ee/jakarta/tck/faces/test/servlet30/ajax/Issue2162IT.java

+ * @see AjaxBehavior
+ * @see https://github.com/eclipse-ee4j/mojarra/issues/2166
+ */
+


I guess this change was unintentional?

Yes very likely a copy paste error, will fix that tomorrow.

…r version

werpu · 2023-01-09T08:30:09Z

Fixed... there was some code which was carried over from the old port, it now is back in sync with the master branch!
Note, the new selenium tests reside in their own packages to allow to run them separately.
They are not replacing the old codebase for now! (I wanted to make this as non intrusive as possible)
(hence also the -Dtest.selenium=true flag, to switch between the old and new tests - see first comment)

…so relies on ajax!

smaller rename for the custom runners

werpu · 2023-01-12T14:18:07Z

ping, is this going to be merged?

arjantijms

Approve in principal. We do need to make double sure this works correctly on our Eclipse CI.

dmatej

I tried to run all tests locally, just Issue2162IT fails, not sure why. I will run it yet with -fae.

[INFO] Running ee.jakarta.tck.faces.test.servlet30.ajax_selenium.Issue2162IT
Jan 12, 2023 5:36:32 PM org.openqa.selenium.devtools.CdpVersionFinder findNearestMatch
WARNING: Unable to find an exact match for CDP version 108, so returning the closest version found: 107
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.533 s <<< FAILURE! - in ee.jakarta.tck.faces.test.servlet30.ajax_selenium.Issue2162IT
[ERROR] ee.jakarta.tck.faces.test.servlet30.ajax_selenium.Issue2162IT.testIssue2162  Time elapsed: 0.43 s  <<< FAILURE!
java.lang.AssertionError
at ee.jakarta.tck.faces.test.servlet30.ajax_selenium.Issue2162IT.testIssue2162(Issue2162IT.java:52)

werpu · 2023-01-13T09:05:02Z

Thanks for report investigating it, seems like this is a mojarra issue (I have had marked this code for further investigation, but forgot to do it)

The test is exactly the same as in HTMLUnit, but it seems like the render all issued in this case fails on Chrome.

Here is the same test running on MyFaces and our reimplemented faces.ts scripts:

[INFO] Running ee.jakarta.tck.faces.test.servlet30.ajax_new.Issue2162IT
Jan 12, 2023 1:46:47 PM org.openqa.selenium.devtools.CdpVersionFinder findNearestMatch
WARNING: Unable to find an exact match for CDP version 108, so returning the closest version found: 107
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.724 s - in ee.jakarta.tck.faces.test.servlet30.ajax_new.Issue2162IT

So the tests passes clearly on MyFaces with the new scripts, I suspect the problem might be somewhere on the faces.js side on Mojarra, which probably uses somewhere a construct which fails on newer browsers but passes on html unit.

I will do a deeper research on it. Btw. not the only case where one of the testcases failed on Chrome, there were 2-3 tests which relied on non dom compliant usage of a Textarea as result logger, which resulted in failing tests on Chrome. Those were fixed by me. But this is not test related. I will come back later with the results of what I can find out.

I also will have another look at the CDP Warning, this is ugly!

Update:
The response looks ok in Chrome: https://gist.github.com/werpu/c1872f400706e1cefbdfc375bfc3a9f4
But the page is not updated accordingly. So the bug is somewhere in the mojarra response handling code on the javascript side.

dmatej · 2023-01-13T10:02:03Z

It seems the cause might be dependency on local operating system settings, I have cs_CZ lang and locale, when I was running unchanged version of this test from GlassFish, it passed (all tests).
In faces TCK we have this:


                    <environmentVariables>
                        <LC_ALL>C</LC_ALL>
                    </environmentVariables>

But it also inherits this from our basic surefire+failsafe settings:

                        <environmentVariables>
                            <LANG>en</LANG>
                        </environmentVariables>

So I believe it is not a problem of this PR.
EDIT: Rerun, I wasn't right, locale did not resolve it.

werpu · 2023-01-13T10:22:35Z

Found the issue regarding the failing test. In fact, this is the first advantage we see here from browser testing. It indeed either is a Chrome bug or a Mojarra issue, but in fact following happens.
Mojarra processes the code just fine until it does the body replacement.

Following code is the problem:

var copyChildNodes = function copyChildNodes(nodeFrom, nodeTo) {
         ...
            } else {
                var ownerDoc = nodeTo.nodeType == Node.DOCUMENT_NODE ? nodeTo : nodeTo.ownerDocument;
                var i;
                if (typeof (ownerDoc.importNode) != "undefined") {
                    for (i = 0; i < nodes.length; i++) {
                        nodeTo.appendChild(ownerDoc.importNode(nodes[i], true))
                    }
                } 
...
        };

What happens here is that:
nodeTo.appendChild(ownerDoc.importNode(nodes[i], true))

the ownerDoc.importNode swallows text nodes and drops them. MyFaces does not run into this issue, because my code does not use importNode for the dom element replacement.

Here you can see the browser console output regarding what happens.
The Init Called text nodes of the incoming node is swallowed and is not present anymore after the importNode call.

I will file a bug report on this on the RI side.
So also not a problem of the unit test itself but the test exposes it. Even if it is a browser bug, it very likely has to be fixed on the RI side, given the wide exposure the chromium engine has.

I have filed an issue for this under: #1776

werpu · 2023-01-13T11:24:09Z

Re the CDP Warning, this seems to be a Selenium issue. The Chrome driver and the underlying Selenium version seem to be in a slight mis-alignment regarding the Chromium Debug tools version (which I use internally to provide some functionality, which HTMLUnit has per default, but the chrome driver does not)

It is a warning, but nevertheless I will look into it, how I can fix this.
In the worst case shutting off the warning should suffice, because the fallback picks the correct version anyway, in my opinion this would justify more as an info than a warning, but oh well!

melloware · 2023-01-13T11:35:54Z

Awesome analysis and explanation!

…st selenium webdrivers

werpu · 2023-01-13T12:38:22Z

I have fixed the CSP warning in a way that it is displayed only once. Note... Selenium atm only supports chrome until 108, so a warning is expected if you trigger a newer webdriver. It does not really matter because it works nevertheless.

Also we might have to do smaller adjustments on the code for ci integration because the webdrivermanager downloads the chrome webdrivers for the installed chrome automatically. This might cause problems with proxy environments, or if we do not want that. There are other ways to initialize an existing webdriver. I might add a manual driver location setup via environmental parameters on Monday!

werpu · 2023-01-13T13:20:15Z

Re CI Integration, please collect info what you still need, then I can add it on monday.

…auling Spec1423 so that it uses less timing and more waiting for page conditions to avoid race conditions

BalusC · 2023-01-17T13:58:51Z

tck/faces23/ajax/src/test/java/ee/jakarta/tck/faces/test/javaee8/ajax_selenium/Spec1423IT.java

+        button.click();
+        page.waitReqJs(Duration.ofMillis(6000));
+        assertTrue(page.findElement(By.id("scriptResult")).getText().trim().equals("addedViaHead"));
+        assertTrue(page.findElement(By.id("stylesheetResult")).getText().isEmpty());


Why not page.waitForCondition()?

BalusC · 2023-01-17T13:59:21Z

tck/faces23/ajax/src/test/java/ee/jakarta/tck/faces/test/javaee8/ajax_selenium/Spec1423IT.java

+
+
+        assertTrue(page.findElement(By.id("scriptResult")).getText().trim().equals("addedViaHead"));
+        assertTrue(page.findElement(By.id("stylesheetResult")).getText().isEmpty());


Unnecessary, these are already asserted by page.waitForCondition().

BalusC · 2023-01-17T13:59:34Z

tck/faces23/ajax/src/test/java/ee/jakarta/tck/faces/test/javaee8/ajax_selenium/Spec1423IT.java

+        button.click();
+        page.waitReqJs(Duration.ofMillis(6000));
+        assertTrue(page.findElement(By.id("scriptResult")).getText().trim().equals("addedViaBody"));
+        assertTrue(page.findElement(By.id("stylesheetResult")).getText().trim().equals("rgb(255, 0, 0)"));


Why not page.waitForCondition()?

BalusC · 2023-01-17T13:59:44Z

tck/faces23/ajax/src/test/java/ee/jakarta/tck/faces/test/javaee8/ajax_selenium/Spec1423IT.java

+
+        page.waitReqJs(Duration.ofMillis(6000));
+        assertTrue(page.findElement(By.id("scriptResult")).getText().trim().equals("addedViaBody"));
+        assertTrue(page.findElement(By.id("stylesheetResult")).getText().trim().equals("rgb(255, 0, 0)"));


Why not page.waitForCondition()?

BalusC · 2023-01-17T14:01:28Z

tck/faces23/ajax/src/test/java/ee/jakarta/tck/faces/test/javaee8/ajax_selenium/Spec1423IT.java

+        page.waitReqJs(Duration.ofMillis(6000));
+
+        assertTrue(page.findElement(By.id("stylesheetResult")).getText().trim().equals("rgb(0, 255, 0)"));
+        assertTrue(page.findElement(By.id("scriptResult")).getText().trim().equals("addedProgrammatically"));


Why not page.waitForCondition()?

waitforCondition is only possible if a state change is supposed to happen after a click.
I will recheck the code if I missed a state change, but that is the basic premise where it can be applied in this case (see also the other comments)

werpu · 2023-01-17T14:02:35Z

WaitforCondition means we wait for a change, if there is none, we still have to use normale timed waits, not ideal, but there is no other way. I have added waitForCondition wherever it made sense for this test!

BalusC · 2023-01-17T14:03:18Z

tck/faces23/ajax/src/test/java/ee/jakarta/tck/faces/test/javaee8/ajax_selenium/Spec1423IT.java

+        assertTrue(page.findElement(By.id("scriptResult")).getText().trim().equals("addedViaBody"));
+        assertTrue(page.findElement(By.id("stylesheetResult")).getText().trim().equals("rgb(255, 0, 0)"));
+
+        button = page.findElement(By.id("form2:addViaInclude"));


It's probably better if page was reloaded so that the script/style results of previous test step are guaranteed to be cleared out.

yes that would mean a complete test rewrite, also this would miss the double include check. (Aka if a script already is loaded it should not be executed a second time)
This test is rather complicated in a sense that it tests for several things at once., first whether the elements are updated, and seondly, that the page does not run into an illegal update state by retriggering certain update patterns over and over again in various combinations.

Btw generally spoken, it would make sense to replace the waits with waitForCondition in the other tests as well, but for this one it was the most pressing.

The main difference and that is a huge one, between HtmlUnit and Selenium is, that with Selenium the browser runs in a separate process.

Hence there is no real synchronisation, between the driver and the browser. The best bet is to wait until the dom is updated into the condition you want to assert, if the browser does a long running or asynchronous operation, otherwise you might run into unwanted race conditions if you set your wait windows too tight!

BalusC · 2023-01-17T14:03:54Z

tck/faces23/ajax/src/test/java/ee/jakarta/tck/faces/test/javaee8/ajax_selenium/Spec1423IT.java

+        });
+
+
+        button = page.findElement(By.id("form1:addViaHead"));


It's probably better if page was reloaded so that the script/style results of previous test step are guaranteed to be cleared out.

See above for my comment on this! Not possible!

volosied · 2023-01-20T14:29:57Z

Hi, I noticed that at least one test is different (and I can look for others):

ajax_selenium: https://github.com/jakartaee/faces/blob/309ddc0a060846d28395e963d69bdb080025b845/t[…]/jakarta/tck/faces/test/servlet50/ajax_selenium/Spec1567IT.java

original:
https://github.com/jakartaee/faces/blob/309ddc0a060846d28395e963d69bdb080025b845/t[…]t/java/ee/jakarta/tck/faces/test/servlet50/ajax/Spec1567IT.java

Was this intentional? @werpu

werpu · 2023-02-03T17:01:34Z

No not intentional, this was a regression on my behalf, could have been an accidental copy paste issue. Have you found others?

volosied · 2023-02-03T17:34:32Z

I just performed a quick diff on the contents of the ajax and ajax_selenium folders for each of the tcks. I didn't see any other problems.

It looks good

werpu added 4 commits December 14, 2022 11:34

jakartaee#1722: Preparations for a pull request with ported selenium …

5f47427

…tests!

Merge remote-tracking branch 'origin/master' into selenium-tests

0f8520f

Merge remote-tracking branch 'origin/master' into selenium-tests

8b56baa

Merge remote-tracking branch 'origin/selenium-tests' into selenium-tests

f1d72a6

werpu changed the title ~~Selenium tests for the ajax part~~ Selenium tests for the ajax TCK (master branch) Dec 14, 2022

werpu changed the title ~~Selenium tests for the ajax TCK (master branch)~~ https://github.com/jakartaee/faces/pull/1732: Selenium tests for the ajax TCK (master branch) Dec 14, 2022

jakartaee#1722: fixing the version

1aaf880

(accidentally added debugging version)

werpu mentioned this pull request Dec 14, 2022

Feature/myfaces 4466 faces.js reimplementation (target 4.0.0 RC4) apache/myfaces#356

Closed

BalusC reviewed Dec 14, 2022

View reviewed changes

jakartaee#1722: reverting accidentally changed code back to the maste…

a76e0ae

…r version

werpu mentioned this pull request Jan 9, 2023

TCK Challenge: Drop usage of HTMLUnit for the Ajax Tests in favor of Selenium Webdrivers #1722

Closed

werpu added 3 commits January 9, 2023 12:04

Merge remote-tracking branch 'origin/master' into selenium-tests

38a24c5

jakartaee#1722: adding the Spec790 namespaced view test because it al…

813ad9d

…so relies on ajax!

jakartaee#1722: fixing the broken selenium non selenium switching

87ff0ac

smaller rename for the custom runners

melloware approved these changes Jan 9, 2023

View reviewed changes

arjantijms approved these changes Jan 12, 2023

View reviewed changes

dmatej approved these changes Jan 12, 2023

View reviewed changes

werpu mentioned this pull request Jan 13, 2023

faces.js: TCK Issue2162IT fails on Chrome (ri issue, not TCK) #1776

Open

jakartaee#1722: fixing the cdp version warning, updateing to the late…

4677bf9

…st selenium webdrivers

jakartaee#1722: removing possible race conditions and timeouts, overh…

3c13c76

…auling Spec1423 so that it uses less timing and more waiting for page conditions to avoid race conditions

BalusC reviewed Jan 17, 2023

View reviewed changes

BalusC approved these changes Jan 18, 2023

View reviewed changes

BalusC merged commit 309ddc0 into jakartaee:master Jan 18, 2023

pnicolucci added this to the 4.0.1-TCK milestone Feb 26, 2023

arjantijms modified the milestones: 4.0.1-TCK, 4.0.2-TCK Mar 10, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

https://github.com/jakartaee/faces/pull/1732: Selenium tests for the ajax TCK (master branch) #1770

https://github.com/jakartaee/faces/pull/1732: Selenium tests for the ajax TCK (master branch) #1770

werpu commented Dec 14, 2022 •

edited

Loading

werpu commented Dec 14, 2022

BalusC Dec 14, 2022

werpu Dec 14, 2022

werpu commented Jan 9, 2023 •

edited

Loading

werpu commented Jan 12, 2023

arjantijms left a comment

dmatej left a comment •

edited

Loading

werpu commented Jan 13, 2023 •

edited

Loading

dmatej commented Jan 13, 2023 •

edited

Loading

werpu commented Jan 13, 2023 •

edited

Loading

werpu commented Jan 13, 2023 •

edited

Loading

melloware commented Jan 13, 2023

werpu commented Jan 13, 2023 •

edited

Loading

werpu commented Jan 13, 2023

BalusC Jan 17, 2023

BalusC Jan 17, 2023

BalusC Jan 17, 2023

BalusC Jan 17, 2023

BalusC Jan 17, 2023

werpu Jan 18, 2023 •

edited

Loading

werpu commented Jan 17, 2023 •

edited

Loading

BalusC Jan 17, 2023

werpu Jan 17, 2023 •

edited

Loading

BalusC Jan 17, 2023

werpu Jan 17, 2023

volosied commented Jan 20, 2023

werpu commented Feb 3, 2023 •

edited

Loading

volosied commented Feb 3, 2023



		assertTrue(page.findElement(By.id("scriptResult")).getText().trim().equals("addedViaHead"));
		assertTrue(page.findElement(By.id("stylesheetResult")).getText().isEmpty());

https://github.com/jakartaee/faces/pull/1732: Selenium tests for the ajax TCK (master branch) #1770

https://github.com/jakartaee/faces/pull/1732: Selenium tests for the ajax TCK (master branch) #1770

Conversation

werpu commented Dec 14, 2022 • edited Loading

werpu commented Dec 14, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

werpu commented Jan 9, 2023 • edited Loading

werpu commented Jan 12, 2023

arjantijms left a comment

Choose a reason for hiding this comment

dmatej left a comment • edited Loading

Choose a reason for hiding this comment

werpu commented Jan 13, 2023 • edited Loading

dmatej commented Jan 13, 2023 • edited Loading

werpu commented Jan 13, 2023 • edited Loading

werpu commented Jan 13, 2023 • edited Loading

melloware commented Jan 13, 2023

werpu commented Jan 13, 2023 • edited Loading

werpu commented Jan 13, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

werpu Jan 18, 2023 • edited Loading

Choose a reason for hiding this comment

werpu commented Jan 17, 2023 • edited Loading

Choose a reason for hiding this comment

werpu Jan 17, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

volosied commented Jan 20, 2023

werpu commented Feb 3, 2023 • edited Loading

volosied commented Feb 3, 2023

werpu commented Dec 14, 2022 •

edited

Loading

werpu commented Jan 9, 2023 •

edited

Loading

dmatej left a comment •

edited

Loading

werpu commented Jan 13, 2023 •

edited

Loading

dmatej commented Jan 13, 2023 •

edited

Loading

werpu commented Jan 13, 2023 •

edited

Loading

werpu commented Jan 13, 2023 •

edited

Loading

werpu commented Jan 13, 2023 •

edited

Loading

werpu Jan 18, 2023 •

edited

Loading

werpu commented Jan 17, 2023 •

edited

Loading

werpu Jan 17, 2023 •

edited

Loading

werpu commented Feb 3, 2023 •

edited

Loading