Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Setup client-side test infrastructure #2743

Open
ebruchez opened this issue Apr 20, 2016 · 30 comments
Open

Setup client-side test infrastructure #2743

ebruchez opened this issue Apr 20, 2016 · 30 comments

Comments

@ebruchez
Copy link
Collaborator

@ebruchez ebruchez commented Apr 20, 2016

Some tests are painful to do manually, in particular:

These tests are also hard to automate with Selenium due to the need to login/logout multiple times.

We should look into hooking up PhantomJS. It supports setting custom headers which would easily allow testing with multiple users using our header-based auth.

@ebruchez ebruchez self-assigned this Apr 20, 2016
@ebruchez

This comment has been minimized.

Copy link
Collaborator Author

@ebruchez ebruchez commented Apr 21, 2016

So PhantomJS has a third-party Selenium driver called https://github.com/detro/ghostdriver, but it seems to have maintenance issue, in particular with PhantomJS 2.0.

That project now also links to a newer driver, https://github.com/MachinePublishers/jBrowserDriver. So that's something to look at.

@ebruchez

This comment has been minimized.

Copy link
Collaborator Author

@ebruchez ebruchez commented Apr 21, 2016

It seems that jBrowserDriver is not a PhantomJS driver but its own thing. Not sure if that makes a difference.

@ebruchez

This comment has been minimized.

Copy link
Collaborator Author

@ebruchez ebruchez commented Apr 21, 2016

Interesting. jBrowserDriver uses WebKit included with Java 8 (part of JavaFX).

@ebruchez

This comment has been minimized.

Copy link
Collaborator Author

@ebruchez ebruchez commented Sep 15, 2016

It seems that jsdom might be the current hotness.

This would require running the tests from Node.js. Also, jsdom "does not perform layout or rendering, and it does not support navigation between pages". But this has benefits too: speed, no dependencies other than Node.js to install, and that still allows testing a lot of what we need to test.

@ebruchez ebruchez changed the title Evaluate PhantomJS for authentication tests Evaluate PhantomJS or other for client-sidetests Sep 20, 2016
@ebruchez

This comment has been minimized.

Copy link
Collaborator Author

@ebruchez ebruchez commented Sep 20, 2016

If we use jsdom or similar, tests will run within a JavaScript environment. It is not likely that we can use the JVM for that (even though there are talks here and here about using Nashorn but it's unclear how successful that is, and the Avatar.js project appears without activity for about 2 years - sad).

If we don't use the JVM, we will use Node.js to run the client-side tests, which means there will be a need to communicate between Node and the JVM. Launching Tomcat as we do now from ant is verbose and brittle.

So we should look at the following:

  1. Run tests with Node.js by using the built-in Scala.js support for tests.
  2. Launch an embedded version of Tomcat, using the Tomcat embedding API.
  3. Communicate between the two with HTTP, as is normally the case between client and server.
@ebruchez

This comment has been minimized.

Copy link
Collaborator Author

@ebruchez ebruchez commented Sep 20, 2016

More confirmation that Avatar.js is dead here and here.

@ebruchez ebruchez changed the title Evaluate PhantomJS or other for client-sidetests Evaluate jdsom or other for client-sidetests Sep 26, 2016
@ebruchez ebruchez changed the title Evaluate jdsom or other for client-sidetests Evaluate jdsom for client-sidetests Sep 26, 2016
@ebruchez ebruchez changed the title Evaluate jdsom for client-sidetests Evaluate jsdom for client-sidetests Sep 26, 2016
@ebruchez ebruchez changed the title Evaluate jsdom for client-sidetests Evaluate jsdom for client-side tests Sep 29, 2016
@ebruchez

This comment has been minimized.

Copy link
Collaborator Author

@ebruchez ebruchez commented Sep 29, 2016

Did some exploratory work with formBuilderClient in build.sbt:

  • jsEnv := JSDOMNodeJSEnv().value: enable jsdom
  • enablePlugins(ScalaJSJUnitPlugin): enable JUnit
  • remove dependency on runtimeDOM
  • jsDependencies += "org.webjars" % "jquery" % "1.12.0" / "1.12.0/jquery.js": add jQuery dependency
  • npm install jsdom
  • persistLauncher in Test := true otherwise getting an error
  • also need an Scala.js main class otherwise launcher is not produced and we get an error
@ebruchez

This comment has been minimized.

Copy link
Collaborator Author

@ebruchez ebruchez commented Sep 29, 2016

With the setup above, we can run JUnit tests written in Scala which run with Node.js and jsdom.

This is still very experimental. As a next step, we need to:

  • see how we can run this on travis (update .travis.yml to npm install stuff)
  • move to be an integration test (it)
    • 2017-08-17: had issues getting it:test to work with Scala.js
  • run dom tests with sbt (JVM only for now, later JS too)
  • run remaining tests with sbt using JUnit (JVM only)
  • start/stop embedded servlet container
  • consider using Node's child_process to run Docker commands
    • this would alleviate issues with Undertow.stop()
  • run at least one test connecting to servlet container
  • write/move useful tests
ebruchez added a commit that referenced this issue Sep 29, 2016
@ebruchez

This comment has been minimized.

Copy link
Collaborator Author

@ebruchez ebruchez commented Sep 29, 2016

Starting/stopping a server and running tests against them is what's covered in sbt by integration testing.

The Tests.Setup() and Tests.Cleanup() options are called by sbt before and after running tests, so maybe that's where we need to hookup starting our embedded server.

@ebruchez

This comment has been minimized.

Copy link
Collaborator Author

@ebruchez ebruchez commented Sep 29, 2016

We don't have to use Tomcat. We could use Jetty or even Undertow.

@ebruchez

This comment has been minimized.

Copy link
Collaborator Author

@ebruchez ebruchez commented Sep 29, 2016

Note that I didn't manage yet to get JUnit working with ScalaTest assertions as we do so far for our JVM tests written in Scala.

@ebruchez

This comment has been minimized.

Copy link
Collaborator Author

@ebruchez ebruchez commented Sep 30, 2016

Made ScalaUtilsTest run by sbt for both JVM and JS. Uses ScalaTest directly without JUnit.

@ebruchez

This comment has been minimized.

Copy link
Collaborator Author

@ebruchez ebruchez commented Aug 8, 2017

We already use Docker for database tests to make Oracle and other databases available to tests.

Instead of embedding Undertow (or other), an alternative would be to use a Tomcat Docker image. It is fairly easy to mount host directories and files into the Docker container.

Would this be better of worse?

Could this be worse for the build time because downloading the Docker image might be slower than getting the Undertow dependency via sbt (which can be cached)? Maybe, but that is not certain.

Using Undertow directly via a Java API is probably more flexible and doesn't require configuration files.

@ebruchez

This comment has been minimized.

Copy link
Collaborator Author

@ebruchez ebruchez commented Aug 9, 2017

One issue is how to run setup and cleanup.

sbt has Tests.Setup and Tests.Cleanup. Now we can hookup to that (verified). However the Scala code that runs within sbt does not have access to test code compiled with sbt. We would have to place it under project, I think, which is annoying because that code cannot leverage lots of utilities we need.

We could use the ClassLoader option of Tests.Setup(). This gives access to the ClassLoaderof the compiled test classes, and then with reflection we can run some setup/cleanup methods. But this only works if the tests do not run in forked mode, which we use.

@ebruchez

This comment has been minimized.

Copy link
Collaborator Author

@ebruchez ebruchez commented Aug 9, 2017

Maybe for now we just do this initialization within a ScalaTest test suite?

@ebruchez

This comment has been minimized.

Copy link
Collaborator Author

@ebruchez ebruchez commented Aug 9, 2017

Ah, but how will this work if we need to run Scala.js tests?

@ebruchez

This comment has been minimized.

Copy link
Collaborator Author

@ebruchez ebruchez commented Aug 12, 2017

Good progress on this for #1529:

  • can start/stop Docker instances from Scala.js
  • can start a Tomcat 8 container pointing to our orbeon-war with custom configuration files
  • can load a simple XForms and it initializes within JSDOM
@ebruchez

This comment has been minimized.

Copy link
Collaborator Author

@ebruchez ebruchez commented Aug 17, 2017

Following discussion with @avernet, it might be simplier to just run external processes directly from Node rather than using the Undertow remote control. That would make things simpler and avoid the hang I am getting with stopping that server at times.

See child_process.

@ebruchez

This comment has been minimized.

Copy link
Collaborator Author

@ebruchez ebruchez commented Aug 18, 2017

Now one issue is that when we launch commands, we need to know the project's base directory. We need to check whether we obtain the right thing with Node. With the current sbt-based start/stop, we simply pass the values we get from sbt, so we are sure to have the right ones.

@ebruchez

This comment has been minimized.

Copy link
Collaborator Author

@ebruchez ebruchez commented Aug 18, 2017

If we find a way, we could also pass parameters to the test from sbt. Could ConfigMap help?

@ebruchez

This comment has been minimized.

Copy link
Collaborator Author

@ebruchez ebruchez commented Aug 23, 2017

Cannot find a way with ConfigMap so far. So working around this with BuildInfo. This way we can still pass values obtained from sbt to the test.

Using Node to deal with docker works just fine. So we can remove our Undertow server.

One issue is that ScalaTest hangs after completing tests, with this:

> last orbeonWarJS/test:testOnly
[debug] Running TaskDef(org.orbeon.fr.OrbeonClientTest, org.scalajs.testadapter.FingerprintSerializers$DeserializedSubclassFingerprint@7f40b38, false, [SuiteSelector])
java.util.concurrent.TimeoutException: Futures timed out after [59988191119 nanoseconds]
	at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
	at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
	at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107)
	at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
	at scala.concurrent.Await$.result(package.scala:107)
	at org.scalajs.jsenv.AsyncJSRunner$class.await(AsyncJSRunner.scala:59)
	at org.scalajs.jsenv.ExternalJSEnv$AsyncExtRunner.await(ExternalJSEnv.scala:172)
	at org.scalajs.jsenv.AsyncJSRunner$class.awaitOrStop(AsyncJSRunner.scala:76)
	at org.scalajs.jsenv.ExternalJSEnv$AsyncExtRunner.awaitOrStop(ExternalJSEnv.scala:172)
	at org.scalajs.testadapter.ScalaJSRunner$$anonfun$6$$anonfun$apply$1.apply$mcV$sp(ScalaJSRunner.scala:96)
	at org.scalajs.testadapter.ScalaJSRunner$$anonfun$6$$anonfun$apply$1.apply(ScalaJSRunner.scala:96)
	at org.scalajs.testadapter.ScalaJSRunner$$anonfun$6$$anonfun$apply$1.apply(ScalaJSRunner.scala:96)
	at scala.util.Try$.apply(Try.scala:161)
	at org.scalajs.testadapter.ScalaJSRunner$$anonfun$6.apply(ScalaJSRunner.scala:96)
	at org.scalajs.testadapter.ScalaJSRunner$$anonfun$6.apply(ScalaJSRunner.scala:96)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
	at scala.collection.immutable.List.foreach(List.scala:318)
	at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
	at scala.collection.AbstractTraversable.map(Traversable.scala:105)
	at org.scalajs.testadapter.ScalaJSRunner.done(ScalaJSRunner.scala:96)
	at sbt.Defaults$$anonfun$allTestGroupsTask$1$$anonfun$16.apply(Defaults.scala:590)
	at sbt.Defaults$$anonfun$allTestGroupsTask$1$$anonfun$16.apply(Defaults.scala:588)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
	at scala.collection.immutable.Map$Map1.foreach(Map.scala:109)
	at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
	at scala.collection.AbstractTraversable.map(Traversable.scala:105)
	at sbt.Defaults$$anonfun$allTestGroupsTask$1.apply(Defaults.scala:588)
	at sbt.Defaults$$anonfun$allTestGroupsTask$1.apply(Defaults.scala:586)
	at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
	at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:40)
	at sbt.std.Transform$$anon$4.work(System.scala:63)
	at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228)
	at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228)
	at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
	at sbt.Execute.work(Execute.scala:237)
	at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)
	at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)
	at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159)
	at sbt.CompletionService$$anon$2.call(CompletionService.scala:28)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

I thought it was due to Undertow hanging, but I have the same exact result with using Node. So it doesn't seem to be related to that. I then thought it could be due to using RösHTTP, which also depends on Monix, but removing that and using plain JSDOM to fetch pages still hangs in the end.

@ebruchez

This comment has been minimized.

Copy link
Collaborator Author

@ebruchez ebruchez commented Aug 24, 2017

The timeout was due to not calling JSDOM window.close().

@ebruchez ebruchez removed this from To Review in Orbeon Forms 2017.2 Dec 1, 2017
@ebruchez ebruchez added this to To Review in Orbeon Forms 2018.1 Dec 1, 2017
@ebruchez ebruchez moved this from To Review to To Do in Orbeon Forms 2018.1 Jan 3, 2018
@ebruchez

This comment has been minimized.

Copy link
Collaborator Author

@ebruchez ebruchez commented Jan 8, 2018

For reference, the integration tests we have now are on ignore() as they don't run with TravisCI. Next steps:

  • figure out why they don't work on TravisCI (use Travis debug mode?)
@ebruchez ebruchez removed this from To Do in Orbeon Forms 2018.1 May 11, 2018
@ebruchez

This comment has been minimized.

Copy link
Collaborator Author

@ebruchez ebruchez commented Feb 11, 2019

@ebruchez

This comment has been minimized.

Copy link
Collaborator Author

@ebruchez ebruchez commented Feb 13, 2019

  • Puppeteer
    • says "Speed: Puppeteer has almost zero performance overhead over an automated page."
    • You install it via npm and we should be able to run it from a Scala.js ScalaTest test.
  • Headless Chrome is supposed to work on Travis.

So the missing part is starting a server. We do have a couple of options:

  • Docker
  • Tomcat/Undertow/other servlet container
@ebruchez

This comment has been minimized.

Copy link
Collaborator Author

@ebruchez ebruchez commented Oct 25, 2019

@avernet

This comment has been minimized.

Copy link
Collaborator

@avernet avernet commented Oct 25, 2019

Hopefully this has changed in the last 2 years.

@ebruchez

This comment has been minimized.

Copy link
Collaborator Author

@ebruchez ebruchez commented Oct 25, 2019

We probably would need to upgrade our Travis setup from the old Trusty to Xenial or Bionic.

@ebruchez

This comment has been minimized.

Copy link
Collaborator Author

@ebruchez ebruchez commented Oct 25, 2019

Hopefully this has changed in the last 2 years.

Doesn't seem like it: you must either pass a JavaScript function or a string containing some JavaScript expression (or function?). Not sure how much this is needed but hopefully we can work around this.

@ebruchez ebruchez added this to To review in Orbeon Forms 2020.1 via automation Jan 9, 2020
@ebruchez ebruchez changed the title Evaluate jsdom for client-side tests Setup client-side test infrastructure Jan 9, 2020
@ebruchez

This comment has been minimized.

Copy link
Collaborator Author

@ebruchez ebruchez commented Jan 9, 2020

What are the next steps for this? For the replication tests, we already have code which:

  • starts and stop Docker containers running Orbeon Forms
  • connects to them via JSDOM
  • run code within JSDOM windows

Thoughts:

  • For simple tests for which JSDOM is enough, we could set up a very similar infrastructure.
  • For more complex tests, using Puppeteer is probably the way to go, but that's a little more investigation.

Steps:

  • decide on first simple test(s)
  • implement JSDOM infrastructure
  • write first test
  • decide on more complex test benefitting from Puppeteer
  • implement Puppeteer infrastructure
  • write first test
@ebruchez ebruchez moved this from To review to To do in Orbeon Forms 2020.1 Jan 10, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Orbeon Forms 2019.1
  
Awaiting triage
2 participants
You can’t perform that action at this time.