Running the integration tests does not work as described

[RKrahl]

I just tried to run the integration tests. My experiences so far:

First try. I followed the instructions from the icat.server wiki. I set up a test server having authn.db-2.0.0, icat.lucene-1.1.0, and icat.server-4.9.1 deployed. I tried to run mvn install in the current source tree. After compile, the deploying of icat.server-4.9.2-SNAPSHOT.war failed with an Initialization failed for Singleton PropertyHandler exception caused by java.lang.IllegalStateException: class org.icatproject.core.IcatException No authenticator of type db is working.

Investigating the issue, it looks like the settings from src/main/resources/run.properties are hard coded to be used in the integration tests. It is pretty obvious that authn.db cannot be reached by the URL https://smfisher:8181 in my environment. As opposed to what is stated in the instructions in the wiki, the serverUrl in ~/.m2/settings.xml seem to be ignored.

Second try. I edited src/main/resources/run.properties to hard code settings that would work in my environment. This time, deploying of icat.server-4.9.2-SNAPSHOT.war failed because the server already has a web module icat.server-4.9.1 loaded at /icat. So the force-deploying was not able to overwrite the already installed older icat.server.

Third try. I uninstalled icat.server-4.9.1 and tried again. Now, the deploying failed with a java.lang.RuntimeException: Invalid resource : jdbc/icat__pm. Apparently, the integration test did not try to create the JDBC connection pool to connect to the database. In fact, I did not found any configuration settings in the sources that one useually would put into the setup.properties to point to the database.

Fourth try. I reinstalled icat.server-4.9.1 and then undeployed it manually. This created the JDBC connection pool as a side effect and left it behind, because I didn't use the setup script to uninstall. Now this did the trick. I got two failures, one errors, and three skipped out of 66 tests, but this is another issue to be investigated separately. But at least the test suite did run at all.

Unless I am overlooking something fundamental, this current state is not suitable to encourage anybody to run the tests.

[RKrahl]

Notes on the test failures: the cause of the error in the test results was that one test tried to login as user CIC with password password. This account was not listed in wiki page and therefore did not exist in my test server.

For the first failing test, I found the cause and suggest a fix in #206.

The second test failure is dubious. The test exportMetaDataQuery in src/test/java/org/icatproject/integration/TestRS.java creates an export file and asserts the size of this file to be 1924 bytes. When I run the test, I get an export file having 1954 bytes. But this export file looks perfectly valid. I suspect the export file format to be not very static. There are some degrees of freedom in the export file format and this might also depend on environment (e.g. date and time format). But I don't have a sample of the expected, supposed to be correct output, so I can't compare. I guess, the size of the export file is simply not a very good criterion for the validity.

[louise-davies]

I believe that the majority of the issues raised in this issue have been solved. I updated the icat.server wiki with instructions that work based on our Vagrant installation tutorial, of which the main points are below:

• Ensure that src/main/resources/run.properties has the correct paths according to your set up. The default file has been updated to have https://localhost.localdomain:8181 instead of https://smfisher:8181 as the base URL, which is the setup required by the Vagrant tutorial
• Ensure that authn.db is installed and has the correct test accounts added
• Run asadmin undeploy on any previously installed version of icat.server

In addition to this, the integration tests are now being ran successfully on TravisCI, which provides an actual example of how to set up the environment to run the tests, using the icat-ansible Ansible script and the .travis.yml file in this repo.

Another issue I fixed was that the tests were not correctly setting up data before running and clearing data after running, which caused problems in that tests depended on other tests being ran first. To fix this, I added @Before and @After functions to the test suites that ensures that data is set up before each test is ran and cleared afterwards, and removed any code that manually did this from the tests themselves. The relevant commit is f1b1f1c

The only leftover point that I can see is that we rely on the JDBC connection pool left behind by a previous installation of icat.server rather than creating the connection pool in the process of running the tests. I'm not certain how difficult this would be to implement, but for now relying on the old connection pool works.

[RKrahl]

I acknowledge big progress has been made on this. I still see two issues though:

• src/main/resources/run.properties is under version control. If the user still needs to make changes to this configuration file in order to run the tests locally, the tests cannot be run in a clean working directory.
• the JDBC connection pool not being setup properly.

I'd suggest to do the following (similar to ids.server) to solve both at the same time:

• dedicate an install directory for the tests. It should be empty save for a suitable .gitignore in the git repo. (ids.server uses src/tests/install, one may argue that a path below src may not be appropriate though.)
• add a script to prepare the tests. It should copy all relevant files for installing icat.server for the tests from the just compiled target to the install directory, similar to an unzipped distro. It should create suitable setup.properties, run.properties, and logback.xml files if not present, but should not overwriting existing files. Finally it should run ./setup install in this directory. See src/test/scripts/prepare_test.py in ids.server for a model.
• the pre-integration-test phase should run this script rather then doing a simple asadmin deploy.