Updates and refactoring of arquillian testsuite.

pom.xml

@@ -1332,7 +1332,6 @@
             <id>distribution</id>
             <modules>
                 <module>distribution</module>
-                <module>testsuite/integration-arquillian</module>
             </modules>
         </profile>
 
@@ -1341,7 +1340,6 @@
             <modules>
                 <module>docbook</module>
                 <module>distribution</module>
-                <module>testsuite/integration-arquillian</module>
             </modules>
         </profile>
 
@@ -1364,6 +1362,14 @@
             </build>
         </profile>
 
+        <profile>
+            <id>arquillian-integration-tests</id>
+            <modules>
+                <module>distribution</module>
+                <module>testsuite/integration-arquillian</module>
+            </modules>
+        </profile>
+
         <!-- Configure the JBoss Early Access Maven repository -->
         <profile>
             <id>jboss-earlyaccess-repository</id>

testsuite/integration-arquillian/README.md

@@ -1,21 +1,61 @@
-Testing admin console with Arquillian
-=====================================
+# Keycloak Integration Testsuite with Arquillian
 
-There are currently two ways of running the tests with help of Arquillian.
+## Structure
 
-Remote mode
-----------
+```
+integration-arquillian
+│
+├──servers  (submodules enabled via profiles)
+│  ├──wildfly
+│  └──eap6
+│
+└──tests
+   ├──base
+   └──adapters  (submodules enabled via profiles, all depend on base)
+      ├──wildfly
+      ├──wildfly-relative  (needs servers/wildfly)
+      ├──wildfly8
+      ├──as7
+      ├──tomcat
+      └──karaf
 
-Just simply typle `mvn verify` and you are all set. This requires the instance of Wildfly with embedded Keycloak to be already running.
+```
 
-Managed mode
-------------
+## General Concepts
 
-You need to pass two arguments to Maven, first is location of your Wildfly server with embedded Keycloak and the other is name of the profile.
+The testsuite supports **multiple server runtimes** for the Keycloak server.
+The **default is Undertow** which is the fastest and easiest option, and runs in the same JVM as the tests.
 
-    mvn verify -Pwildfly-8-managed -DjbossHome=/your/server/location
+Other options are **Wildfly 9** and **EAP 6**. These have some additional requirements and limitations:
+1. The selected server module must be built before any tests can be run. 
+All server-side configuration is done during this build (e.g. datasource configuration).
+Once server artifact is built the tests modules can unpack it via `maven-dependency-plugin` into their working directory before running.
+2. Before the selected server module can be built the `keycloak/distribution` module also needs to be built.
 
-Browser
--------
+### Server Runtimes
 
-There are currently two supported browsers - PhantomJS and Firefox. PhantomJS is the default one, in order to use Firefox just specify `-Dbrowser=firefox` parameter in the Maven command. 
+TODO: explain why separate module, list config options, note on migration modules
+
+### Base Testsuite
+
+login flows + account management
+
+admin ui
+
+abstract adapter tests
+
+### Adapter Tests
+
+test servlets: demo, session
+
+examples
+
+## Running the Tests
+
+### Undertow
+
+### Wildfly or EAP 6
+
+### Adapters
+
+### Supported Browsers

testsuite/integration-arquillian/README_old.md

@@ -0,0 +1,189 @@
+# Keycloak Integration Testsuite with Arquillian
+
+*OUT OF DATE - NEEDS REWRITE*
+
+## Usage
+
+Running the tests: `mvn test` or `mvn clean test`
+
+## Test suite
+
+### Selecting container for Keycloak Server
+
+The testsuite requires a container for Keycloak Server to be selected.
+This container is used by all tests in the suite during a single test execution.
+
+*By default* the tests run with server on embedded *Undertow*.
+A different container can be selected with profile, e.g. `-Pauth-server-wildfly`.
+
+### Containers Supported for Keycloak Server
+
+| Container | Arquillian Qualifier | Maven | Dependencies |
+| --- | --- | --- | --- |
+| **Undertow** | `auth-server-undertow` | **default** | `undertow-core`, `resteasy-undertow` |
+| **Wildfly 9** | `auth-server-wildfly` | `-Pauth-server-wildfly` | `keycloak-server-dist` or `wildfly-dist`+`keycloak-server-overlay` |
+| **EAP 6.4** | `auth-server-eap6` | `-Pauth-server-eap6` | `keycloak-server-dist` or `eap6-dist`+`keycloak-server-overlay` |
+
+See the relevant container definitions in `arquillian.xml` located in the **test resources** folder.
+
+### Test Class Hierarchy
+```
+AbstractKeycloakTest
+├── AbstractAdminConsoleTest
+└── AbstractAdapterTest
+```
+
+### AbstractKeycloakTest
+
+Handles test realms. Provides Admin Client for REST operations.
+
+* **@BeforeClass**
+ 1. Updates the admin password to enable the admin user.
+* **@Before**
+ 1. Initiates admin client
+ 2. Imports test realms. (Loading test realms is overriden in subclasses.)
+* **@After**
+ 1. Removes test realms.
+ 2. Closes admin client.
+
+### ContainersTestEnricher
+
+Manages *container lifecycles*.
+
+`ContainersTestEnricher` is a custom Arquillian observer that handles lifecycles of auth server and app server containers for each test class.
+Containers are started during `@BeforeClass` and shut down during `@AfterClass` event.
+
+*Optionally* each test class can be annotated with `@AuthServerContainer("qualifier")` and `@AppServerConatiner("qualifier")` annotations 
+to indicate containers required for the test.
+
+* In case `@AuthServerContainer` is not present the *auth server qualifier* is loaded from `auth.server.container` property.
+* In case `@AppServerContainer` is not present or it's value is the same as *auth server qualifier*, the app server isn't started for the test class.
+
+## Admin Console Tests
+
+Tests for admin console are located in `org/keycloak/testsuite/console`.
+Related non-test classes are located on the same path in the **main sources**.
+
+Admin console tests are **ENABLED by default**. They can be disabled by `-P no-console`.
+
+
+## Adapter Tests
+
+Adapter tests are located in `org/keycloak/testsuite/adapter`.
+Related non-test classes can be found on the same path in the **main sources**.
+
+Adapter tests are **DISABLED by default**. They can be enabled by profiles.
+Multiple profiles can be enabled for a single test execution.
+
+*Note:* When testing adapter with multiple containers in a single run it is better 
+to use the `--fail-at-end` (`-fae`) strategy instead of the default `--fail-fast` one.
+This will allow Maven to continue building other modules even if some of them have test failures.
+
+### Containers Supported for Adapter Tests
+
+| Container | Arquillian Qualifier | Maven | Dependencies |
+| --- | --- | --- | --- |
+| **Wildfly 9** Relative | `auth-server-wildfly` | `-Pauth-server-wildfly` | `keycloak-server-dist` or `wildfly-dist`+`keycloak-server-overlay`, `keycloak-adapter-dist-wf9` |
+| **Wildfly 9** | `app-server-wildfly` | `-Papp-server-wildfly` | `wildfly-dist`, `keycloak-adapter-dist-wf9` |
+| **Wildfly 8** | `app-server-wildfly` | `-Papp-server-wildfly8` | `wildfly-dist:8.2.1.Final`, `keycloak-adapter-dist-wf8` |
+| **JBoss AS 7** | `app-server-as7` | `-Papp-server-as7` | `jboss-as-dist`, `keycloak-adapter-dist-as7` |
+| **Tomcat 8** | `app-server-tomcat` | `-Papp-server-tomcat` | `tomcat`, `keycloak-tomcat8-adapter-dist` |
+| **Karaf 3** | `app-server-karaf` | `-Papp-server-karaf` | `apache-camel`, `apache-cxf`, `keycloak-osgi-features`, `keycloak-fuse-example-features` |
+
+See the relevant container definitions in `tests/adapter/<container>/src/main/xslt/arquillian.xsl`.
+
+***Important:*** Arquillian cannot load multiple controllers for JBossAS/Wildfly containers in a single run (because same class name)
+but a different controller is required for JBossAS7/EAP6 than for WF8/9. Because of this:
+
+ - Adapter tests for *Wildfly 8/9* cannot be run against server on *EAP 6*. `-Papp-server-wildfly*` ⇒ `!auth-server-eap6`
+ - Adapter tests for *JBossAS 7* can only be run against server on *EAP 6*. `-Papp-server-as7,auth-server-eap6`
+
+### Adapter Test Types
+
+1. Using **test servlets**.
+2. Using **example/demo wars**.
+
+```
+AbstractKeycloakTest
+└── AbstractAdapterTest
+    ├── AbstractServletsAdapterTest
+    |   ├── Relative…
+    |   ├── Wildfly…
+    |   ├── Tomcat…
+    |   …
+    └── AbstractExampleAdapterTest
+        ├── AbstractDemoExampleAdapterTest
+        |   ├── Relative…
+        |   ├── Wildfly…
+        |   ├── Tomcat…
+        |   …
+        ├── AbstractBasicAuthExampleAdapterTest
+        |   ├── Relative…
+        |   ├── Wildfly…
+        |   ├── Tomcat…
+        |   …
+        …
+```
+
+### Relative vs Non-relative scenario
+
+The test suite can handle both types.
+It automatically modifies imported test realms and deployments' adapter configs based on scenario type.
+
+| Scenario | Description | Realm config (server side) | Adapter config (client side) |
+| --- | --- | --- | --- |
+| **Relative** | Both Keycloak Server and test apps running in the same container. | client `baseUrl`, `adminUrl` and `redirect-uris` can be relative | `auth-server-url` can be relative |
+| **Non-relative** | Test apps run in a different container than Keycloak Server. | client `baseUrl`, `adminUrl` and `redirect-uris` need to include FQDN of the app server | `auth-server-url` needs to include FQDN of the auth server|
+
+### Adapter Libraries Mode
+
+1. **Provided.** By container, e.g. as a subsystem. *Default.*
+2. **Bundled.** In the deployed war in `/WEB-INF/libs`. Enable with `-Dadapter.libs.bundled`. *Wildfly only*.
+
+### Adapter Config Mode
+
+1. ~~**Provided.** In `standalone.xml` using `secure-deployment`. *Wildfly only.*~~ WIP
+2. **Bundled.** In the deployed war in `/WEB-INF/keycloak.json`. *Default.*
+
+### Adapters Test Coverage
+
+| Module | Coverage | Supported Containers |
+| --- | --- | --- |
+| ***Test Servlets*** | Good | All |
+| **Demo** | Minimal, WIP | `auth-server-wildfly` (relative) |
+| **Admin Client** |  |
+| **Cordova** |  |
+| **CORS** |  |
+| **JS Console** | Good | `auth-server-wildfly` (relative) |
+| **Providers** |  |
+| Themes |  |
+| Multitenancy | WIP |  |
+| **Basic Auth** | Good | All |
+| **Fuse** | Good | `app-server-karaf` |
+| SAML |  |
+| LDAP |  |
+| Kerberos |  |
+
+## Supported Browsers
+
+| Browser | Maven |
+| --- | --- | 
+| **PhantomJS** | `-Dbrowser=phantomjs` **default** |
+| **Firefox** | `-Dbrowser=firefox` |
+
+
+## Custom Arquillian Extensions
+
+Custom extensions are registered in `META-INF/services/org.jboss.arquillian.core.spi.LoadableExtension`.
+
+* Multiple containers extension
+ * Replaces Arquillian's default container handling.
+ * Allows to manage multiple container instances of different types within a single test run.
+ * Allows to skip loading disabled containers based on `enabled` config property in `arquillian.xml`.
+* Custom extension
+ * `ContainersTestEnricher` - Handles lifecycles of auth-server and app-server.
+ * `CustomUndertowContainer` - A custom container controller for JAX-RS-enabled Undertow with Keycloak Server.
+ * `DeploymentArchiveProcessor` - Modifies adapter config before deployment on app server based on relative/non-relative scenario.
+ * `URLProvider` - Fixes URLs injected by Arquillian which contain 127.0.0.1 instead of localhost.
+ * `JiraTestExecutionDecider` - Skipping tests for unresolved JIRAs.
+