Be able to run configuration testing on developer machines and as part of tests

[vmassol]

On XWiki we have the following use cases related to configuration testing:

• UC1: Fast to start XWiki in a given environment/configuration
• UC2: Solution must be usable both for running the functional tests and for distributing XWiki
• UC3: Be able to execute tests both on CI and locally on developer's machines
• UC4: Be able to debug functional tests easily locally
• UC5: Support the following configuration options (i.e we can test with variations of those and different versions): OS, Servlet container, Database, Clustering, Office Server, external SOLR, Browser
• UC6: Choose a solution that's as fast as possible for functional test executions

This means being able to run various configurations on developer's machines and to be able to debug from developer's IDEs, and start/stop the whole environment from the JUnit tests.

FTR this is what I'm currently developing using TestContainers (https://www.testcontainers.org/).

I'm currently documenting the experiments I've done and will be posting a link in this issue soon.

[danzone]

I'm interested in this topic and I'm looking forward to see your results! By the way, I didn't know testcontainers, so it would be interesting to check whether CAMP can provide some advantage compared to (something more than) testcontainers or if the two can be used together to manage test configurations more effectively.

[fchauvel]

Using v0.1.0, you should now be able to install camp on a developer machine, either using docker or using pip as for other Python application. Note that in the second case, you need to have the Z3 solver installed as well.

As for the generation of the configurations, you now can simply invoke camp generate -d my-workspace to get the configurations that you can run using Docker.

I close the issue for now on, but feel free to reopen, should you face any problem with that.

[vmassol]

I'd like to reopen this issue since the issue says "as part of the tests" and also I'm not sure it obeys UC1 through 6.

Also the idea is that this issue would be implemented with the TestContainers-based testing framework we implemented on XWiki so that CAMP could be run with 2 executors:

• Executor 1: standalone
• Executor 2: TC-based inside unit tests

WDYT?

PS: I'll also demonstrate this at the next STAMP meeting and we can decide what we do with it.