Integration tests independent from OC, with auto-install

[PVince81]

Wouldn't it be nice if the current integration tests that are stored in core's "build/integration" could be separated and made to run on any OC instance ?

Or at least we could add support for sentences like "OC 8.2.8 is installed".
This would make it possible to cover known bugs that only appear after upgrading.
For example (dummy):

Scenario: create share in OC 8.2, update in OC 9.0 and run transfer ownership
    When ownCloud v8.2.8 is installed
    And user "user0" creates a share "test" with permissions 23
    And ownCloud is updated to v9.0.5
    Then when transferring ownership of "user0" to "user1"
    And the command is successful
    ...

(there was a case where this would fail)

@owncloud/qa @DeepDiver1975

[SergioBertolinSG]

Well I see some problems with this approach.

After a test including update, the remaining tests would run against newer version.

How would the server be updated? git? tarball?

Tests including update increases execution time perhaps too much for each PR.

Wouldn't it be nice if the current integration tests that are stored in core's "build/integration" could be separated and made to run on any OC instance ?

This would be nice yes. Since it is not configured that way at the moment, it would require changes in the way run.sh works. perhaps making some kind of switch between running on current code or against another server. I'll expect problems with the federated tests part. 🔮

[SergioBertolinSG]

@DeepDiver1975 What do you think about having tests independent able to run against any server?

That would require changes in jenkins.

Also tests in <=8.2 branches are a bit different to >=9.0. In case of going this way I recommend to support only >=9.0, where almost all tests from master would pass and it would require less tags and modifications.

[SergioBertolinSG]

Some thoughts about this:

• It requires code in a different repo?
• These tests require access to core files and occ, so testing "any server" won't be possible.
• It would clone tests repo and core repo, in the same machine, install oC using some predefined config file.
• Running tests would need a heavy use of tags, having tests tagged and choose which ones need to pass.
• Perhaps having different jobs in jenkins configured to run specific tags.
• The update option would require a way to reboot the oC set up after ending the test. Otherwise only 1 test would run.

[DeepDiver1975]

Hmmm .... generally speaking: it is quite complex to setup tests like this. I'm not sure on how sure tests are helpful through out the regular development and testing.

I mean: it will be hard to detect such issues upfront - tests are not helping. In addition it is unlikly to break this test the same way in the future.

[PVince81]

I think we discussed that we wanted this last time in Nbg.

Goal is to be able to combine different docker setups with the docker containing the tests.

Docker 1 contains integration tests.
Docker 2 contains an ownCloud instance to be tested, preconfigured with whatever settings the test matrix require (ex: object store, php version, etc)

The tests are run in Docker 1 and connect to Docker 2 through APIs to perform the tests.

[butonic]

Hm would be great to have a testsuite we can give to customers so they can run it in their testing environment after upgrading

[patrickjahns]

some are already available as docker container
Litmus: https://hub.docker.com/r/owncloud/litmus/
Smashbox: https://hub.docker.com/r/owncloud/smashbox/

There needs to be a lot of changes in the test suite / their assumptions:

• right now the test suites require certain users to be created/deleted (what happens if a user with that name exists on the server?)
• the test suites adjust/change settings to test certain scenarios -> will the customer block normal operations for the test suite to do its thing? Otherwise test suites might interfere with the everyday operations
• some tests require admin privileges to run
• test suites require testingapp to be present for setup / teardown of scenarios (which can be considered a security vulnerability)

It might be possible to tag test suites, that are "harmless" and be able to provide them via a docker container to customers. (cc @individual-it @phil-davis ) - but I doubt the usefulness.

[phil-davis]

so they can run it in their testing environment after upgrading

as long as the customer really has a testing environment that they do not care so much about, then they can run tests that add/delete users and change system-wide settings

We can easily tell customers the user and group names that the tests use, and the test suite can also check for those and if any already exist, it just refuses to start.

The capabilities setup (global sharing options etc) tries hard to put things back the way they were on exit. So that "should" be OK. (but of course will make trouble during the test run if used on a live production system!)

We can abstract the admin password somewhere even more, so it can be passed in by the person running the test script, for places that do not use just admin/admin on their test system.

The tests could first download (clone from GutHub) the testing app (if it is not already there), then enable it... then (optionally) remove it at the end.

In the end, this sort of thing really needs to be done on a testing system (which can be a backup of a production system, restored to some test server)