Tests: re-work docker-compose usage. #6212 (#6344)

* Tests: merge all docker compose files into one. #6212

Use profiles to activate additional features when needed. For example:
```
docker-compose --file etc/docker/dev/docker-compose.yml --profile storage --profile monitoring up
```
instead of
```
docker-compose --file etc/docker/dev/docker-compose-storage-monit.yml up
```

Hope this will reduce the amount of un-maintained and duplicated stuff
in these files

* Tests: don't open docker-compose ports by default. #6212

Opening ports is not a requirement. It's a comfort feature which some
developers use to access services directly from their machine. But
this is also opening the door for security-related problems.

This commit stops opening ports by default. If somebody needs this
feature, they can call docker-compose with two --file arguments:
``--file docker-compose.yaml --file docker-compose.ports.yml` to merge
the both files together and achieve the same behavior.

* Tests: directly install gridsite in alma9

it's now available in epel.

* Tests: use docker-compose in autotests

Remove the home-made suites.py which was manually starting the same
docker containers, but with a slightly different configuration.
The difference in configuration was not justified in any way.

The relevant autotests will now use docker-compose to run
database/elastic/activemq/influx. The same way it's done for integration
tests. An override docker-compose file is used to set the container
image and to disable the default 'ruciodb' postgres from running.

This commit doesn't change the interface exposed by run_tests.py. In
theory, donkeyrider should still run correctly for all those 1 person
who actually use it, but will have a dependency on docker-compose,
which wasn't the case until now.

Rollback the oracle container version in docker-compose. Put there
the version which was actually used in suites.py (18).

* Tests: fix typo in matrix

* Tests: merge run_tests.sh and run_tests_docker.sh

Why do we even have two of them ?

* Tests: factorize common rucio.cfg

This also has the benefit of testing the tools/merge_rucio_configs.py
script, which is critical for our containers, but is not tested
anywhere.

* Tests: use sqlalchemy to wait for database readiness

It's much more generic this way. This will probably allow to
completely get rid of the before_script.sh later.

* Tests: merge before_script.sh into install_script.sh

The content is now small enough that it doesn't justify having a
separate script. The only tricky part was configuration of the
oracle database. Luckily, the container which we use already supports
mounting a shell script to be executed on initialization. It's enough
to mount it at the correct location.

* Tests: merge install_script.sh into test.sh

These two scripts are always called one after the other. Make it
explicit. Especially considering that the second heavily depends
on the first.

Mostly just code movement, no refactoring (which would be very welcomed
in a later commit).

* Tests: remove with_venv.sh file

It's not needed for rucio operation or tests. If somebody uses this
simple wrapper, they can keep a local version of it.

* Tests: remove purge_bin.py

It was used for housekeeping when testing on the CERN oracle devdb
instance, but the recommended way to run tests against oracle now
is using docker-compose.

* Tests: remove submit_fts3_transfer.py

This script is broken for a long time now

* Tests: generate rucio.cfg on container startup

This will allow to easily create a test env for a specific database:
```
RDBMS=mysql8 docker-compose --file etc/docker/dev/docker-compose.yml --profile mysql8 up
```

Mount the rucio source into the rucio containers and use an alternative
entry-point script to generate the desired configuration before running
httpd.

Also copy the "default" rucio.cfg and alembic.ini from rucio/containers
repository. This will reduce the dependency between the rucio/rucio and
rucio/containers repositories.

Move the httpd restart command in run_tests.sh to fix the sqlite env.
Otherwise, httpd continues to use the old database even if we re-created
it a couple of lines above.

* Tests: reorganize test.sh by suite

Make it very clear what code is executed for what suite. Avoid any
confusion. Except for the multi_vo tests, where a little rework was
needed, everything else is mostly just code movement to put everything
under a unique if/else block.

Remove rucio_syntax.cfg. It's identical to the sqlite one.

* Tests: integrate iam docker-compose into the main one

Run the additional containers starting docker-compose with
`--profile iam`

.github/workflows/integration_tests.yml

@@ -98,7 +98,7 @@ jobs:
         shell: bash
         run: |
           docker login https://ghcr.io -u ${{ github.actor }} -p ${{ secrets.GITHUB_TOKEN }}
-          docker-compose -f $GITHUB_WORKSPACE/dev/rucio/etc/docker/dev/docker-compose-storage-externalmetadata.yml pull
+          docker-compose -f $GITHUB_WORKSPACE/dev/rucio/etc/docker/dev/docker-compose.yml --profile storage --profile externalmetadata pull
           i=0; until [ "$i" -ge 3 ]; do
             IMAGES=$(echo '${{ toJson(matrix.cfg) }}' | $GITHUB_WORKSPACE/dev/rucio/tools/test/build_images.py --output list \
                 --cache-repo ghcr.io/${{ github.repository }} --branch "${{ needs.setup.outputs.branch }}" \
@@ -117,14 +117,14 @@ jobs:
         run: |
           docker image ls
           sed -i 's;image: docker.io/rucio/rucio-dev.*;image: ${{ fromJSON(steps.images.outputs.images)[0] }};' \
-              $GITHUB_WORKSPACE/dev/rucio/etc/docker/dev/docker-compose-storage-externalmetadata.yml
+              $GITHUB_WORKSPACE/dev/rucio/etc/docker/dev/docker-compose.yml
       - name: Start containers
-        run: docker-compose -f $GITHUB_WORKSPACE/dev/rucio/etc/docker/dev/docker-compose-storage-externalmetadata.yml up -d
+        run: docker-compose -f $GITHUB_WORKSPACE/dev/rucio/etc/docker/dev/docker-compose.yml --profile storage --profile externalmetadata up -d
       - name: Initialize tests
         shell: bash
         run: |
           docker exec -t dev_rucio_1 cp etc/rse-accounts.cfg.template etc/rse-accounts.cfg
-          docker exec -t dev_rucio_1 tools/run_tests_docker.sh -ir
+          docker exec -t dev_rucio_1 tools/run_tests.sh -ir
       - name: File Upload/Download Test
         run: docker exec -t dev_rucio_1 tools/pytest.sh -v --tb=short tests/test_rucio_server.py
       - name: UploadClient Test
@@ -173,4 +173,4 @@ jobs:
       - name: Test external metadata plugin (postgres)
         run: docker exec -t dev_rucio_1 tools/pytest.sh -v --tb=short tests/test_did_meta_plugins.py::TestDidMetaExternalPostgresJSON
       - name: Stop containers
-        run: docker-compose -f $GITHUB_WORKSPACE/dev/rucio/etc/docker/dev/docker-compose-storage-externalmetadata.yml down
+        run: docker-compose -f $GITHUB_WORKSPACE/dev/rucio/etc/docker/dev/docker-compose.yml --profile storage --profile externalmetadata down

etc/docker/dev/README.rst

@@ -49,11 +49,11 @@ Finally, you can jump into the container with::
 
 To verify that everything is in order, you can now either run the full unit tests or only set up the database. Running the full testing suite takes ~10 minutes::
 
-    tools/run_tests_docker.sh
+    tools/run_tests.sh
 
 Alternatively, you can bootstrap the test environment once with the `-i` option and then selectively or repeatedly run test case modules, test case groups, or even single test cases, for example::
 
-    tools/run_tests_docker.sh -i
+    tools/run_tests.sh -i
     tools/pytest.sh tests/test_replica.py
     tools/pytest.sh -vvv tests/test_replica.py::TestReplicaCore
     tools/pytest.sh -vvv --full-trace tests/test_replica.py::TestReplicaCore::test_delete_replicas
@@ -63,13 +63,13 @@ Using the environment including storage
 
 Again run the containers using docker-compose::
 
-    docker-compose --file etc/docker/dev/docker-compose-storage.yml up -d
+    docker-compose --file etc/docker/dev/docker-compose.yml --profile storage up -d
 
 This should show you a few more running containers: the Rucio server, the PostgreSQL database, FTS and its associated MySQL database, the Graphite monitoring, and three XrootD storage servers.
 
 With this container you can upload and download data to/from the storage and submit data transfers. To set this up, add the `-r` option to the setup.::
 
-    tools/run_tests_docker.sh -ir
+    tools/run_tests.sh -ir
 
 This creates a few random files and uploads them, creates a few datasets and containers, and requests a replication rule for the container, which starts in state REPLICATING. To demonstrate the transfer capability, the daemons can be run in single-execution mode in order:::
 
@@ -89,14 +89,14 @@ Using the environment including monitoring
 
 Again run the containers using docker-compose::
 
-    docker-compose --file etc/docker/dev/docker-compose-storage-monit.yml up -d
+    docker-compose --file etc/docker/dev/docker-compose.yml --profile storage --profile monitoring up -d
 
 
 Now you will have the same containers as before plus a full monitoring stack with Logstash, Elasticsearch, Kibana and Grafana.
 
 To create some events and write them to Elasticsearch first run again the tests as before::
 
-    tools/run_tests_docker.sh -ir
+    tools/run_tests.sh -ir
 
 
 Then you will have to run the transfer daemons (conveyor-\*) and messaging daemon (hermes) to send the events to ActiveMQ. There a script for that which repeats these daemons in single execution mode from the section in a loop::
@@ -210,7 +210,7 @@ Start the daemons
 
 Daemons are not running in the docker environment, but all daemons support single-execution mode with the --run-once argument. Reset the system first with::
 
-    tools/run_tests_docker.sh -ir
+    tools/run_tests.sh -ir
 
 
 Some files are created. Let's add them to a new dataset::