run-test-plan.py: Run in parallel where possible

Use python's asyncio / await / async to run modules within one test plan in parallel if they don't have an alias, and to run plans in parallel where they use no / different aliases. The parallelism of tests within a plan should speed up the CIBA tests and OIDC test modules where we use dcr can hence can proceed without any alias. Unfortunately the CIBA tests had to have parallelism disabled because Authlete's CIBA simulated authentication device doesn't seem to cope with multiple parallel. Reduces the time to run oidcc-basic-certification-test-plan from 313 seconds to 63 seconds on my local machine - most of which comes from the test that sleeps for 30 seconds before reusing an auth code. The ability to run plans with different aliases in parallel means we can run some of the FAPI tests in parallel with other tests. There may be further potential to speed this up by tweaking the aliases and redirect urls we use in (say) half of the FAPI tests. There was some hassle with python modules/node, I tried switching away from alpine for the reasons given here, where we were seeing issues installing the aiohttp module: https://pythonspeed.com/articles/alpine-docker-python/ but ending up switching back because the version of node (10) in Debian buster is ancient and doesn't support some of the crypto Fillip's client needs. Luckily the aiohttp module is one of the ones that can relatively easily be made to work on alpine. We add a retrying http client, as it seems either the parallelisation or something about the asyncio http client ends up giving weird http errors - I think we're running into the same bug as here: aio-libs/aiohttp#4581 Switching to https://www.python-httpx.org might be an option to avoid that, but that's still in beta. part of #783
openid-certification · Jun 8, 2021 · fc76e29 · fc76e29
1 parent cdd9283
commit fc76e29
Show file tree

Hide file tree

Showing 8 changed files with 321 additions and 155 deletions.
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
@@ -215,7 +215,7 @@ stop_normal:
 .deployment_test: &deployment_test
   stage: test
   interruptible: true
-  image: python:alpine
+  image: python:3.9-alpine
   only:
     refs:
       - branches
@@ -224,6 +224,7 @@ stop_normal:
     when: always
     paths:
       - "*.zip"
+      - "*.txt"
 
 client_test:
   <<: *deployment_test
@@ -257,7 +258,7 @@ local_test:
     - docker-compose -f $CI_PROJECT_DIR/docker-compose-localtest.yml up --detach httpd
     - sleep 5
     - docker-compose -f $CI_PROJECT_DIR/docker-compose-localtest.yml logs --tail="all" oidcc-provider
-    - docker-compose -f $CI_PROJECT_DIR/docker-compose-localtest.yml run test
+    - docker-compose -f $CI_PROJECT_DIR/docker-compose-localtest.yml run test > run-tests-log.txt
   after_script:
     - docker-compose -f $CI_PROJECT_DIR/docker-compose-localtest.yml logs --tail="all" oidcc-provider > $CI_PROJECT_DIR/oidcc-provider-log.txt
     - docker-compose -f $CI_PROJECT_DIR/docker-compose-localtest.yml logs --tail="all" server > $CI_PROJECT_DIR/server-log.txt
@@ -270,6 +271,7 @@ local_test:
       - server-log.txt
       - mongo/data/mongodb-dump.gz
       - "*.zip"
+      - "*.txt"
 
 # ---------------------------------------------------------------------------
 
@@ -516,8 +518,7 @@ local_test:
 
   function install_test_dependencies() {
     echo "Installing extra dependencies"
-    apk add -U openssh-client git
-    apk add --update nodejs nodejs-npm
+    apk add openssh-client git gcc musl-dev bash
     eval $(ssh-agent -s)
     echo "$SSH_PRIVATE_KEY" | ssh-add -
     cd ..
@@ -527,22 +528,24 @@ local_test:
 
   function set_up_for_running_test_plan() {
     install_test_dependencies
-    pip install requests
+    pip install aiohttp aiofiles aiohttp-retry
   }
 
   function run_client_test_plan() {
+    apk add nodejs npm
     echo "Running automated tests against $CONFORMANCE_SERVER"
-    ../conformance-suite/.gitlab-ci/run-tests.sh --client-tests-only
+    ../conformance-suite/.gitlab-ci/run-tests.sh --client-tests-only > ../conformance-suite/run-tests-log.txt
   }
 
   function run_server_test_plan() {
     echo "Running automated tests against $CONFORMANCE_SERVER"
-    ../conformance-suite/.gitlab-ci/run-tests.sh --server-tests-only
+    ../conformance-suite/.gitlab-ci/run-tests.sh --server-tests-only > ../conformance-suite/run-tests-log.txt
   }
 
   function run_ciba_test_plan() {
       echo "Running automated tests against $CONFORMANCE_SERVER"
-      ../conformance-suite/.gitlab-ci/run-tests.sh --ciba-tests-only
+      ../conformance-suite/.gitlab-ci/run-tests.sh --ciba-tests-only > ../conformance-suite/run-tests-log.txt
+      ls -alR ..
     }
 
 before_script:

diff --git a/.gitlab-ci/local-provider-oidcc-conformance-config.json b/.gitlab-ci/local-provider-oidcc-conformance-config.json
@@ -1,5 +1,4 @@
 {
-    "alias": "local-oidcc",
     "description": "oidc-provider OIDC",
     "server": {
         "discoveryUrl": "https://oidcc-provider:3000/.well-known/openid-configuration"
@@ -54,7 +53,7 @@
                 },
                 {
                     "task": "Verify Complete",
-                    "match": "*/test/a/local-oidcc/callback*",
+                    "match": "*/test/*/callback*",
                     "commands": [
                         [
                             "wait",
@@ -88,14 +87,14 @@
                         [
                             "wait",
                             "contains",
-                            "/test/a/local-oidcc/post",
+                            "/test/*/post",
                             10
                         ]
                     ]
                 },
                 {
                     "task": "Verify Complete",
-                    "match": "*/test/a/local-oidcc/post*"
+                    "match": "*/test/*/post*"
                 }
             ]
         }
@@ -155,7 +154,7 @@
                         },
                         {
                             "task": "Verify Complete",
-                            "match": "*/test/a/local-oidcc/callback*",
+                            "match": "*/test/*/callback*",
                             "commands": [
                                 [
                                     "wait",
@@ -223,7 +222,7 @@
                         },
                         {
                             "task": "Verify Complete",
-                            "match": "*/test/a/local-oidcc/callback*",
+                            "match": "*/test/*/callback*",
                             "commands": [
                                 [
                                     "wait",
@@ -450,7 +449,7 @@
                         },
                         {
                             "task": "Verify Complete",
-                            "match": "*/test/a/local-oidcc/callback*",
+                            "match": "*/test/*/callback*",
                             "commands": [
                                 [
                                     "wait",
@@ -529,7 +528,7 @@
                         },
                         {
                             "task": "Verify Complete",
-                            "match": "*/test/a/local-oidcc/callback*",
+                            "match": "*/test/*/callback*",
                             "commands": [
                                 [
                                     "wait",
@@ -608,7 +607,7 @@
                         },
                         {
                             "task": "Verify Complete",
-                            "match": "*/test/a/local-oidcc/callback*",
+                            "match": "*/test/*/callback*",
                             "commands": [
                                 [
                                     "wait",
@@ -687,7 +686,7 @@
                         },
                         {
                             "task": "Verify Complete",
-                            "match": "*/test/a/local-oidcc/callback*",
+                            "match": "*/test/*/callback*",
                             "commands": [
                                 [
                                     "wait",
@@ -766,7 +765,7 @@
                         },
                         {
                             "task": "Verify Complete",
-                            "match": "*/test/a/local-oidcc/callback*",
+                            "match": "*/test/*/callback*",
                             "commands": [
                                 [
                                     "wait",
@@ -845,7 +844,7 @@
                         },
                         {
                             "task": "Verify Complete",
-                            "match": "*/test/a/local-oidcc/callback*",
+                            "match": "*/test/*/callback*",
                             "commands": [
                                 [
                                     "wait",
@@ -935,7 +934,7 @@
                         },
                         {
                             "task": "Verify Complete",
-                            "match": "*/test/a/local-oidcc/callback*",
+                            "match": "*/test/*/callback*",
                             "commands": [
                                 [
                                     "wait",
@@ -1025,7 +1024,7 @@
                         },
                         {
                             "task": "Verify Complete",
-                            "match": "*/test/a/local-oidcc/callback*",
+                            "match": "*/test/*/callback*",
                             "commands": [
                                 [
                                     "wait",

diff --git a/.gitlab-ci/run-tests.sh b/.gitlab-ci/run-tests.sh
@@ -1,4 +1,4 @@
-#!/bin/sh
+#!/bin/bash
 
 set -e
 cleanup() {
@@ -32,8 +32,9 @@ EXPECTED_FAILURES_FILE="../conformance-suite/.gitlab-ci/expected-failures-server
 EXPECTED_SKIPS_FILE="../conformance-suite/.gitlab-ci/expected-skips-server.json|../conformance-suite/.gitlab-ci/expected-skips-ciba.json|../conformance-suite/.gitlab-ci/expected-skips-client.json"
 
 makeClientTest() {
-    . node-client-setup.sh
-    . node-core-client-setup.sh
+	ls -al
+    . ./node-client-setup.sh
+    . ./node-core-client-setup.sh
 
     # client FAPI-RW-ID2
     TESTS="${TESTS} fapi-rw-id2-client-test-plan[client_auth_type=private_key_jwt][fapi_profile=plain_fapi] automated-ob-client-test.json"
@@ -272,6 +273,7 @@ elif [ "$#" -eq 1 ] && [ "$1" = "--client-tests-only" ]; then
     TESTS="${TESTS} --expected-skips-file ${EXPECTED_SKIPS_FILE}"
     TESTS="${TESTS} --show-untested-test-modules client"
     TESTS="${TESTS} --export-dir ../conformance-suite"
+    TESTS="${TESTS} --no-parallel" # there seemed to be a lot of "Server disconnected" failures trying to run these in parallel
     echo "Run client tests"
     makeClientTest
 elif [ "$#" -eq 1 ] && [ "$1" = "--server-tests-only" ]; then
@@ -291,6 +293,7 @@ elif [ "$#" -eq 1 ] && [ "$1" = "--ciba-tests-only" ]; then
     TESTS="${TESTS} --expected-skips-file ${EXPECTED_SKIPS_FILE}"
     TESTS="${TESTS} --show-untested-test-modules ciba"
     TESTS="${TESTS} --export-dir ../conformance-suite"
+    TESTS="${TESTS} --no-parallel" # the authlete authentication device simulator doesn't seem to support parallel authorizations
     echo "Run ciba tests"
     makeCIBATest
 elif [ "$#" -eq 1 ] && [ "$1" = "--local-provider-tests" ]; then