Added support for distributed performance testing - CADC-8604 (#178)

* Added support for distributed performance testing

vos/perftest/README.md

@@ -9,29 +9,62 @@ There are 2 types of tests:
   * `site` tests the performance of a site. No transfer negotiantion is required. Just A&A and bytes transfer.
 ## Running it
 
-The system runs on port 8089 with Docker and requires two volumes to be created and populated first:
+The system runs on port 8089 with Docker and requires a volume to be created and populated first with:
 
- * `cert`: to hold the `cadcproxy.pem` file of the authenticated user that runs the tests.
- * `config`: to hold the `test_config` file of the authenticated user that runs the tests.
+ * `cadcproxy.pem` file of the authenticated user that runs the tests.
+ * `test_config` file of the authenticated user that runs the tests.
 
 ### Docker Run
 
 On one of the `global` or `site` directories, the image will need to be built first:
 
-`$ docker build -t myimage .`
+`$ docker build -t [newstorage_site_test|newstorage_global_test] .`
 
 Foreground run:
-`$ docker run -t --rm -p 8089:8089 myimage`
+`$ docker run -t --rm -p 8089:8089 [newstorage_site_test|newstorage_global_test]`
 
 Background run:
-`$ docker run -t -d --name locust -p 8089:8089 myimage`
+`$ docker run -t -d --name locust -p 8089:8089 [newstorage_site_test|newstorage_global_test]`
 
 Then you can visit http://localhost:8089 to use the UI and start a run.
 
 Both tests can be run at the same time on the same machine if different ports on the host are used.
 
 ### Running on distributed system
 
-TBD
+`global` tests are not CPU or network intensive and can be run (swarmed) from the same host. `site` tests on the other
+hand transfer bytes between the service and host and could be run on a distributed system. Tests are easier to run
+with `docker-compose` commands using the `docker-compose.yml` provided for each type of test.
+
+#### Multiple workers from the same host
+
+To start the system with 4 workers, run the command:
+`$docker-compose up --scale worker=4`
+
+Then use the web page at `http://0.0.0.0:8089/` to run the tests and collect data.
+
+To shut down the system run:
+`docker-compose down`
+
+#### Multiple workers on different hosts
+
+Each host needs to have a config file as above.
+
+Start master node:
+`$docker-compose run master --expected-workers=X`
+
+Then on each node start the workers:
+`$docker-compose run worker --master-host=X.X.X.X --master-port=8089`
+
+Number of attached workers will be see at the `http://0.0.0.0:8089/`. Tests cannot be started untill all the workers have
+attached.
+
+Distributed tests can also be run on Kubernetes. Please consult the `locust` documentation
+`https://docs.locust.io/en/stable/running-locust-docker.html#running-locust-docker) to find out the procedure as well
+as other configurations and modes that can be set up.
+
+
+
+
 
 IMPORTANT: Make sure that the testing machines are accessible (they are either public or the appropriate VPN is running)

vos/perftest/global/Dockerfile

@@ -13,8 +13,6 @@ RUN mkdir /locust
 WORKDIR /locust
 COPY locustfile.py . 
 COPY testers.py .
-RUN mkdir cert
-COPY cert/* cert/
 RUN mkdir config
 COPY config/test_config config/
 

vos/perftest/global/config/test_config

@@ -1,6 +1,5 @@
 [GENERAL]
 REG=mach275.cadc.dao.nrc.ca
-CERT_FILE=./cert/cadcproxy.pem
 # number of calls/user/sec
 CALL_FREQ=120
 #ANON_TESTS_ONLY=True

vos/perftest/global/docker-compose.yml

@@ -0,0 +1,16 @@
+version: '3'
+
+services:
+  master:
+    image: newstorage_global_test
+    ports:
+     - "8089:8089"
+    volumes:
+      - ./config:/locust/config
+    command: locust --master -H http://master:8089
+
+  worker:
+    image: newstorage_global_test
+    volumes:
+      - ./config:/locust/config
+    command: locust --worker --master-host master

vos/perftest/global/entrypoint.sh

@@ -1,3 +1,2 @@
 #!/bin/sh
-locust
-#exec "${@}"
+exec "${@}"

vos/perftest/global/testers.py

@@ -25,9 +25,9 @@ def __init__(self, *args, **kwargs):
             global_config['RESOURCE_ID'])
         try:
             self.global_auth_client = storage_inventory.Client(
-            global_config['RESOURCE_ID'], certfile=config['CERT_FILE'])
+            global_config['RESOURCE_ID'], certfile='./config/cadcproxy.pem')
         except KeyError:
-            raise AttributeError('CERT_FILE not set in the config file')
+            raise AttributeError('RESOURCE_ID not set in the config file')
         self.gpub_uris = None
         self.gpriv_uris = None
         try:

vos/perftest/site/Dockerfile

@@ -13,8 +13,6 @@ RUN mkdir /locust
 WORKDIR /locust
 COPY locustfile.py . 
 COPY testers.py .
-RUN mkdir cert
-COPY cert/* cert/
 RUN mkdir config
 COPY config/test_config config/
 
@@ -25,3 +23,4 @@ RUN chmod +x /entrypoint.sh
 USER locust
 
 ENTRYPOINT ["/entrypoint.sh"]
+CMD ["/entrypoint.sh"]

vos/perftest/site/config/test_config

@@ -1,6 +1,5 @@
 [GENERAL]
 REG=mach275.cadc.dao.nrc.ca
-CERT_FILE=./cert/cadcproxy.pem
 # number of calls/user/sec
 CALL_FREQ=1
 # cannot set both ANON_TESTS_ONLY and AUTH_TESTS_ONLY

vos/perftest/site/docker-compose.yml

@@ -0,0 +1,16 @@
+version: '3'
+
+services:
+  master:
+    image: newstorage_site_test
+    ports:
+     - "8089:8089"
+    volumes:
+      - ./config:/locust/config
+    command: locust --master -H http://master:8089
+
+  worker:
+    image: newstorage_site_test
+    volumes:
+      - ./config:/locust/config
+    command: locust --worker --master-host master

vos/perftest/site/entrypoint.sh

@@ -1,3 +1,3 @@
 #!/bin/sh
-locust
-#exec "${@}"
+#locust
+exec "${@}"

vos/perftest/site/testers.py

@@ -35,9 +35,9 @@ def __init__(self, *args, **kwargs):
             site_config['RESOURCE_ID'])
         try:
             self.site_auth_client = storage_inventory.Client(
-            site_config['RESOURCE_ID'], certfile=config['CERT_FILE'])
+            site_config['RESOURCE_ID'], certfile='./config/cadcproxy.pem')
         except KeyError:
-            raise AttributeError('CERT_FILE not set in the config file')
+            raise AttributeError('RESOURCE_ID not set in the config file')
         self.gpub_uris = None
         self.gpriv_uris = None
         try:
@@ -55,7 +55,7 @@ def __init__(self, *args, **kwargs):
 
     def get_next_file_size(self):
         # returns the next size of the file.
-        # round rubin
+        # round-robin
         next_size = self.file_size * 10**self.current_size_order
         # or random
         # next_size = self.file_size * \