Add an alternative locust client (#46)

* add locust client as an alternative

* add a feature: retry if FAIL_RATIO's high

* provide both httpclient and grpcclient

* Add README.md of the locust client

pkg/client_locust/Dockerfile

@@ -0,0 +1,27 @@
+FROM python:3.6
+
+RUN if [ "$(uname -m)" = "ppc64le" ] || [ "$(uname -m)" = "aarch64" ]; then \
+        apt-get -y update && \
+        apt-get -y install gfortran libopenblas-dev liblapack-dev && \
+        pip install cython 'numpy>=1.13.3'; \
+    fi
+RUN GRPC_HEALTH_PROBE_VERSION=v0.3.1 && \
+    if [ "$(uname -m)" = "ppc64le" ]; then \
+	wget -qO/bin/grpc_health_probe https://github.com/grpc-ecosystem/grpc-health-probe/releases/download/${GRPC_HEALTH_PROBE_VERSION}/grpc_health_probe-linux-ppc64le; \
+    elif [ "$(uname -m)" = "aarch64" ]; then \
+        wget -qO/bin/grpc_health_probe https://github.com/grpc-ecosystem/grpc-health-probe/releases/download/${GRPC_HEALTH_PROBE_VERSION}/grpc_health_probe-linux-arm64; \
+    else \
+	wget -qO/bin/grpc_health_probe https://github.com/grpc-ecosystem/grpc-health-probe/releases/download/${GRPC_HEALTH_PROBE_VERSION}/grpc_health_probe-linux-amd64; \
+    fi && \
+    chmod +x /bin/grpc_health_probe
+
+WORKDIR /workspace
+ADD requirements.txt requirements.txt
+RUN /usr/local/bin/python -m pip install --upgrade pip
+#RUN pip install -i https://pypi.tuna.tsinghua.edu.cn/simple --no-cache-dir -r requirements.txt
+RUN pip install --no-cache-dir -r requirements.txt
+
+ADD . /workspace
+
+ENV BATCH_SIZE 1
+ENTRYPOINT ["/bin/sh", "-c", "sleep 100000"]

pkg/client_locust/README.md

@@ -0,0 +1,41 @@
+## Motivation
+As we might want the client to perform a more customized load test to the inference server in a more simple and flexible way, an idea is to use locust as the backend so that the user can directly specify load test params or provide its own locusttest script. A demo client image based on locust is provided here.
+
+## File structure description
+Three meta files describe how to build and use the image.
+
+- `README.md`
+- `Dockerfile`: used to build the client image
+- `jobtemplate.yaml`: a trial client job template using `morphling-client-plugin` image
+
+The other files will be used in the client image.
+
+- `requirements.txt`: python scripts dependencies
+- `morphling_client_locust.py`: main function, responsible for args/env parsing, locust invocation and result saving. When the error rate exceeds `FAIL_RATE`, it will retry with a lower `LOCUST_NUM_USERS`
+- `invokust`: locust wrapper，responsible for launching the pressure test and returning the results. As locust itself does not provide a complete library API, and is more often called as a CLI tool, a wrapper is needed here. In addition, a new feature which supports prometheus metrics export is added.
+- `locust_grpc.py`: constructed a `GrpcUser` super class, easy to be inherited and extended by the user. It will automatically record the results via the event hook `events.request.fire`. An error threshold exit mechanism is also enabled here by the event listener.
+- `locustfile_grpcuser.py` or `locustfile_httpuser.py`: the load test demo script, reserved models are provided which can be used directedly; when cusomization is needed, what we need to do is just to inherit class `GrpcUser` or the native `HttpUser`, and define the function decorated by `@task` 
+- `api_pb2.py` and `api_pb2_grpc.py`: used for database connection 
+- `image.jpg`: used for the demo load test
+
+As described above, the client's functions are decoupled into several modules/files as a plug-in approach. The user can just specify the param or provids its own locustfile without effort. gRPC and HTTP protocols are both supported.  
+
+## Usage
+Currently, we can specify test parameters via ENV.
+
+The exposed APIs are as follows, see examples in the `jobtemplate.yaml`:
+
+- `LOCUST_NUM_USERS`: maximum number of concurrent users, default: 10
+- `LOCUST_SPAWN_RATE`: spawning rate of concurrent users, default: 10
+- `LOCUST_RUN_TIME`: test running duration, default: 15
+- `LOCUST_LOCUSTFILE`: custom test file name, default: locustfile.py
+- `LOCUST_METRICS_EXPORT`: export Prometheus metrics or not, default: False
+- `FAIL_RATIO`: fail ratio threshold, default: 0.2
+- `PRINTLOG`: print log or not, default: false
+
+## Conclusion / Pro and Cons
+To avoid reinvent the wheels, the open-source locust load testing tool is chosen here because of its ease of use and good scalability. It supports HTTP/gRPC and many other protocols and at the same time it can be extended easily to perform distributed load testing. Therefore, compared with the heavyweight Jmeter and lightweight K6, which is hard to implement complex requests, locust is more suitable.
+But there are also some drawbacks. 
+
+- locust itself does not provide a complete Library API and is mostly called as a CLI tool. Therefore, the invokust locust wrapper is used here, which is responsible for launching the load test and returning the results. However, only local runner mode is currently supported. The distributed load tests are not supported now. Since in most cases the QPS is less than 1000 for DL inference tasks on CPU, the pressure generated by the local runner is sufficient.
+- QPS results with SLO guarantee is currently not supported yet. Most of the pressure testing tools do not have this option. It might be realized in the future by the same event hook mechanism as `FAIL_RATIO` retry strategy or by filtering the final qps result.