Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
Profile and benchmark the ara callback and api server #6
ansible-role-ara_api currently supports a few different deployment options:
All of these options are already integration tested and we could leverage these tests to benchmark the performance of each option. The data would be interesting but also valuable in finding issues and improvement opportunities.
Interested in benchmarking a particular use case: instead of having a single central API server with a central database, instead have a local API server with a remote central database.
So instead of this:
The callback would instead use a local API server:
@dmsimard do you know offhand if multiple API servers behind a LB is a supported configuration too? I may try that in an attempt to address performance.
That is to say:
@zxaos the state is stored in the database so it should be possible.
AWX is built with the same backend (django/django-rest-framework) and they have a docker-compose stack that includes haproxy.
However, in the testing that I've done, the main bottleneck wasn't the API server, it was the callback plugin itself.
For now I'll keep profiling to see if there are low hanging fruits (like ansible-community/ara@974fa25) and then I'll tag 1.2.
For the sake of testing 1.2 before release, I ran Ansible's own integration tests with it and figured it would be worthwhile to capture various metrics.
The API server being tested ran gunicorn with nginx in front and was configured to use a remote MariaDB server located in another virtual machine on the same hypervisor (<1ms latency).
The two virtual machines had modest specs (4vcpu, 4gb ram) and were not very loaded throughout the tests.
The data is currently available on https://api.trunk.demo.recordsansible.org right now.
Here are some results from after all tests completed:
37216 total API calls:
First call: [30/Oct/2019:21:29:11 +0000]
Longest playbook: 8m34s
I also ran the logs through goaccess which provided the following information:
The client and server errors are worth digging into, it looks like most of them are related to files, for example:
The 5xx errors were doing POSTs on results:
Edit: Another data point -- a MySQL dump weighed 9MB in plain text and 4.5MB gzipped.