Docs: operations and deployment

[dmsimard]

We have good enough documentation on how to install and configure ara but there is a gap when it comes to day-to-day operations or deployment considerations.

Things like:

• How to monitor the API server (healthcheck)
• How to prune the database from old records (See issue)
• Database migrations (See issue)
• Offline not working because of ipv6: Ara offline mode not working… #134
• Reference architectures (between database backends, wsgi servers, frontend servers, etc.) and deployment considerations (e.g, latency between callback and API server or between API server and database server)
• How to use the included Ansible roles with example playbooks

Whenever we get around to benchmarking and profiling the different deployment options, we could probably reference them there as well.

[zxaos]

+1 to benchmarking and profiling docs. I see a nearly 300% slowdown on playbooks with ara enabled, and don't really know where to even start tuning it.

[dmsimard]

@zxaos I'm going to spend some time on this in the near future.

In order to get some relevant numbers and testing scenarios, could you tell me a little bit about your deployment ?

What's your latency/ping between:

• The Ansible control node (where playbooks are running from) to the API server ?
• If not using sqlite, the API server and the database server ?

Also:

• How many hosts are you running playbooks against ?
• Are you running the API via ara-manage runserver or through an application server like gunicorn or uWSGI ?
• Are you exposing the API behind a web server like apache or nginx ?

Thanks !

[zxaos]

I appreciate the help!

Latency from control node to API Server

100 packets transmitted, 100 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 12.838/34.468/228.610/46.864 ms

Database

Postgres 9.6

(from api-server to db)
100 packets transmitted, 100 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.632/0.832/3.795/0.337 ms

Host Count

166
(it's just a gather_facts for all hosts as a test playbook)

Application Server

gunicorn 2.1.1, 4 threads

Intermediate Web Server

none

[dmsimard]

@zxaos thanks -- from limited testing, there are definitely performance improvement opportunities. I've already cut a lot of API calls through #80 and there's a lot more to iterate on in the near future.

[zxaos]

@dmsimard much appreciated! I'm excited to it out when the patch lands!